SOLD: Sinhala offensive language dataset

Tharindu Ranasinghe; Isuri Anuradha; Damith Premasiri; Kanishka Silva; Hansi Hettiarachchi; Lasitha  Uyangodage; Marcos Zampieri

doi:10.1007/s10579-024-09723-1

SOLD: Sinhala offensive language dataset

Tharindu Ranasinghe^*, Isuri Anuradha, Damith Premasiri, Kanishka Silva, Hansi Hettiarachchi, Lasitha Uyangodage , Marcos Zampieri

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

The widespread of offensive content online, such as hate speech and cyber-bullying, is a global phenomenon. This has sparked interest in the artificial intelligence (AI) and natural language processing (NLP) communities, motivating the development of various systems trained to detect potentially harmful content automatically. These systems require annotated datasets to train the machine learning (ML) models. However, with a few notable exceptions, most datasets on this topic have dealt with English and a few other high-resource languages. As a result, the research in offensive language identification has been limited to these languages. This paper addresses this gap by tackling offensive language identification in Sinhala, a low-resource Indo-Aryan language spoken by over 17 million people in Sri Lanka. We introduce the Sinhala Offensive Language Dataset (SOLD) and present multiple experiments on this dataset. SOLD is a manually annotated dataset containing 10,000 posts from Twitter annotated as offensive and not offensive at both sentence-level and token-level, improving the explainability of the ML models. SOLD is the first large publicly available offensive language dataset compiled for Sinhala. We also introduce SemiSOLD, a larger dataset containing more than 145,000 Sinhala tweets, annotated following a semi-supervised approach.

Original language	English
Journal	Language Resources and Evaluation
Early online date	6 Mar 2024
DOIs	https://doi.org/10.1007/s10579-024-09723-1
Publication status	E-pub ahead of print - 6 Mar 2024

Bibliographical note

Copyright © The Author(s) 2024. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/

Data Access Statement

Data is available at https://huggingface.co/sinhala-nlp. Code is available at https://github.com/Sinhala-NLP/SOLD.

Keywords

Offensive language identification
Low-resource languages
Deep learning
Transformers

Access to Document

10.1007/s10579-024-09723-1Licence: CC BY 4.0

T Ranasinghe et al_SOLD Sinhala Offensive Language Dataset
Copyright © The Author(s) 2024. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/
Final published version, 2.28 MBLicence: CC BY 4.0

Cite this

@article{2b0c1e49be974462920efc113fd26350,

title = "SOLD: Sinhala offensive language dataset",

abstract = "The widespread of offensive content online, such as hate speech and cyber-bullying, is a global phenomenon. This has sparked interest in the artificial intelligence (AI) and natural language processing (NLP) communities, motivating the development of various systems trained to detect potentially harmful content automatically. These systems require annotated datasets to train the machine learning (ML) models. However, with a few notable exceptions, most datasets on this topic have dealt with English and a few other high-resource languages. As a result, the research in offensive language identification has been limited to these languages. This paper addresses this gap by tackling offensive language identification in Sinhala, a low-resource Indo-Aryan language spoken by over 17 million people in Sri Lanka. We introduce the Sinhala Offensive Language Dataset (SOLD) and present multiple experiments on this dataset. SOLD is a manually annotated dataset containing 10,000 posts from Twitter annotated as offensive and not offensive at both sentence-level and token-level, improving the explainability of the ML models. SOLD is the first large publicly available offensive language dataset compiled for Sinhala. We also introduce SemiSOLD, a larger dataset containing more than 145,000 Sinhala tweets, annotated following a semi-supervised approach.",

keywords = "Offensive language identification, Low-resource languages, Deep learning, Transformers",

author = "Tharindu Ranasinghe and Isuri Anuradha and Damith Premasiri and Kanishka Silva and Hansi Hettiarachchi and Lasitha Uyangodage and Marcos Zampieri",

note = "Copyright {\textcopyright} The Author(s) 2024. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/",

year = "2024",

month = mar,

day = "6",

doi = "10.1007/s10579-024-09723-1",

language = "English",

journal = "Language Resources and Evaluation",

issn = "1574-020X",

publisher = "Paradigm Press",

}

TY - JOUR

T1 - SOLD: Sinhala offensive language dataset

AU - Ranasinghe, Tharindu

AU - Anuradha, Isuri

AU - Premasiri, Damith

AU - Silva, Kanishka

AU - Hettiarachchi, Hansi

AU - Uyangodage , Lasitha

AU - Zampieri, Marcos

N1 - Copyright © The Author(s) 2024. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/

PY - 2024/3/6

Y1 - 2024/3/6

N2 - The widespread of offensive content online, such as hate speech and cyber-bullying, is a global phenomenon. This has sparked interest in the artificial intelligence (AI) and natural language processing (NLP) communities, motivating the development of various systems trained to detect potentially harmful content automatically. These systems require annotated datasets to train the machine learning (ML) models. However, with a few notable exceptions, most datasets on this topic have dealt with English and a few other high-resource languages. As a result, the research in offensive language identification has been limited to these languages. This paper addresses this gap by tackling offensive language identification in Sinhala, a low-resource Indo-Aryan language spoken by over 17 million people in Sri Lanka. We introduce the Sinhala Offensive Language Dataset (SOLD) and present multiple experiments on this dataset. SOLD is a manually annotated dataset containing 10,000 posts from Twitter annotated as offensive and not offensive at both sentence-level and token-level, improving the explainability of the ML models. SOLD is the first large publicly available offensive language dataset compiled for Sinhala. We also introduce SemiSOLD, a larger dataset containing more than 145,000 Sinhala tweets, annotated following a semi-supervised approach.

AB - The widespread of offensive content online, such as hate speech and cyber-bullying, is a global phenomenon. This has sparked interest in the artificial intelligence (AI) and natural language processing (NLP) communities, motivating the development of various systems trained to detect potentially harmful content automatically. These systems require annotated datasets to train the machine learning (ML) models. However, with a few notable exceptions, most datasets on this topic have dealt with English and a few other high-resource languages. As a result, the research in offensive language identification has been limited to these languages. This paper addresses this gap by tackling offensive language identification in Sinhala, a low-resource Indo-Aryan language spoken by over 17 million people in Sri Lanka. We introduce the Sinhala Offensive Language Dataset (SOLD) and present multiple experiments on this dataset. SOLD is a manually annotated dataset containing 10,000 posts from Twitter annotated as offensive and not offensive at both sentence-level and token-level, improving the explainability of the ML models. SOLD is the first large publicly available offensive language dataset compiled for Sinhala. We also introduce SemiSOLD, a larger dataset containing more than 145,000 Sinhala tweets, annotated following a semi-supervised approach.

KW - Offensive language identification

KW - Low-resource languages

KW - Deep learning

KW - Transformers

UR - https://link.springer.com/article/10.1007/s10579-024-09723-1

UR - http://www.scopus.com/inward/record.url?scp=85186921179&partnerID=8YFLogxK

U2 - 10.1007/s10579-024-09723-1

DO - 10.1007/s10579-024-09723-1

M3 - Article

SN - 1574-020X

JO - Language Resources and Evaluation

JF - Language Resources and Evaluation

ER -

SOLD: Sinhala offensive language dataset

Abstract

Bibliographical note

Data Access Statement

Keywords

Access to Document

Other files and links

Fingerprint

Cite this