I2C-UHU at HOMO-MEX2024: Leveraging Large Language Models and Ensembling Transformers to Identify and Classify Hate Messages Towards the LGBTQ+ Community

Abstract

This paper was presented at the I International Workshop on Conspiracy theories and hate speech online: Comparison of patterns in narratives and social media about Covid 19, immigrants, refugees and LGTBIQ+ people. Universidad de Huelva, July 12 14, 2023 (https://eventos.uhu.es/99642/detail/i-international-workshop-nonconspirahate-project.html). This study presents the strategies advanced by the I2C Group to address the IberLEF-2024 Task HOMO-MEX: Hate speech detection in Online Messages directed towards the Mexican Spanish-speaking LGBTQ+ population. The major contribution has been the integration of Large Language Models (LLMs) for classification through prompting, alongside an ensemble of Transformers. By leveraging the advanced capabilities of LLMs for direct classification tasks, significant improvements in performance were achieved. The ensemble approach, which combines multiple models, further enhanced the results by leveraging the individual strengths of each model. The experiments highlighted the importance of selecting appropriate hyperparameters during the model training process. Through meticulous experimentation and evaluation of different hyperparameter combinations, the optimal settings for achieving the best performance were identified. In the experiments for Task 1, several models were tested, and multiple ensemblers were created. The first ensembler combined Transformers, and its result was further ensembled with two LLMs, obtaining the best F1-Score for this dataset. The model submitted for Task 1 achieved an F1-Score of 87.64%, ranking in 3rd place in the competition. The paper is part of the I+D+i Project titled "Conspiracy Theories and Hate Speech Online: Comparison of Patterns in Narratives and social networks about COVID-19, immigrants, refugees, and LGBTI people [NON-CONSPIRA-HATE!]", PID2021-123983OB-I00, funded by MCIN/AEI/10.13039/501100011033/ and by "ERDF/EU." (https://eseis.es/investigacion/discursos-de-odio/discursos-odio-tc). We are also grateful for the support of our research group: "Estudios Sociales E Intervención Social" (GrupoESEIS), and the research center "Pensamiento Contemporáneo e Innovación para el Desarrollo Social" (COIDESO), and the Applied Computational Social Science Lab, CISCOA-Lab, at the University of Huelva.