Emotional intelligence of Large Language Models

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities across numerous disciplines, primarily assessed through tasks in language generation, knowledge utilization, and complex reasoning. However, their alignment with human emotions and values, which is critical for real-world applications, has not been systematically evaluated. Here, we assessed LLMs' Emotional Intelligence (EI), encompassing emotion recognition, interpretation, and understanding, which is necessary for effective communication and social interactions. Specifically, we first developed a novel psychometric assessment focusing on Emotion Understanding (EU), a core component of EI. This test is an objective, performance-driven, and text-based evaluation, which requires evaluating complex emotions in realistic scenarios, providing a consistent assessment for both human and LLM capabilities. With a reference frame constructed from over 500 adults, we tested a variety of mainstream LLMs. Most achieved above-average Emotional Quotient (EQ) scores, with GPT-4 exceeding 89% of human participants with an EQ of 117. Interestingly, a multivariate pattern analysis revealed that some LLMs apparently did not rely on the human-like mechanism to achieve human-level performance, as their representational patterns were qualitatively distinct from humans. In addition, we discussed the impact of factors such as model size, training method, and architecture on LLMs' EQ. In summary, our study presents one of the first psychometric evaluations of the human-like characteristics of LLMs, which may shed light on the future development of LLMs aiming for both high intellectual and emotional intelligence.

This study aims to measure the EI capabilities of LLMs, particularly their competence in EU. To this end, we developed a novel standardized test, known as the Situational Evaluation of Complex Emotional Understanding (SECEU), which was structured to emulate real-life scenarios requiring complex emotional understanding and used data from over 500 young adults to establish a norm. A wide variety of prominent LLMs were evaluated using the SECEU, with their scores standardized against the established norm to allow for a direct comparison with human responses. Our primary findings indicated that most of the LLMs tested achieved above-average EQ scores. However, individual differences across models were significant. GPT-4 stood out, scoring the highest EQ while also exhibiting human-like response patterns, demonstrating a balanced mechanism for high EU proficiency. This research constitutes a comprehensive psychometric examination of LLMs' EI, and illuminates the potential influence of various factors like model size, training methods, and architecture on models' EU performance. Given the ever-growing role of LLMs in human-computer interaction, our study underscores the critical need for emotional intelligence in these systems. The insights gained here will inform future development of LLMs, facilitating the creation of models that embody high levels of both intellectual and emotional intelligence.

Results

Figure : The family tree of LLMs and their corresponding EQ. The SECEU's scores of the models were converted into standard Emotional Quotient (EQ) scores, following a normal distribution where an average score of 100 represents an individual's EU ability relative to the population average. Each node in the tree represents an LLM, whose vertical position along the y-axis indicates the launch time. The size of each node corresponds to the parameter size of the LLM. Note that the size of GPT-4 and xx (if any) was estimated based on publicly available information. Color donates the EQ scores, with red color for higher scores and blue color for lower scores. Note that white color shows that models failed to conduct the SECEU. The color of the branches distinguishes between open-source (light gray) and closed-source (dark gray) models.

Table : EQ, representational patterns, and properties

Citing Emotional Intelligence

If you find the emotional intelligence or data useful, please consider citing:

@article{doi:10.1177/18344909231213958,
                  author = {Xuena Wang and Xueting Li and Zi Yin and Yue Wu and Jia Liu},
                  title = {Emotional intelligence of Large Language Models},
                  journal = {Journal of Pacific Rim Psychology},
                  volume = {17},
                  number = {},
                  pages = {18344909231213958},
                  year = {2023},
                  doi = {10.1177/18344909231213958},
                  URL = {https://doi.org/10.1177/18344909231213958},
                  eprint = {https://doi.org/10.1177/18344909231213958}
                }
                
                

Dataset License

Copyright 2023 ABC Lab

The emotional intelligence dataset is licensed under arXiv.org perpetual, non-exclusive license.