Automation and modeling in design and management Automation and modeling in design and management Автоматизация и моделирование в проектировании и управлении 2658-3488 2658-6436 57984 10.30987/2658-6436-2023-1-51-62 Управление в организационных системах Management in organizational systems Управление в организационных системах Decision support in providing personalized services using emotional artificial intelligence Поддержка принятия решений при оказании персонализированных услуг с применением эмоционального искусственного интеллекта Богданова Диана Радиковна Bogdanova Diana Radikovna

кандидат технических наук;

candidate of technical sciences;

 Котельников Виталий Александрович Kotelnikov Vitaly Aleksandrovich vit_kot@mail.ru

кандидат технических наук;

candidate of technical sciences;

 Юламанова Аделина Рамилевна Yulamanova Adelina Ramilevna adel551155@mail.ru Уфимский университет науки и технологий Россия Ufa University of Science and Technology Russian Federation Уфимский университет науки и технологий Россия Ufa University of Science and Technology Russian Federation Уфимский университет науки и технологий Россия Ufa University of Science and Technology Russian Federation 31 03 2023 31 03 2023 2023 1 51 62 28 03 2023 https://zh-szf.ru/en/nauka/article/57984/view

Описывается подход к персонализированному оказанию услуг, основанный на применении технологий аффективных вычислений. Предлагаемый подход состоит из учета эмоциональных состояний клиентов и их индивидуальных особенностей в процессе оказания услуг. Процесс оказания услуг дополняется этапами формализации эмоциональных состояний и эмоциональной поддержки клиента. В качестве предметной области исследования в работе рассматривается дистанционное образование. Приведено общее описание процесса дистанционного обучения. Делается вывод об отсутствии коррекции эмоционального состояния обучающихся во время занятия. Выявлена зависимость уровня знаний обучающихся от их эмоционального состояния. Приведен обзор существующих подходов к учету эмоциональных состояний обучающихся в процессе дистанционного обучения. Проанализирована специфика поведения обучающихся во время занятия. Также рассмотрены особенности академических эмоций. Поставлена задача повышения эффективности дистанционного обучения за счет учета эмоциональных состояний обучающихся и их индивидуальных особенностей и осуществления эмоциональной поддержки в процессе обучения. Предлагается подход к формализации эмоциональных состояний обучающихся на основе применения движений мышц лица в качестве универсального способа распознавания эмоций. Также подробно описываются стадии распознавания эмоций обучающихся во время занятия. Поставлена задача подбора эмоциональной поддержки на основе классификации обучающихся по эмоциональному состоянию и их индивидуальным особенностям с применением метода ближайшего соседа.

An approach to personalized service rendering based on using affective computing technologies is described. The proposed approach consists of considering clients’ emotional states and their individual characteristics in the process of providing services. Rendering services is supplemented by the formalization stages of the client’s emotional state and emotional support. The paper considers online learning as the subject area of research. A general description of the online learning process is given. It is concluded that there is no correction of the learners’ emotional state during the lesson. The dependence of the learners’ knowledge level on their emotional state is revealed. A review of existing approaches to considering learners’ emotional states in the process of online learning is given. Learners’ specific behaviour during the lesson is analysed. The features of academic emotions are also considered. The objective is set to increase the online learning effectiveness by taking into account learners’ emotional states and their individual characteristics and by providing emotional support in the learning process. An approach is proposed to formalise learners’ emotional states based on using facial muscle movements as a universal way of recognizing emotions. The stages of recognizing learners’ emotions during the lesson are also described in detail. The task is set to select emotional support based on the learners’ classification according to their emotional state and their individual characteristics using the nearest neighbour method.

 персонализированные услуги аффективные вычисления распознавание эмоций обучающихся дистанционное обучение распознавание мимики классификация метод ближайшего соседа personalized services affective computing learners’ emotion recognition online learning facial expression recognition classification nearest neighbour method работа выполнена при поддержке гранта РНФ №22-19-00471 The work was supported by the Russian Science Foundation grant No. 22-19-00471.

Пырьев Е.А. Эмоции в структуре мотивации учебно-профессиональной деятельности студентов вуза // Вестник Костромского государственного университета. Серия: Педагогика. Психология. Социокинетика. 2016. №1. Pyriev E.A. Emotions in the Motivation Structure of Educational and Professional Activities of University Students. Vestnik of Kostroma State University. Series: Pedagogy. Psychology. Sociokinetics. 2016;1. Mohammad Nehal Hasnine, Huyen T.T. Bui, Thuy Thi Thu Tran, Ho Tran Nguyen, Gökhan Akçapınar, Hiroshi Ueda, Students’ emotion extraction and visualization for engagement detection in online learning, Procedia Computer Science. 2021. Vol. 192. P. 3423-3431. Hasnine M.N., Bui H.T.T., Tran T.T.T., Nguyen H.T., Akçapınar G., Ueda H. Students’ Emotion Extraction and Visualization for Engagement Detection in Online Learning, Procedia Computer Science. 2021;192:3423-3431. Weifeng Liu, Lianbo Zhang, Dapeng Tao, Jun Cheng, Reinforcement online learning for emotion prediction by using physiological signals. Pattern Recognition Letters. 2018. Vol. 107. P. 123-130. Liu W., Zhang L., Tao D., Cheng J. Reinforcement Online Learning for Emotion Prediction by Using Physiological Signals. Pattern Recognition Letters. 2018;107:123-130. Ling Cen, Fei Wu, Zhu Liang Yu, Fengye Hu, Chapter 2 - A Real-Time Speech Emotion Recognition System and its Application in Online Learning, Editor(s): Sharon Y. Tettegah, Martin Gartmeier, In Emotions and Technology, Emotions, Technology, Design, and Learning. Academic Press. 2016. P. 27-46. Cen L, Wu F, Yu ZhL, Hu F. A Real-Time Speech Emotion Recognition System and Its Application in Online Learning. In: Tettegah SY, Gartmeier M, editors. Emotions and Technology, Emotions, Technology, Design, and Learning. Academic Press; 2016. Chapter 2, p. 27-46. Aimee L. Whiteside, Amy Garrett Dikkers, Chapter 11 - Leveraging the Social Presence Model: A Decade of Research on Emotion in Online and Blended Learning, Editor(s): Sharon Y. Tettegah, Michael P. McCreery, In Emotions and Technology, Emotions, Technology, and Learning, Academic Press. 2016. P. 225-241. Whiteside AL, Dikkers AG. Leveraging the Social Presence Model: A Decade of Research on Emotion in Online and Blended Learning. In: Tettegah ShY, McCreery MP, editors. Emotions and Technology, Emotions, Technology, and Learning. Academic Press; 2016. Chapter 11, p. 225-241. Krystle Phirangee, Jim Hewitt, Chapter 4 - Loving this Dialogue: Expressing Emotion Through the Strategic Manipulation of Limited Non-Verbal Cues in Online Learning Environments, Editor(s): Sharon Y. Tettegah, Michael P. McCreery, In Emotions and Technology, Emotions, Technology, and Learning, Academic Press. 2016. P. 69-85. Phirangee Kr, Hewitt J. Loving this Dialogue: Expressing Emotion through the Strategic Manipulation of Limited Non-Verbal Cues in Online Learning Environments. In: Tettegah ShY, McCreery MP, editors. Emotions and Technology, Emotions, Technology, and Learning. Academic Press; 2016. Chapter 4, p. 69-85. Mathew Swerdloff, Chapter 8 - Online Learning, Multimedia, and Emotions, Editor(s): Sharon Y. Tettegah, Michael P. McCreery, In Emotions and Technology, Emotions, Technology, and Learning, Academic Press. 2016. P. 155-175. Swerdloff M. Online Learning, Multimedia, and Emotions. In: Tettegah ShY, McCreery MP, editors. Emotions and Technology, Emotions, Technology, and Learning. Academic Press; 2016. Chapter 8, p. 155-175. D. Yang, Abeer Alsadoon, P.W.C. Prasad, A.K. Singh, A. Elchouemi, An Emotion Recognition Model Based on Facial Recognition in Virtual Learning Environment, Procedia Computer Science. 2018. Vol. 125. P. 2-10. Yang D., Alsadoon A., Prasad P.W.C., Singh A.K., Elchouemi A. An Emotion Recognition Model Based on Facial Recognition in Virtual Learning Environment. Procedia Computer Science. 2018;125:2-10. Maryam Imani, Gholam Ali Montazer, A survey of emotion recognition methods with emphasis on E-Learning environments, Journal of Network and Computer Applications. 2019. Vol. 147. 102423. Imani M., Montazer Gh.A. A Survey of Emotion Recognition Methods with Emphasis on E-Learning Environments. Journal of Network and Computer Applications. 2019;147:102423. M. Bouhlal, K. Aarika, R. Ait Abdelouahid, S. Elfilali, E. Benlahmar, Emotions recognition as innovative tool for improving students’ performance and learning approaches, Procedia Computer Science. Vol. 175. 2020. P. 597-602. Bouhlal M., Aarika K., Abdelouahid R.A., Elfilali S., Benlahmar E. Emotions Recognition as Innovative Tool for Improving Students’ Performance and Learning Approaches. Procedia Computer Science. 2020;175:597-602. X. Song and Y. Song, "Research and Implementation of Online Learning System Based on Electroencephalogram Emotion Computing," 2020 5th International Conference on Mechanical, Control and Computer Engineering (ICMCCE). 2020. pp. 1663-1666. Song X, Song Y. Research and Implementation of Online Learning System Based on Electroencephalogram Emotion Computing. In: Proceedings of the 5th International Conference on Mechanical, Control and Computer Engineering (ICMCCE); 2020. p. 1663-1666. T. C. Sandanayake and A. P. Madurapperuma, "Affective e-learning model for recognising learner emotions in online learning environment," 2013 International Conference on Advances in ICT for Emerging Regions (ICTer). 2013. pp. 266-271. Sandanayake TC, Madurapperuma AP. Affective E-Learning Model for Recognising Learner Emotions in Online Learning Environment. In: Proceedings of International Conference on Advances in ICT for Emerging Regions (ICTer); 2013. p. 266-271. C. Ma, C. Sun, D. Song, X. Li and H. Xu, "A Deep Learning Approach for Online Learning Emotion Recognition," 2018 13th International Conference on Computer Science & Education. 2018. pp. 1-5. Ma C, Sun C, Song D, Li X, Xu H. A Deep Learning Approach for Online Learning Emotion Recognition. In: Proceedings of the 13th International Conference on Computer Science & Education; 2018. p. 1-5. X. Zhang, C. Luo, T. He, X. Yang, Z. Lu and B. Huang, "Online learner emotional analysis based on big dataset of online learning forum," 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). 2017. pp. 1-5. Zhang X, Luo C, He T, Yang X, Lu Z, Huang B. Online Learner Emotional Analysis Based on Big Dataset of Online Learning Forum. In: Proceedings of the 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI); 2017. p. 1-5. J. Min Han Pang, T. Connie and G. Kah Ong Michael, "Recognition of Academic Emotions in Online Classes," 2021 9th International Conference on Information and Communication Technology (ICoICT). 2021. pp. 445-450. Pang JMH, Connie T, Michael GKO. Recognition of Academic Emotions in Online Classes. In: Proceedings of the 9th International Conference on Information and Communication Technology (ICoICT); 2021. p. 445-450. A. Jain, H. R. Sah and A. Kothari, "Study for Emotion Recognition of Different Age Groups Students during Online Class," 2021 8th International Conference on Computing for Sustainable Global Development (INDIACom). 2021. pp. 621-625. Jain A, Sah HR, Kothari A. Study for Emotion Recognition of Different Age Groups Students during Online Class. In: Proceedings of the 8th International Conference on Computing for Sustainable Global Development (INDIACom); 2021. p. 621-625. Iolie Nicolaidou, Federica Tozzi, Athos Antoniades, A gamified app on emotion recognition and anger management for pre-school children, International Journal of Child-Computer Interaction. Vol. 31. 2022. 100449. Nicolaidou I., Tozzi F., Antoniades A. A Gamified App on Emotion Recognition and Anger Management for Pre-School Children. International Journal of Child-Computer Interaction. 2022;31:100449. Li Lyu, Ya Zhang, Meng-Ya Chi, Fei Yang, Shu-Gang Zhang, Peng Liu, Wei-Gang Lu, Spontaneous facial expression database of learners’ academic emotions in online learning with hand occlusion, Computers & Electrical Engineering, Vol. 97. 2022. 107667. Lyu L., Zhang Y., Chi M-Y., Yang F., Zhang Sh-G., Liu P., Lu W.-G. Spontaneous Facial Expression Database of Learners’ Academic Emotions in Online Learning with Hand Occlusion. Computers & Electrical Engineering. 2022;97:107667. Soumya Kuruvayil, Suja Palaniswamy, Emotion recognition from facial images with simultaneous occlusion, pose and illumination variations using meta-learning, Journal of King Saud University - Computer and Information Sciences. 2021. Kuruvayil S., Palaniswamy S. Emotion Recognition from Facial Images with Simultaneous Occlusion, Pose and Illumination Variations Using Meta-Learning. Journal of King Saud University - Computer and Information Sciences. 2021. Belinda Berweger, Sebastian Born, Julia Dietrich, Expectancy-value appraisals and achievement emotions in an online learning environment: Within- and between-person relationships, Learning and Instruction. 2022. Vol. 77. 101546. Berweger B., Born S., Dietrich J. Expectancy-Value Appraisals and Achievement Emotions in an Online Learning Environment: Within- and between-Person Relationships, Learning and Instruction. 2022;77:101546. Экман П., Фризен В. В. Система кодирования лицевых движений: метод измерения движений лица. // Консультации психологов Пресс, Пало-Альто. Калифорния. 1978. Ekman P., Friesen V.V. Facial Action Coding System: a Technique for the Measuring of Facial Movements. Palo Alto (Cal): Consultations of Psychologists Press; 1978. Якупова А. В., Сметанина О. Н., Сазонова Е. Ю. Программное решение задачи сегментирования на основе интеллектуальных технологий // Вестник УГАТУ. 2021. Т. 25. № 3(93). С.132-144. Yakupova A.V., Smetanina O.N., Sazonova E.Yu. Software Solution of Segmentation Problem Based on Intelligent Technologies. Vestnik of Ufa State Aviation Technical University. 2021;25(3-93):132-144. Iulamanova A., Bogdanova D., Kotelnikov V. Decision Support in the Automated Compilation of Individual Training Module Based on the Emotional State of Students // IFAC-PapersOnLine series. 2021. Vol. 54. Issue 13. P. 85-90. Iulamanova A., Bogdanova D., Kotelnikov V. Decision Support in the Automated Compilation of Individual Training Module Based on the Emotional State of Students. IFAC-PapersOnLine Series. 2021;54(13):85-90.