1.Paturi Sai Spandana, 2.Ponnaganti Dharani Naga Priya, 3. OSNVJBS Harini, 4.Gowrni Sai Sandhya, 5.Bandaru Shiva Nagendra

DOI: https://doi.org/10.5281/zenodo.23254669

Mental and emotional well-being can be difficult to support consistently because professional care may be limited by cost, availability, geographic distance, or social stigma. This paper presents the design of a Smart Mental Health Assistant that combines text-based and facial-expression-based emotion recognition to estimate a user’s emotional state. The framework uses text processing and facial analysis as complementary sources, followed by emotion fusion and downstream modules for mood trend analysis, personalized supportive responses, explainability, privacy-aware processing, and safety-aware escalation. Existing digital wellness systems often rely on a single modality or isolated support mechanisms, which can provide an incomplete representation of the user’s emotional state. The proposed framework addresses this limitation by integrating multiple emotional cues within a single pipeline and by explicitly separating emotional-wellness support from clinical diagnosis. The system is intended to provide supportive conversations, recommendations, and appropriate crisis-support resources when risk indicators are detected. At the current development stage, the implementation is being evaluated on a local machine and full empirical testing and deployment are not yet complete; therefore, this manuscript does not claim unverified accuracy or clinical effectiveness.

Multimodal Emotion Recognition; Text Emotion Recognition; Facial Emotion Recognition; Emotion Fusion; Mental Wellness Assistant; Conversational AI; Mood Tracking; Crisis Detection.

This paper presents a Smart Mental Health Assistant framework that combines emotion recognition from text and facial expressions with decision-level fusion, emotional conflict detection, trend analysis, supportive recommendations, adaptive conversation, and safety-aware escalation. The framework is designed as an emotional-wellness support tool rather than a clinical diagnostic system. The current implementation runs locally and remains under development; therefore, the paper deliberately avoids unsupported performance or deployment claims

[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186.
[2] A. J. Hong, D. DiStefano, and S. Dua, “Can CNNs Accurately Classify Human Emotions? A Deep-Learning Facial Expression Recognition Study,” arXiv preprint arXiv:2310.09473, 2023.
[3] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal Machine Learning: A Survey and Taxonomy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443, 2019.
[4] H. Gaffney, W. Mansell, and S. Tai, “Conversational Agents in the Treatment of Mental Health Problems: Mixed-Method Systematic Review,” JMIR Mental Health, vol. 6, no. 10, e14166, 2019.
[5] “Systematic Review and Meta-Analysis of AI-Based Conversational Agents for Promoting Mental Health and Well-Being,” PMC10730549, 2023.
[6] T. Aledavood et al., “Multimodal Digital Phenotyping Study in Patients With Major Depressive Episodes and Healthy Controls (Mobile Monitoring of Mood): Observational Study,” 2025.
[7] “Personalizing a Mental Health Texting Intervention Using Reinforcement Learning” (StayWell), PMC12673119, 2025.
[8] “Effectiveness of a Digital Health Intervention Leveraging Reinforcement Learning: Results From the Diabetes and Mental Health Adaptive Notification Tracking and Evaluation (DIAMANTE) Randomized Clinical Trial,” PMC11496924, 2024.
[9] “Explainable AI for Clinical Risk Prediction: A Survey of Concepts, Methods, and Modalities,” arXiv preprint arXiv:2308.08407, 2023.
[10] “Multimodal Observable Cues in Mood, Anxiety, and Borderline Personality Disorders: A Review of Reviews to Inform Explainable AI in Mental Health,” PMC12723009, 2024.
[11] M. T. Ribeiro, S. Singh, and C. Guestrin, “Why Should I Trust You?: Explaining the Predictions of Any Classifier,” in Proc. ACM SIGKDD, 2016.
[12] S. M. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” in Proc. NeurIPS, 2017.
[13] N. Sarwar, “FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Framework,” arXiv preprint arXiv:2503.05786, 2025.
[14] “Federated Learning in Healthcare Ethics: A Systematic Review of Privacy-Preserving and Equitable Medical AI,” PMC12896918, 2025.
[15] Y. Ophir, R. Tikochinski, C. S. C. Asterhan, I. Sisso, and R. Reichart, “Deep Neural Networks Detect Suicide Risk from Textual Facebook Posts,” Scientific Reports, vol. 10, p. 16685, 2020.

1.Paturi Sai Spandana, 2.Ponnaganti Dharani Naga Priya, 3. OSNVJBS Harini, 4.Gowrni Sai Sandhya, 5.Bandaru Shiva Nagendra (2026). Personalized Mental Health Assistant Using Emotion Recognition from Text and Facial Expressions: A Multimodal Framework for Emotion-Aware Wellness Support. International Journal of Computer Science Engineering Techniques, 10(5), 247–251. ISSN: 2455-135X. DOI: https://doi.org/10.5281/zenodo.23254669