Application of multi-modal data fusion based on deep learning in diagnosis of depression
DOI:
https://doi.org/10.56294/dm2025863Keywords:
Multi-modal data, Deep learning (DL), Diagnosis, Depression, Dynamic Dolphin Echolocation-tuned Effective Temporal Convolutional Networks (DDE-ETCN)Abstract
Depression is a frequent mental condition requiring precise diagnosis in its early onset. Traditional methods are less than accurate and occur late. Following these deficits, this investigates the multi-modal data fusion and Deep Learning (DL) with the purpose of enhancing accuracy for diagnosis. A new DL model, Dynamic Dolphin Echolocation-tuned Effective Temporal Convolutional Networks (DDE-ETCN), is utilized for depression diagnosis. Different sources of data, such as physiological signals (EEG, heart rate), behavioral indicators (facial expressions), and biometric data (activity levels), are fused. Data preprocessing includes wavelet transformation and normalization of biometric and physiological data, and median filtering of behavioral data to provide smooth inputs. Feature extraction is performed through Fast Fourier Transform (FFT) to obtain frequency-domain features of depression indicators. Feature-level fusion is a good fusion of all data sources, which improves the model's performance. The DDE tuning mechanism improves temporal convolution layers to improve the model's ability in detecting sequential changes. The proposed DDE-ETCN model highly improves depression diagnosis when it is developed in Python. The model attains an RMSE of 3.59 and an MAE of 3.09. It has 98.72% accuracy, 98.13% precision, 97.65% F1-score, and 97.81% recall, outperforming conventional diagnostic models and other deep learning-based diagnostic models. The outcomes show the efficiency of the model, rendering a more objective and accurate depression diagnosis. Its higher performance justifies its potential for practical use, providing enhanced accuracy and reliability compared to traditional approaches. This innovation emphasizes the necessity of incorporating deep learning for enhanced mental health evaluations.
References
1. Meshram, P. and Rambola, R.K., 2023. RETRACTED: Diagnosis of depression level using multimodal approaches using deep learning techniques with multiple selective features. Expert Systems, 40(4), p.e12933. https://doi.org/10.1111/exsy.12933
2. Wang, Y., Wang, Z., Li, C., Zhang, Y. and Wang, H., 2022. Online social network individual depression detection using a multitask heterogenous modality fusion approach. Information Sciences, 609, pp.727-749. https://doi.org/10.1016/j.ins.2022.07.109
3. Rajawat, A.S., Bedi, P., Goyal, S.B., Bhaladhare, P., Aggarwal, A. and Singhal, R.S., 2023. Fusion fuzzy logic and deep learning for depression detection using facial expressions. Procedia Computer Science, 218, pp.2795-2805. https://doi.org/10.1016/j.procs.2023.01.251
4. Zhou, L., Liu, Z., Shangguan, Z., Yuan, X., Li, Y. and Hu, B., 2022. TAMFN: time-aware attention multimodal fusion network for depression detection. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31, pp.669-679.https://doi.org/10.1109/TNSRE.2022.3224135
5. Patil, M., Mukherji, P. and Wadhai, V., 2024. Federated learning and deep learning framework for mri image and speech signal-based multi-modal depression detection. Computational Biology and Chemistry, 113, p.108232. https://doi.org/10.1016/j.compbiolchem.2024.108232
6. Kuttala, R., Subramanian, R. and Oruganti, V.R.M., 2023. Multimodal hierarchical CNN feature fusion for stress detection. IEEE Access, 11, pp.6867-6878. https://doi.org/10.1109/ACCESS.2023.3237545
7. Tai, C.H., Chung, K.H., Teng, Y.W., Shu, F.M. and Chang, Y.S., 2021. Inference of mood state indices by using a multimodal high-level information fusion technique. IEEE Access, 9, pp.61256-61268. https://doi.org/10.1109/ACCESS.2021.3073733
8. Gu, Y., Peng, S., Li, Y., Gao, L. and Dong, Y., 2025. FC-HGNN: A heterogeneous graph neural network based on brain functional connectivity for mental disorder identification. Information Fusion, 113, p.102619. https://doi.org/10.1016/j.inffus.2024.102619
9. Tian, F., Zhang, L., Zhu, L., Zhao, M., Liu, J., Dong, Q. and Zhao, Q., 2024. Advancements in affective disorder detection: Using multimodal physiological signals and neuromorphic computing based on snns. IEEE Transactions on Computational Social Systems. https://doi.org/10.1109/TCSS.2024.3420445
10. Wang, G., Fan, F., Shi, S., An, S., Cao, X., Ge, W., Yu, F., Wang, Q., Han, X., Tan, S. and Tan, Y., 2024. Multi-modality fusion transformer with spatio-temporal feature aggregation module for psychiatric disorder diagnosis. Computerized Medical Imaging and Graphics, 114, p.102368. https://doi.org/10.1016/j.compmedimag.2024.102368
11. Liu, J., Huang, Y., Chai, S., Sun, H., Huang, X., Lin, L. and Chen, Y.W., 2022. Computer-aided detection of depressive severity using multimodal behavioral data. Handbook of Artificial Intelligence in Healthcare: Vol. 1-Advances and Applications, pp.353-371. https://doi.org/10.1007/978-3-030-79161-2_14
12. Kong, Y., Niu, S., Gao, H., Yue, Y., Shu, H., Xie, C., Zhang, Z. and Yuan, Y., 2022. Multi-stage graph fusion networks for major depressive disorder diagnosis. IEEE Transactions on Affective Computing, 13(4), pp.1917-1928. https://doi.org/10.1109/TAFFC.2022.3205652
13. Gupta, C., Khullar, V., Goyal, N., Saini, K., Baniwal, R., Kumar, S. and Rastogi, R., 2023. Cross-Silo, Privacy-Preserving, and Lightweight Federated Multimodal System for the Identification of Major Depressive Disorder Using Audio and Electroencephalogram. Diagnostics, 14(1), p.43. https://doi.org/10.3390/diagnostics14010043
14. Sun, H., Liu, J., Chai, S., Qiu, Z., Lin, L., Huang, X. and Chen, Y., 2021. Multi-modal adaptive fusion transformer network for the estimation of depression level. Sensors, 21(14), p.4764. https://doi.org/10.3390/s21144764
15. Xie, W., Wang, C., Lin, Z., Luo, X., Chen, W., Xu, M., Liang, L., Liu, X., Wang, Y., Luo, H. and Cheng, M., 2022. Multimodal fusion diagnosis of depression and anxiety based on CNN-LSTM model. Computerized Medical Imaging and Graphics, 102, p.102128. https://doi.org/10.1016/j.compmedimag.2022.102128
16. Fang, M., Peng, S., Liang, Y., Hung, C.C. and Liu, S., 2023. A multimodal fusion model with multi-level attention mechanism for depression detection. Biomedical Signal Processing and Control, 82, p.104561.https://doi.org/10.1016/j.bspc.2022.104561
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Aimin Pan (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
The article is distributed under the Creative Commons Attribution 4.0 License. Unless otherwise stated, associated published material is distributed under the same licence.
