Data lake management using topic modeling techniques
DOI:
https://doi.org/10.56294/dm2024282Keywords:
Data Lake, Big Data, Machine Learning, Topic Modeling, LDA, LSAAbstract
With the rapid rise of information technology, the amount of unstructured data from the data lake is rapidly growing and has become a great challenge in analyzing, organizing and automatically classifying in order to derive the meaningful information for a data-driven business. The scientific document has unlabeled text, so it's difficult to properly link it to a topic model. However, crafting a topic perception for a heterogeneous dataset within the domain of big data lakes presents a complex issue. The manual classification of text documents requires significant financial and human resources. Yet, employing topic modeling techniques could streamline this process, enhancing our understanding of word meanings and potentially reducing the resource burden. This paper presents a comparative study on metadata-based classification of scientific documents dataset, applying the two well-known machine learning-based topic modelling approaches, Latent Dirichlet Analysis (LDA) and Latent Semantic Allocation (LSA). To assess the effectiveness of our proposals, we conducted a thorough examination primarily centred on crucial assessment metrics, including coherence scores, perplexity, and log-likelihood. This evaluation was carried out on a scientific publications corpus, according to information from the title, abstract, keywords, authors, affiliation, and other metadata aspects. Results of these experiments highlight the superior performance of LDA over LSA, evidenced by a remarkable coherence value of (0,884) in contrast to LSA's (0,768)
References
1. Boyd-Graber, J., Hu, Y., & Mimno, D. (2017). Applications of Topic Models. Foundations and Trends® in Information Retrieval, 11, 143‑296. https://doi.org/10.1561/1500000030
2. Boussaadi, S., Aliane, D. H., & Abdeldjalil, P. O. (2020). The Researchers Profile with Topic Modeling. IEEE International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS), 1‑6. https://doi.org/10.1109/ICECOCS50124.2020.9314588
3. Kherwa, P., & Bansal, P. (2018). Topic Modeling : A Comprehensive Review. ICST Transactions on Scalable Information Systems, 7, 159623. https://doi.org/10.4108/eai.13-7-2018.159623
4. Anupriya, P., & Karpagavalli, S. (2015). LDA based topic modeling of journal abstracts. 2015 International Conference on Advanced Computing and Communication Systems, 1‑5. https://doi.org/10.1109/ICACCS.2015.7324058
5. Newman, D., Noh, Y., Talley, E., Karimi, S., & Baldwin, T. (2010). Evaluating topic models for digital libraries. Proceedings of the 10th Annual Joint Conference on Digital Libraries, 215‑224. https://doi.org/10.1145/1816123.1816156
6. Syed, S., & Spruit, M. (2017). Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation (p. 174). https://doi.org/10.1109/DSAA.2017.61
7. Wörner, J., Konadl, D., Schmid, I., & Leist, S. (2021, juin 14). Comparison of topic modelling techniques in marketing—Results from an analysis of distinctive use cases. European Conference on Information Systems (ECIS).
8. Hua, T., Lu, C.-T., Choo, J., & Reddy, C. (2020). Probabilistic Topic Modeling for Comparative Analysis of Document Collections. ACM Transactions on Knowledge Discovery from Data, 14, 1‑27. https://doi.org/10.1145/3369873
9. Vayansky, I., & Kumar, S. (2020). A review of topic modeling methods. Information Systems, 94, 101582. https://doi.org/10.1016/j.is.2020.101582
10. Cherradi, M., & El Haddadi, A. (2023). EMEMODL : Extensible Metadata Model for Big Data Lakes. International Journal of Intelligent Engineering and Systems, 16. https://doi.org/10.22266/ijies2023.0630.18
11. Zagan, E., & Danubianu, M. (2020). Data Lake Approaches : A Survey. International Conference on Development and Application Systems (DAS), 189‑193. https://doi.org/10.1109/DAS49615.2020.9108912
12. Cherradi, M., & El Haddadi, A. (2022). Data Lakes : A Survey Paper. In Innovations in Smart Cities Applications, Vol. 5 (p. 823‑835). Springer. https://doi.org/10.1007/978-3-030-94191-8_66
13. Cherradi, M., EL Haddadi, A., & Routaib, H. (2022). Data Lake Management Based on DLDS Approach. Proceedings of Networking, Intelligent Systems and Security, 679‑690. https://doi.org/10.1007/978-981-16-3637-0_48
14. Cherradi, M., & El Haddadi, A. (2022). A Scalable framework for data lakes ingestion. Procedia Computer Science, 215, 2022, 809‑814. https://doi.org/10.1016/j.procs.2022.12.083
15. Cherradi, M., Bouhafer, F., & El Haddadi, A. (2023). Data Lake Governance using IBM-Watson Knowledge Catalog. Scientific African, 21, e01854. https://doi.org/10.1016/j.sciaf.2023.e01854
16. Li, Z., Shang, W., & Yan, M. (2016). News text classification model based on topic model. 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS), 1‑5. https://doi.org/10.1109/ICIS.2016.7550929
17. Jipeng, Q., Zhenyu, Q., Yun, L., Yunhao, Y., & Xindong, W. (2019). Short Text Topic Modeling Techniques, Applications, and Performance : A Survey (arXiv:1904.07695). arXiv. https://doi.org/10.48550/arXiv.1904.07695
18. Amami, M., Faiz, R., Stella, F., & Pasi, G. (2017). A graph based approach to scientific paper recommendation. Proceedings of the International Conference on Web Intelligence, 777‑782. https://doi.org/10.1145/3106426.3106479
19. Younus, A., Qureshi, M. A., Manchanda, P., O’Riordan, C., & Pasi, G. (2014). Utilizing Microblog Data in a Topic Modelling Framework for Scientific Articles’ Recommendation. In L. M. Aiello & D. McFarland (Éds.), Social Informatics (Vol. 8851, p. 384‑395). Springer International Publishing. https://doi.org/10.1007/978-3-319-13734-6_28
20. Yu, K., Zhang, B., Zhu, H., Cao, H., & Tian, J. (2012). Towards Personalized Context-Aware Recommendation by Mining Context Logs through Topic Models (Vol. 7301, p. 443). https://doi.org/10.1007/978-3-642-30217-6_36
21. Dai, T., Zhu, L., Cai, X., Pan, S., & Yuan, S. (2018). Explore semantic topics and author communities for citation recommendation in bipartite bibliographic network. Journal of Ambient Intelligence and Humanized Computing, 9(4), 957‑975. https://doi.org/10.1007/s12652-017-0497-1
22. Uys, W., Du Preez, N., & Uys, E. W. (2008). Leveraging unstructured information using topic modelling (p. 961). https://doi.org/10.1109/PICMET.2008.4599703
23. Vanamala, M., Yuan, X., & Roy, K. (2020). Topic Modeling And Classification Of Common Vulnerabilities And Exposures Database. In International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD) (p. 5). https://doi.org/10.1109/icABCD49160.2020.9183814
24. Costa Silva, C., Galster, M., & Gilson, F. (2021). Topic modeling in software engineering research. Empirical Software Engineering, 26. https://doi.org/10.1007/s10664-021-10026-0
25. Lamba, M., & Margam, M. (2022). Topic Modelling and its Application in Libraries : A review of specialized literature. World Digital Libraries, 15, 105‑120. https://doi.org/10.18329/09757597/2022/15207
26. Barua, A., Thomas, S. W., & Hassan, A. E. (2014). What are developers talking about? An analysis of topics and trends in Stack Overflow. Empirical Software Engineering, 19(3), 619‑654. https://doi.org/10.1007/s10664-012-9231-y
27. Mathkunti, N., & Rangaswamy, S. (2020). Machine Learning Techniques to Identify Dementia. SN Computer Science, 1. https://doi.org/10.1007/s42979-020-0099-4
28. Chen, T.-H., Thomas, S. W., & Hassan, A. E. (2016). A survey on the use of topic models when mining software repositories. Empirical Software Engineering, 21(5), 1843‑1919. https://doi.org/10.1007/s10664-015-9402-8
29. Kunsabo, J., & Dobša, J. (2022). A Systematic Literature Review on Topic Modelling and Sentiment Analysis. In International Scientific Conference Central European Conference on Information and Intelligent Systems.
30. Albalawi, R., Yeap, T., & Benyoucef, M. (2020). Using Topic Modeling Methods for Short-Text Data : A Comparative Analysis. Frontiers in Artificial Intelligence, 3. https://doi.org/10.3389/frai.2020.00042
31. BELLAOUAR, S., BELLAOUAR, M. M., & GHADA, I. E. (2021). Topic Modeling : Comparison of LSA and LDA on Scientific Publications. 2021 4th International Conference on Data Storage and Data Engineering, 59‑64. https://doi.org/10.1145/3456146.3456156
32. Mohammed, S. H., & Al-augby, S. (2020). LSA & LDA topic modeling classification : Comparison study on e-books. Indonesian Journal of Electrical Engineering and Computer Science, 19(1), Article 1. https://doi.org/10.11591/ijeecs.v19.i1.pp353-362
33. Kherwa, P., & Bansal, P. (2021). A Comparative Empirical Evaluation of Topic Modeling Techniques. In D. Gupta, A. Khanna, S. Bhattacharyya, A. E. Hassanien, S. Anand, & A. Jaiswal (Éds.), International Conference on Innovative Computing and Communications (p. 289‑297). Springer. https://doi.org/10.1007/978-981-15-5148-2_26
34. Kalepalli, Y., Tasneem, S., Phani Teja, P. D., & Manne, S. (2020). Effective Comparison of LDA with LSA for Topic Modelling. International Conference on Intelligent Computing and Control Systems (ICICCS), 1245‑1250. https://doi.org/10.1109/ICICCS48265.2020.9120888
35. Maarif, M. R. (2022). Summarizing Online Customer Review using Topic Modeling and Sentiment Analysis. JISKA (Jurnal Informatika Sunan Kalijaga), 7(3), Article 3. https://doi.org/10.14421/jiska.2022.7.3.177-191
36. Daud, S., Ullah, M., Rehman, A., Saba, T., Damaševičius, R., & Sattar, A. (2023). Topic Classification of Online News Articles Using Optimized Machine Learning Models. Computers, 12(1), Article 1. https://doi.org/10.3390/computers12010016
Published
Issue
Section
License
Copyright (c) 2024 Mohamed Cherradi, Anass El Haddadi (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
The article is distributed under the Creative Commons Attribution 4.0 License. Unless otherwise stated, associated published material is distributed under the same licence.
