Evaluation of Instagram's Neural Machine Translation for Literary Texts: An MQM-Based Analysis

Altaf Fakih, Mozhgan Ghassemiazghandi, Abdul Hafeed Fakih, Manjet K. M. Singh

Abstract


Addressing the global increase in social media users, platforms such as Instagram introduced automatic translation to broaden information dissemination and improve cross-cultural communication. Yet, the accuracy of these platforms' machine translation systems is still a concern. Therefore, this paper aims to explore the potential of Neural Machine Translation utilized by Instagram in producing high-quality translations. In doing so, this study attempts to scrutinize the reliability of Instagram's "See Translation" feature in the translation of literary texts from Arabic to English. A selection of auto-translated Instagram captions is analyzed through the identification, classification, and assignment of error types and penalty points, utilizing the MQM core typology. Subsequently, the Overall Quality Score of the error-based analysis is calculated automatically using the ContentQuo platform. Furthermore, the study investigates whether Instagram Neural Machine Translation can effectively convey the intended message within literary texts. From 30 purposively selected Instagram captions with literary content, the study found Instagram's machine translation lacking in 90% of cases, particularly in accuracy, fluency, and style. Among these, 61 errors were identified: 26 in fluency, 25 in accuracy, and 10 in style, adversely affecting the quality and failing to convey the original message. The findings suggest a need for enhanced algorithms and linguistic architecture in Neural Machine Translation systems to better recognize linguistic variants and text genres for more accurate and fluent translations.


Keywords


Literary Text Translation; Multidimensional Quality Metrics; Neural Machine Translation; Translation Quality Assessment

Full Text:

PDF

References


Alkhatib, M., & Shaalan, K. (2018). The key challenges for Arabic machine translation. Intelligent Natural Language Processing: Trends and Applications, 139-156.

Ameur, M. S. H., Meziane, F., & Guessoum, A. (2020). Arabic machine translation: A survey of the latest trends and challenges. Computer Science Review, 38, 100305. https://doi.org/10.1016/j.cosrev.2020.100305.

Arnold, D., Balkan, L., Meijer, S., Humphreys, R., & Sadler, L. (1994). Machine translation: An introductory guide. London: Blackwell.

Arvianti, G. F. (2018). Human translation versus machine translation of Instagram’s captions: Who is the best? In English Language and Literature International Conference (ELLiC) Proceedings (Vol.2, pp.531-536).‏

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

Bentivogli, L., Bisazza, A., Cettolo, M., & Federico, M. (2016). Neural versus phrase-based machine translation quality: a case study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 257–267). arXiv preprint arXiv:1608.04631.

Bhattacharyya, S. (2022). Meta's machine translation journey. Analytics India Magazine. Retrieved March22,2023, from https://analyticsindiamag.com/metas-machine-translation journey/#text=Meta%20used%20neural%20machine%20translation,training%20time%20from%2024%20hours.

Bounhas, I., & Slimani, Y. (2009). A hybrid approach for Arabic multi-word term extraction. In 2009 International Conference on Natural Language Processing and Knowledge Engineering (pp. 1-8). IEEE.

Cahyani, N. L. D. (2022). Derivational affixes found in the caption of selected posts of @bawabali_official account on Instagram (Doctoral dissertation, Universitas Mahasaraswati Denpasar).

Castilho, S., Moorkens, J., Gaspari, F., Calixto, I., Tinsley, J., & Way, A. (2017a). Is neural machine translation the new state of the art? The Prague Bulletin of Mathematical Linguistics, 108, 109–120.

Castilho, S., Moorkens, J., Gaspari, F., Sennrich, R., Sosoni, V., Georgakopoulou, P., ... & Gialama, M. (2017b). A comparative quality evaluation of PBSMT and NMT using professional translators. In Proceedings of Machine Translation Summit XVI: Research Track (pp. 116-131).

Chatzikoumi, E. (2020). How to evaluate machine translation: A review of automated and human metrics. Natural Language Engineering, 26(2),137-161.‏

Chéragui, M. A. (2012). Theoretical overview of machine translation. ICWIT, 160-169.

Das, A. K. (2018). Translation and artificial intelligence: Where are we heading. International Journal of Translation, 30(1),72-101.

Cho, K., Van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8). arXiv preprint arXiv:1409.1259.

Dorr B., Snover M. & Madnani N. (2011). Chapter 5.1 introduction. In Olive J., McCary J. and Christianson C. (eds), Handbook of Natural Language Processing and Machine Translation. DARPA Global Autonomous Language Exploitation. New York: Springer, (pp. 801–803).

España-Bonet, C., & Costa-jussà, M. R. (2016). Hybrid machine translation overview. Hybrid Approaches to Machine Translation, 1-24.

Fadilah, E. (2017). Semantic Errors Analysis of Instagram Machine Translation from Indonesian to English. Published thesis, Syarif Hidayatullah State Islamic University of Jakarta, Indonesia.

Habash, N., Dorr, B., & Monz, C. (2009). Symbolic-to-statistical hybridization: extending generation-heavy machine translation. Machine Translation, 23, 23-63.

Hunsicker, S., Chen, Y., & Federmann, C. (2012, June). Machine learning for hybrid machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, 312-316.

Hutchins, W. J. (1995). Machine translation: A brief history. In Concise History of the Language Sciences (pp. 431-445). Pergamon.

Hutchins, J. (2007). Machine translation: A concise history. Computer Aided Translation: Theory and Practice, 13(29-70), 11.

Koby G.S., Fields P., Hague D., Lommel A. & Melby A. (2014). Defining translation quality. Tradumàtica 12,413–420.

Koehn, P. (2009). Statistical machine translation. Cambridge University Press.

Larassati, A., Setyaningsih, N., Nugroho, R. A., Suryaningtyas, V. W., Cahyono, S. P., & Pamelasari, S. D. (2019). Google vs. Instagram machine translation: multilingual application program interface errors in translating procedure text genre. In 2019 International Seminar on Application for Technology of Information and

Communication (iSemantic) (pp. 554-558). IEEE.‏

Lommel, A., & Melby, A. K. (2018). Tutorial: MQM-DQF: A good marriage (Translation quality for the 21st Century). In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 2: User Track).

Mawarni, B., Pambudi, B. D., & Ghasani, B. I. (2017). The problem of cultural untranslatability found in the English translation of Jokowi’s Instagram posts. In UNNES International Conference on ELTLT (pp. 104-108).

Meilasari, P. (2019). When Instagram translation machine translates ecology terms: Accurate or not? In The 7th Library Studied Conference (p. 129).

Melby, A. K. (2002). The mentions of equivalence in translation. Meta, 35(1),207-213.

Meta AI. (2022). 200 languages within a single AI model: A breakthrough in high-quality machine translation. Meta AI. Retrieved March 22, 2023, from https://ai.facebook.com/blog/nllb-200-high-quality-machine-translation/

Mannes, J. (2017, August 3). Facebook finishes its move to neural machine Translationt. TechCrunch. Retrieved March 22, 2023, from https://techcrunch.com/2017/08/03/facebook-finishes-its-move-to-neural-machine-translation/?guccounter=1

Moorkens, J., Toral, A., Castilho, S., & Way, A. (2018). Translators’ perceptions of literary post-editing using statistical and neural machine translation. Translation Spaces, 7(2),240-262.

Nord, C. (2005). Text analysis in translation: Theory, methodology, and didactic application of a model for translation-oriented text analysis. New York: Rodopi.

Nord, C. (1997). Functionalist approaches explained. Manchester, UK: St. Jerome Publishing.

Omar, N., & Al-Tashi, Q. (2018). Arabic nested noun compound extraction based on linguistic features and statistical measures. GEMA Online® Journal of Language Studies, 18(2).

Omar, A., & Gomaa, Y. (2020). The machine translation of literature: Implications for translation pedagogy. International Journal of Emerging Technologies in Learning (IJET), 15(11),228-235.

Pino, J. M., Sidorov, A., & Ayan, N. F. (2017). Transitioning to neural machine translation. Tech at Meta. Retrieved March 22, 2023, from https://tech.facebook.com/artificial-intelligence/2017/8/transitioning-entirely-to-neural-machine-translation

Pujakesuma, G. A. (2022). The Performance of Instagram's auto-translate and google translate in translating house of highlight's Instagram captions [Thesis, Yogyakarta:Sanata Dharma University]. http://repository.usd.ac.id/id/eprint/42510

Purwaningsih, D. R., Sholikhah, I. M., & Wardani, E. (2019). Revealing translation techniques applied in the translation of Batik Motif names in see Instagram. Celt: A Journal of Culture, English Language Teaching & Literature, 19(2), 287-301.‏

Ragni, V., & Nunes Vieira, L. (2022). What has changed with neural machine translation? A critical review of human factors. Perspectives, 30(1),137-158.

Sabtan, Y. M. N., Hussein, M. S. M., Ethelb, H., & Omar, A. (2021). An evaluation of the accuracy of the machine translation systems of social media language. International Journal of Advanced Computer Science and Applications, 12(7), 406-415. DOI: 10.14569/IJACSA.2021.0120746

S. Dixon. (2023). Instagram users worldwide 2025. Statista. Retrieved March 22, 2023, from https://www.statista.com/statistics/183585/instagram-number-of-global-users/

Sipayung, K. T., Sianturi, N. M., Arta, I. M. D., Rohayati, Y., & Indah, D. (2021). Comparison of Translation Techniques by Google Translate and U-Dictionary: How Differently Does Both Machine Translation Tools Perform in Translating? Elsya: Journal of English Language Studies, 3(3),236-245.

Susanti, E. (2018). Lexical errors produced by Instagram machine translation [Doctoral dissertation, Universitas Islam Negeri Maulana Malik Ibrahim].‏

Stymne, S., & Ahrenberg, L. (2012, May). On the practice of error analysis for machine translation evaluation. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) (pp. 1785-1790).‏

Tambouratzis, G. (2014, April). Comparing CRF and template-matching in phrasing tasks within a Hybrid MT system. In Proceedings of the 3rd Workshop on Hybrid Approaches to Machine Translation (HyTra) (pp. 7-14).

Thurmair, G. (2009). Comparing different architectures of hybrid machine translation systems. In Proceedings of Machine Translation Summit XII: Posters.

Widiastuti, N. M. A. (2021). Translation procedures of English phrasal verbs into Indonesian on Instagram captions. In International Seminar on Austronesian Languages and Literature IX (399-408). Denpasar: Udayana University.

Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., & Dean, J. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.




DOI: http://dx.doi.org/10.17576/gema-2024-2401-13

Refbacks

  • There are currently no refbacks.


 

 

 

eISSN : 2550-2131

ISSN : 1675-8021