Readability, Fluency and Error identification when using Machine Translation and AI to translate medical research texts

Raquel Moreira

doi:10.5565/rev/languesparole.148

Lisibilité, fluidité et identification des erreurs lors de l'utilisation de la traduction automatique et de l'IA pour traduire des textes de recherche médicale

Auteurs

Raquel Moreira Universidade do Porto https://orcid.org/0000-0002-8728-985X

PDF (EN)

Résumé

Avec l’amélioration des algorithmes utilisés pour développer des outils de traduction automatique en ligne et l’importance croissante de l’IA, la qualité des résultats de ces outils suscite des inquiétudes croissantes. Par conséquent, nous avons cherché à évaluer la lisibilité des résultats de la traduction automatique et de l’IA pour la traduction de textes de recherche médicale de l’anglais vers le portugais européen. Au total, 16 articles scientifiques ont été collectés et deux traductions de chaque texte à traduire ont été obtenues à l’aide de Google Translate et de ChatGPT (n = 32). Plusieurs formules et indices de lisibilité ont été évalués, ainsi que la fluidité, la fidélité et la fréquence des erreurs linguistiques. Une analyse selon l’outil de traduction sélectionné n’a révélé aucune différence significative entre les résultats issus de Google Translate et de ChatGPT, en ce qui concerne l’analyse quantitative de la lisibilité, ce qui signifie que leurs performances étaient similaires malgré les problèmes perçus de fluidité et de fidélité. Cependant, les erreurs de discours étaient plus fréquentes dans les textes issus de ChatGPT (p < 0,001). Ces résultats sont encourageants et plaident en faveur de l’emploi d’outils de traduction automatique et d’IA pour traduire les textes de recherche médicale. Cependant, la relecture et la post-édition humaines ne peuvent être exclues, car les résultats des algorithmes dépendent largement de la complexité du texte.

Mots-clés

traduction automatique, intelligence artificielle, recherche médicale, lisibilité, qualité de la traduction

Références

ANTUNES, H., LOPES, T. C., Analyzing the Adequacy of Readability Indicators to a Non-English Language, Lecture Notes in Computer Science, 2019, 149-155. https://doi.org/10.1007/978-3-030-28577-7_10

ANTUNES, H. M. M., Automatic Assessment of Health Information Readability. [Master’s dissertation, Universidade do Porto], Repositório Aberto da Universidade do Porto, 2019. https://hdl.handle.net/10216/121810.

CHÉRAGUI, M. A., Theoretical Overview of Machine translation, ICWIT, 2012, 160-169.

COLEMAN, M., LIAU, L. T., A computer readability formula designed for machine scoring, JOURNAL OF APPLIED PSYCHOLOGY, 1975, 60(2), 283-284. https://doi.org/10.1037/h0076540

COLINA, S., MARRONE, N., INGRAM, M., SÁNCHEZ, D., Translation Quality Assessment in Health Research: A Functionalist Alternative to Back-Translation, EVALUATION & THE HEALTH PROFESSIONS, 2017, 40(3), 267-293. https://doi.org/10.1177/0163278716648191

COLLINS-THOMPSON, K., Computational assessment of text readability, RECENT ADVANCES IN AUTOMATIC READABILITY ASSESSMENT AND TEXT SIMPLIFICATION, 2014, 165(2), 97-135. https://doi.org/10.1075/itl.165.2.01col

COSTA, Â., LING, W., LUÍS, T., CORREIA, R., COHEUR, L., A linguistically motivated taxonomy for Machine Translation error analysis, MACHINE TRANSLATION, 2015, 29(2), 127-161. https://doi.org/10.1007/s10590-015-9169-0

COSTA, P. S., XAVIER, V. M. C., MORAIS, R. S., TIBÉRIO, C. S. B., Estudo Empírico do Grau de Compreensibilidade e Legibilidade dos Relatórios de Administração e da Escala de Locus de Controle nos Cursos de Administração e Ciências Contábeis, XXX Encontro Nacional Da Anpad, 2006.

FLETCHER, R. B., DAMERY, S., AIYEGBUSI, L. O., ANDERSON, N., CALVERT, M., COCKWELL, P., FERGUSON, J., HORTON, M., PAAP, S. C. M., SIDEY-GIBBONS, C., SLADE, A., TURNER, N., KYTE, D., Symptom burden and health-related quality of life in chronic kidney disease: A global systematic review and meta-analysis, PLOS MEDICINE, 2022, 19(4), e1003954. https://doi.org/ 10.1371/journal.pmed.1003954

GENCER, A., Readability analysis of ChatGPT’s responses on lung cancer, SCIENTIFIC REPORTS, 2024, 14(1). https://doi.org/10.1038/s41598-024-67293-2

GUNNING, R., The Technique of Clear Writing, New York, McGraw-Hill, 1952.

HENDY, A., MOHAMED, A., AMR, S., VIKAS, R., MOHAMED, G., HITOKAZU, M., JIN, Y. K., MOHAMED, A., HASSAN, H. A.,How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation, 2023. https://doi.org/ 10.48550/arXiv.2302.09210

IKRAMOVNA, G. N., LAZIZOVNA, K. D., The problems of translation and some useful strategies in translating english medical terms into uzbek, SO ‘NGI ILMIY TADQIQOTLAR NAZARIYASI, 2024, 7(4), 25-29.

ISMAYILLI, T., Navigating Complexities in Medical Text Translation: Challenges, Strategies, and Solutions, ACTA GLOBALIS HUMANITATIS ET LINGUARUM, 2024, 1(2), 170-176. https://doi.org/10.69760/aghel.01024080

KAMANDHARI, H. H., The Definitions and the Measurement of Legibility and Readability in Instructional Text Design: An Integrated Literature Review, INFORMATION, MEDIUM, AND SOCIETY: JOURNAL OF PUBLISHING STUDIES, 2020, 18(2), 1-21. https://doi.org/10.18848/2691-1507/CGP/v18i02/1-21

KENNY, D., Human and machine translation, MACHINE TRANSLATION FOR EVERYONE: EMPOWERING USERS IN THE AGE OF ARTIFICIAL INTELLIGENCE, 2022, 18, 23. https://doi.org/10.5281/zenodo.6759976

KINCAID, J. P., Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel, Chief of Naval Technical Training, 1975.

MCDONALD, V. S., Accuracy, readability, and acceptability in translation, Applied Translation, 2020. https://doi.org/10.51708/apptrans.v14n2.1238

OWEN, M. C., ASOPA, S., SMART, A. N., KING, N., Microplegia in cardiac surgery: Systematic review and meta‐analysis, JOURNAL OF CARDIAC SURGERY, 2020, 35(10), 2737-2746. https://doi.org/10.1111/jocs.14895

PEREIRA, S. C., SANTIAGO, M. L., SILVA, R. I., FERREIRA, P., Validation and Cultural Adaptation of the Problem Areas in Diabetes-5 (PAID-5) Scale to European Portuguese, ACTA MED PORT, 2023, 37(1), 36-41. https://doi.org/ 10.20344/amp.18995

RIVERA-TRIGUEROS, I., Machine translation systems and quality assessment: a systematic review, LANGUAGE RESOURCES AND EVALUATION, 2022, 56(2), 593-619. https://doi.org/10.1007/s10579-021-09537-5

SMITH, E. A., SENTER, R. J., Automated Readability Index, Aerospace Medical Research Laboratories, Aerospace Medical Division, Air Force Systems Command, Vol. 66, nº220, 1967.

SMITH, M., TAFFLER, R., Readability and Understandability: Different Measures of the Textual Complexity of Accounting Narrative, ACCOUNTING, AUDITING & ACCOUNTABILITY JOURNAL, 1992, 5(4). https://doi.org/10.1108/09513579210019549

SUSANTHI, I. G. A. A. D., Translation on Medical Terms into Indonesian, International JOURNAL OF ARTS AND SOCIAL SCIENCE, 2023, 6(1), 22-29.

Source Corpus References

BRANCH, R. H. K., GATEWOOD, O. M., KUDENCHUK, J. P., MAYNARD, C., SAYRE, R. M., CARLBOM, J. D., EDWARDS, M. R., COUNTS, R. C., PROBSTFIELD, L. J., BRUSEN, R., JOHNSON, N., GUNN, L. M., Diagnostic yield, safety, and outcomes of Head-to-pelvis sudden death CT imaging in post arrest care: The CT FIRST cohort study, RESUSCITATION, 2023, 188, 109785. https://doi.org/10.1016/j.resuscitation.2023.109785

BURNS, C., YEATES, L., SWEETING, J., SEMSARIAN, C., INGLES, J., Evaluating a communication aid for return of genetic results in families with hypertrophic cardiomyopathy: A randomized controlled trial, JOURNAL OF GENETIC COUNSELLING, 2023, 32(2), 425-434. https://doi.org/10.1002/jgc4.1651

DAVEY, G. M., TEMPERLEY, C. H., O’SULLIVAN, J. N., MARCELINO, V., RYAN, K. O., RYAN, J. É., DONLON, E. N., JOHNSTON, M. S., ROBB, B. W., Minimally Invasive and Open Gastrectomy for Gastric Cancer: A Systematic Review and Network Meta-Analysis of Randomized Clinical Trials, ANNALS OF SURGICAL ONCOLOGY, 2023, 30(9), 5544-5557. https://doi.org/10.1245/s10434-023-13654-6

FLETCHER, R. B., DAMERY, S., AIYEGBUSI, L. O., ANDERSON, N., CALVERT, M., COCKWELL, P., FERGUSON, J., HORTON, M., PAAP, S. C. M., SIDEY-GIBBONS, C., SLADE, A., TURNER, N., KYTE, D., Symptom burden and health-related quality of life in chronic kidney disease: A global systematic review and meta-analysis, PLOS MEDICINE, 2022, 19(4), e1003954. https://doi.org /10.1371/journal.pmed.1003954

HALL, R. C., RAMASESHAN, R., REID, A., JONES, D. A., MATHUR, A., Case report: Cytokine therapy and an intracoronary autologous bone marrow-derived cell infusion with Impella support in a patient with dilated cardiomyopathy and a severely reduced ejection fraction, FRONT CARDIOVASC MED, 2022, 9, 1002508. https://doi.org/10.3389/fcvm.2022.1002508

HAYES, N. W., LAING, E., BROWN, R., SILSBY, L., SMITH, L., THOMAS, H., KALOYIROU, F., SHARMA, R., GRIFFITHS, J., HUME-SMITH, H., MARKS, D. S., KESSARIS, N., CHRISTIAN, M., DUDLEY, J., SHENOY, M., MALINA, M., MUORAH, M., WARE, N., YADAV, P., PETERS, J. M., A pragmatic, open-label, randomized controlled trial of Plasma-Lyte-148 versus standard intravenous fluids in children receiving kidney transplants (PLUTO), KIDNEY INTERNATIONAL, 2024, 105(2), 364-375. https://doi.org/10.1016/j.kint.2023.09.032

HORWOOD, G. A. C., PATEL, D. N., WALKER, D. J., EVANS, R. C., Nonanemic Iron Deficiency in Cardiac Surgery: A Retrospective Observational Study, JOURNAL OF CARDIOTHORACIC AND VASCULAR ANESTHESIA, 2024, 38(9), 1899-1906. https://doi.org/10.1053/j.jvca.2024.05.039

LAW, M. Y., HSU, C., HINGORANI, R. S., RICHARDS, M., MCMULLAN, M. D., JEFFERIES, H., HIMMELFARB, J., KATZ, R., Randomized controlled trial of remote ischemic preconditioning in children having cardiac surgery, JOURNAL OF CARDIOTHORACIC SURGERY, 2024, 19(1), 5. https://doi.org/10.1186/s13019-023-02450-8

LAZARUS, B., POLKINGHORNE, K. R., GALLAGHER, M., COGGAN, S., GRAY, N. A., TALAULIKAR, G., KOTWAL, S., INVESTIGATORS, R., Tunneled hemodialysis catheter tip design and risk of catheter dysfunction: an Australian nationwide cohort study, AMERICAN JOURNAL OF KIDNEY DISEASES, 2024, 83(4), 445-455. https://doi.org/10.1053/j.ajkd.2023.09.021

MCCLAINE, S., FEDOR, J., BARTEL, C., CHEN, L., DURICA, K. C., LOW, C. A., Engagement with daily symptom reporting, passive smartphone sensing, and wearable device data collection during chemotherapy: longitudinal observational study, JMIR cancer, 2024, 10(1), e57347. https://doi.org/10.2196/57347

NUTTING, C., FINNERAN, L., ROE, J., SYDENHAM, A. M., BEASLEY, M., BHIDE, S., BOON, C., COOK, A., WINTON, D. E., EMSON, M., FORAN, B., FROGLEY, R., PETKAR, I., PETTIT, L., ROONEY, K., ROQUES, T., SRINIVASAN, D., TYLER, J., HALL, E., MACGREGOR, C.. Dysphagia-optimised intensity-modulated radiotherapy versus standard intensity-modulated radiotherapy in patients with head and neck cancer (DARS): a phase 3, multicentre, randomised, controlled trial, THE LANCET ONCOLOGY, 2023, 24(8), 868-880. https://doi.org/10.1016/S1470-2045(23)00265-6

OWEN, M. C., ASOPA, S., SMART, A. N., KING, N., Microplegia in cardiac surgery: Systematic review and meta‐analysis, JOURNAL OF CARDIAC SURGERY, 2020, 35(10), 2737-2746. https://doi.org/10.1111/jocs.14895

SWEENEY, A. C., QUADER, M., KIM, C., Thrombotic Microangiopathic Anemia After Cardiac Surgery, JOURNAL OF CARDIOTHORACIC AND VASCULAR ANESTHESIA, 2024, 38(3), 784-787. https://doi.org/10.1053/j.jvca.2023.11.033

POWERS, M. C., VERMA, H., ORLOFF, J., PIONTKOWSKI, J. A., TIERSTEN, A., LAMB, A., GULATI, N., Use of a topical Janus kinase inhibitor in immune checkpoint inhibitor-induced eczematous reaction: a case report, JOURNAL OF DERMATOLOGICAL TREATMENT, 2024, 35(1), 2336118. https://doi.org /10.1080 /09546634.2024.2336118

THOMAS, A. D., TRAINOR, R., SHEINGOLD, Z., SAMARNEH, M., A Case of COVID-19-Associated Focal Segmental Glomerulosclerosis, CUREUS, 2023, 15(4). https://doi.org/10.7759/cureus.37547

WINCHESTER, D. E., MERRITT, J., WAHEED, N., NORTON, H., MANJA, V., SHAH, N. R., HELFRICH, C. D., Implementation of appropriate use criteria for cardiology tests and procedures: a systematic review and meta-analysis, EUROPEAN HEART JOURNAL-QUALITY OF CARE AND CLINICAL OUTCOMES, 2021, 7(1), 34-41. https://doi.org/10.1093/ehjqcco/qcaa029

Biographie de l'auteur

Raquel Moreira, Universidade do Porto

Raquel Moreira est étudiante en master d'arts en traduction et services linguistiques à la Faculté des Lettres et des Sciences humaines de l'Université de Porto, au Portugal. Son parcours universitaire comprend un diplôme en physiologie cardiopulmonaire et un master en physiopathologie cardiovasculaire. Elle a travaillé dans le cadre d'un programme de bourse à la Faculté de Médecine de l'Université de Porto et poursuit actuellement ses recherches, dans le cadre de son nouveau master, dans le domaine de la traduction médicale, en raison de son intérêt pour ces domaines scientifiques interdisciplinaires.