Breaking Down the Barriers of Technical Documentation Translation

In today's globalized software development landscape, multilingual collaboration has become the norm. However, the challenge of translating technical documentation, particularly defect reports filled with specialized terminology and code snippets, remains a daunting task. A recent study delves into the performance of three leading machine translation systems – DeepL, AWS Translate, and ChatGPT – in translating Visual Studio Code defect reports, seeking to determine which system can most accurately capture the essence of defects while preserving technical details.

As global software development increasingly relies on multilingual collaboration, the importance of accurately translating technical documentation cannot be overstated. Defect reports, with their domain-specific terminology, software version references, and embedded code snippets, often pose significant challenges for general-purpose machine translation systems. Despite significant advances in neural machine translation in recent years, its effectiveness in translating specialized texts like software defect reports remains to be thoroughly explored. This study focuses on Visual Studio Code defect reports labeled "english-please" on the official GitHub repository, which require translation from other languages to English for broader understanding. The accurate translation of these reports is crucial, as even minor errors can obscure the nature and severity of defects.

To evaluate translation quality, the study employs five automated evaluation metrics – BLEU, BERTScore, COMET, METEOR, and ROUGE – to compare the performance of ChatGPT, AWS Translate, and DeepL. The dataset consists of 1,300 multilingual defect reports collected from the Visual Studio Code GitHub repository between March 2019 and June 2024, covering various aspects such as functionality, user interface, and performance. After filtering out invalid issues, focusing on non-English reports, and standardizing code snippets, the dataset provides a solid foundation for assessing the three translation systems.

The study follows a systematic approach to ensure repeatability and reliability, adhering to established machine translation evaluation protocols. The preprocessing stage involves deleting irrelevant metadata and standardizing report structures. Each defect report is accompanied by a reference English translation verified by bilingual reviewers, serving as the evaluation benchmark. The study submits the 1,300 defect reports to the three translation systems, maintaining all formats without providing additional context.

The results show that DeepL excels in most automated evaluation metrics, demonstrating higher accuracy and semantic alignment. AWS Translate scores highest in METEOR, showcasing its strength in handling synonyms. ChatGPT performs the weakest across all metrics. In language identification, each model has its strengths and weaknesses, with performance varying under different evaluation standards. The best choice for multilingual translation tasks depends on the emphasis on precision, recall, or overall accuracy.

In conclusion, DeepL outperforms in most automated evaluation metrics, showcasing its ability to handle structured text and maintain semantic consistency. AWS Translate excels in METEOR, while ChatGPT underperforms. However, domain adaptation remains a key challenge for neural machine translation systems, particularly when handling specialized terminology. The study highlights the importance of domain-specific adaptation when translating defect reports, emphasizing the need for software teams to integrate terminology tables, customize dictionaries, and utilize automatic recognition and preservation of code markers to minimize translation errors.