Towards a Combination of Metrics for Machine Translation

Abstract

In this scholar, we compare three metrics for machine translation, from English to French and vice versa, and we give some combination formulas based on some schemes, algorithms, and machine learning tools. As an experimental dataset, we consider 10 English and French theses abstracts published in the web with four free in charge machine translation systems. Five combinations, with the same implicit weights, are considered namely: (BLEU+NIST), (BLEU+ (1-WER)), (NIST+(1-WER)), (BLEU+NIST+(1-WER)), and (FR(BLEU)+FR(NIST)+FR(WER)). These combinations are also considered differently through generating weights parameters on the basis of regression. The results of 12 formulas are computed and compared then in total. According to the obtained results, average regression combinations based on machine learning step are the best, especially with the three basic metrics, followed by average WER metric in the case of English to French. For French to English, (FR(BLEU)+FR(NIST)+FR(WER)) combination is the best followed respectively by the average regression combination with both first parameters (Reg(α,β)) and average BLEU basic metric. Another performance criterion is considered here, in the second position, namely: the number of times, over the 10 abstracts, where the formula is the best. Based on the obtained results, combination with regression based on the first and the last parameters (Reg(α,γ)) outperforms the others, in the case of English to French, with 3 times followed by Reg(β,γ), Reg(α,β,γ), NIST+(1-WER), and the basic metrics (BLEU, NIST, and WER) with 2 times for each of them. For French to English, the basic WER metric outperforms the others with three times followed by BLEU, (BLEU+ (1-WER)), (FR(BLEU)+FR(NIST)+FR(WER)), and Reg(α,γ) with 2 times for each of them. To note that there is a room of improvement for the combinations with1.0914 in the case of English to French and 1.01 in the case of French to English.

Keywords

• Machine Translation
• Machine Translation Metrics
• Combination of Machine Translation Metrics