Post-editing for Professional Translators: Cheer or Fear?

Currently, post-editing of machine translation (MT) has been introduced as a regular practice in the translation workflow, especially since the good results in quality obtained by neural MT (NMT). This fact is linked to the efforts LSPs and customers have done to reduce costs due to the recent global crisis and the increasing globalization, which has had a negative impact on translators' revenues and on their working practices. In this context, post-editing is often perceived with a negative bias by translators. We study attitudes of translators post-editing for the first time and relate them to their productivity rates. We also compare the results with a survey answered by professional post-editors assessing their perception of the task in the current marketplace.


Introduction
Machine translation (MT) has been used as part of the translation workflow since the 1980s. However, in the last years, the good results obtained by neural machine translation (NMT) in terms of quality (Bojar et al. 2018) have attracted the attention of both research and industry. These promising results have driven a technological shift from (phrase-based) statistical machine translation (SMT) to neural machine translation (NMT) in many translation industry scenarios. As part of this change, machine translation post-editing (PE) has increased its presence in the translation workflow and half of the LSPs offer it as a service (Lommel and Depalma 2016).
In the last decade, the translation marketplace has suffered important changes that have also affected the translation profession. Both the rapid globalization propelled by the neoliberal policies and the recent global economic crisis have increased the effort from LSPs and customers to reduce costs in the translation workflow, which has had an important impact both on translators' revenues and working practices (Moorkens 2017). Furthermore, the increased technologisation means that many translators need to include different working routines into their profession, which in some cases may limit the scope of their work (Vieira and Alonso 2018).
The current study has two objectives: firstly, to assess the attitude of translators who are post-editing MT output for the first time before and after the task and to relate it to productivity rates in an industrial scenario. Secondly, to compare the results with the perceptions of current professional post-editors.
To achieve our objectives, we set four professional translators to the task of postediting and translating from scratch from Spanish into English general domain texts.
Translators answered questions regarding their perceptions and attitude before and after post-editing, and we recorded the number of keystrokes and the time spent while performing the tasks. Additionally, 50 participants answered an online survey we conducted addressed to professional post-editors regarding their perception of the job.
The paper is structured as follows. First, we review previous research on post-editing and translators' perceptions regarding machine translation. Then, we specify the Sergi Alvarez Vidal Antoni Oliver Toni Badia Post-editing for Professional Translators: Cheer or Fear? Revista Tradumàtica 2020, Núm. 18 51 51 methodology followed in this paper. In the next section, we detail the post-editing task carried out by translators. It includes the description of the LSP, the questionnaires the four translators answered before and after post-editing for the first time and the report on productivity, measured as technical and temporal effort. We also study the results from an online survey answered by 50 post-editors regarding their current perception of post-editing. Finally, we discuss the results.

Related Work
Post-editors "edit, modify and/or correct pre-translated text that has been processed by an MT system from a source language into (a) target language(s)" (Allen 2003, 296). In the best scenario, MT output only requires some small modifications, but sometimes post-editors delete the remaining segments and translate everything from scratch if they consider it will take them less time (Parra-Escartín and Acedillo 2015).
In the last years, a lot of empirical investigations have been conducted to analyze post-editing because it is increasing its presence in the translation workflow (Lommel and Depalma 2016). This is due to the good results obtained by NMT in terms of quality (Bojar et al. 2018) and the need to reduce costs (Guerberof 2009a;Sosoni and Rogers 2013) and shorten time cycles. Results consistently show that post-editing is faster than translating from scratch (O'Brien 2005; Plitt and Masselot 2010), although for general language texts some studies see no significant improvement in speed (Screen 2017). However, post-editing productivity means "not only the ratio of quantity and quality to time but also the cognitive effort expended; and the higher the effort, the lower the productivity" (O'Brien 2011, 198).
There are several works studying post-editing effort (Specia 2011, Koponen 2016, Jia et al. 2019), all of which use the division established by Krings (2001): temporal effort (time spent post-editing), technical effort (number of edits, often measured using keystroke analysis), and cognitive effort (usually measured with eye-tracking or thinkaloud protocols). Research shows cognitive effort correlates with technical and temporal PE effort (Moorkens et al. 2015). Additionally, records of gaze data reveal that the reading time of the source text and target text is very different in post-editing compared to from-scratch translation (Carl et al. 2015;Daems et al. 2017).
More than two decades ago, translation memories (TMs) were first introduced in the translation arena and have since been adopted by the majority of translators (Christensen and Schjoldager 2016;Doherty 2016). Currently, very few translators ignore tools such as term banks, translation memory systems and quality checkers in the daily translation tasks. However, studies show that translators still regard the use of technology mainly as a threat (Katan 2011). LeBlanc (2013) reported on translators' perception of TMs and described the main advantages and disadvantages. Although translators admitted it helped increase productivity and reduced repetitive work, their main concern was that it was a barrier for creativity and made translators increasingly passive and lazy. Regarding the use of MT, research shows that translators perceive they are less productive post-editing, even when a quantitative analysis shows otherwise (Gaspari et al. 2014). They consider MT output to be tedious to post-edit (Moorkens and O'Brien 2017) and they prefer to translate from scratch even if this has a negative impact on productivity (Teixeira 2014).

Methodology
Our aim was to study translators' perception when post-editing for the first time and compare the results to their productivity measured as technical and temporal effort. To this end, we worked with an LSP called Incyta. Four experienced translators without previous post-editing experience who usually collaborated with the LSP participated in the experiment translating from Spanish into English. First of all, translators answered a short questionnaire before post-editing to assess their attitudes towards the task. Then, we selected three general domain documents with similar terminology and type-token ratio. They translated from scratch a document of 2437 words. Then, they post-edited two documents of 2189 and 1920 words, respectively, that had been translated using DeepL. 1 In both tasks, they were asked to produce printable-quality translations.
Instead of trying to reproduce the working conditions of translators, which could vary greatly among individuals, we used PET (Aziz et al  before beginning the task, we sent them the tool and delivered detailed information on its use, together with a short text they could use to test it. We used this testing period to answer questions and solve usability issues so that all translators could have a fair knowledge of the tool before beginning the task.
Once the task was finished, translators answered another questionnaire with questions about their perception of the task and, in some cases, they were asked to answer follow-up questions by mail to clarify some of the answers.
Additionally, we prepared a larger survey targeting experienced post-editors to study their perception of the task and to compare it with the opinions expressed by translators post-editing for the first time. For the survey, we used the web-based SurveyMonkey platform because it allows to create online surveys that can be easily distributed and also allows to analyze and summarize all data collected in different formats. We published a 33-question survey targeted exclusively to translators who had already worked as post-editors. We tested the wording of the questions in a pilot study with two professional translators to ensure that there was no ambiguity, so that all answers could provide enough data for a complete analysis. Once the survey was ready, we published it on Linkedin and sent it to three translation associations. 50 participants answered the survey.

Post-editing task Language Service Provider
Incyta is a Barcelona-based LSP founded in 1993. It is also the provider of Lucy Software 2 , a commercial rule-based machine translation engine. It has previous experience in post-editing and it is currently working with the Spanish-Catalan, Spanish-Galician and Spanish-Portuguese language combinations, mainly to translate news on a daily basis for newspapers. To post-edit these language combinations, they use their own commercial MT software.
The company was interested in introducing post-editing for some new language combinations because of the increase in the demand and it was planning to implement Spanish-English post-editing for some news workflows in 2021. After the quality assessment it conducted internally, for this new language combination it was going to use a NMT engine. The translators with whom the company usually worked were quite skeptical about the quality of the MT output and the economic repercussions in their earnings and weren't willing to begin post-editing. On the other hand, the LSP wanted to continue the current collaboration with these translators due to the good quality of their work. For post-editing tasks, the company envisaged to pay translators 70% of their current rate based on their previous experience working with closely-related languages, although they were paid their regular rate for this experiment. The four professional translators (T1, T2, T3 and T4) who carried out the task had extensive experience translating (12, 16, 15 and 18 years, respectively) (Q1) and had worked more than three years with this LSP translating from Spanish into English (Q2).
T1 and T2 had never used CAT tools before, while T3 and T4 used them only sometimes for certain specific projects (Q4). We also asked them about their previous experience correcting human outputs, as research has shown translators often relate it to post-editing (Guerberof 2013). T1 and T4 had done corrections of human translations before, but T2 and T3 did only translations (Q3). Although none of them were regular consumers of MT in their daily lives, they believed the general quality of MT had improved considerably in recent years (Q5).
Regarding their attitude towards post-editing (Q6), none of the translators were looking forward to it. They recognised MT was "getting better, but still cannot compare to a (decent) human translation" (T2). T4 showed concerns MT will take over the industry and T3 thought it would take as much time as translating from scratch and "can influence my own translation." When asked about their expectations (Q7), T1 thought it would be like reviewing translations by non-natives, where "sentence structure and context acquire special importance". T2 thought it was not going to be an enjoyable job and she also had "ethical conflicts with my profession disappearing and only becoming post-editing, which is more poorly paid and frankly less fun and creative". Only T3 highlighted the quality of the MT output as a key factor. If the quality is good, it will be a "positive experience". Q8 and Q9 offered the participants a Likert scale where 1 was "Very bad" and 5 was "Excellent". The majority of the translators thought the quality of the MT output was going to be very good (4), but their experience post-editing was only going to be good (3). In the last question (Q9), from the three options offered as answers, T1 and T3 thought post-editing would be similar to reviewing human translations and T2 and T4 believed it would be like translating with fuzzy matches. None of them believed post-editing would be similar to translating from scratch.
Although their general attitudes regarding post-editing were mainly negative, they didn't think the post-editing experience was going to be bad. In fact, some of the fears expressed were related to rates and other market practices.
Perceptions after Post-editing was "Very bad" and 5 was "Excellent". T1 and T3 thought the experience had been good (3), T2 thought it had been bad (2) and T4 qualified it as excellent (5). Except T2, who was quite disappointed with the experience, the other three translators would be willing to post-edit in a regular basis (Q2), but only if the "rate was right". T1 stressed "there is nothing enjoyable (to me) about post-editing, whereas translating is enjoyable," even though they recognised the improved quality of the MT output. As positive elements (Q3) they thought post-editing saved them time typing and they didn't have to correct basic mistakes. On the downsides (Q4), they thought it "constrained creativity when reformulating sentences" and it was "total roteness" because of the lack of creativity. These opinions coincide mostly with the ones expressed in previous studies regarding the adoption of MT by translators ).
Three of the translators believed that a proper training would improve their performance post-editing (Q5) and two of them thought it would be positive to have information on the MT engine (Q6). Regarding the MT quality (Q7), the mean rating was 3.75 out of 5, even though they found some important errors (Q8) while postediting. T1 and T3 highlighted the high number of grammatical errors, while T2 thought the main errors were "too-literal translation of the sentences" and missing nuances. T4 also detected some inconsistencies (pliego had been translated both as document and specifications; sobre had been translated as about and envelope) and some words that had been badly translated (unión temporal had been translated as temporary union instead of joint venture; garantía had been translated as security instead of bid bond).
All four translators agreed that post-editing was not similar to revising human translations (Q9) because the errors were of "a different nature". They also explained (Q10) that some errors were hard to spot because there was a lack of uniformity.
There were not usual errors, such as "typos and spelling mistakes and I had to pay All four translators agreed productivity was higher when post-editing (Q11) but that this was not the only important factor. They stated a lower degree of satisfaction ("I become a 5th-grade teacher correcting essays, and that is not the profession I signed up for!"). They also agreed their final product was of similar quality as if they had translated it from scratch (Q12). They were proud of the results (Q13) after all the corrections had been introduced.
In general, their experience was better than they had expected. They found the MT quality to be good enough, although mistakes were sometimes difficult to spot.

Productivity results
We analyzed the technical and temporal effort collected during the translation with PET to calculate the productivity differences between post-editing and translating from scratch. As it can be seen in Table 1, although there is a great variability among translators, the mean shows there is an increase of 53.14% in productivity in words per hour if we compare the translation from scratch and the post-editing task. This increase ranges fom 33.29% in the case of T1 to a 81.01% in the case of T3. If we consider exclusively the productivity figures, the rate reduction of 30% suggested by the LSP could be considered in line with these results.

Survey for post-editors
In our study, translators who post-edited for the first time showed in general a negative perception of the task even though there was an increase in their productivity.
We  To all of those who had received some sort of training, we asked them to state which one and give their opinion about the quality of the training (Q5). Most of them explained they were only given instructions about the post-editing process and the final quality required (52%). Some of them received training by the LSP they were working for (17%) when they first began post-editing, and the remaining translators (31%) followed a more formal course, such as the ones developed by SDL or TAUS. All participants who followed these courses found them helpful. All respondents who had not followed any training agreed it would have been a great help when they began post-editing (Q6).  We also asked about the type of texts they post-edited (Q11), which were usually technical or medical, domains in which post-editing has traditionally achieved better results (Aymerich 2005;Kirchoff 2011). Very few translators (5%) post-edited general domain documents.  higher efficiency (Moorkens and O'Brien 2014), even though some participants stressed the great variation of errors found in the MT outputs. However, regarding the effort it entailed in relation to translating (Q14), 50% considered it required more effort than translating using translation memories and 22% believed it was a task that entailed more effort than revising human translations. In order to obtain a better picture of their current working situation, we asked about post-editing rates. As we can see in Figure 8, participants in the survey mostly believed that at the end of a day in which they only post-edited, they earned less money (52%) or the same amount of money (36%) than if they had been translating (Q15). This fact can be linked to the effort LSPs and customers have done to reduce costs since the 2008 crisis, which has negatively affected translating rates (Moorkens 2017). Regarding the quality of the MT they had to post-edit (Q16), most of them thought it was acceptable though it needed many editions (44%), or they even believed that it was of borderline quality (22%). Another 22% of the respondents even erased the whole MT output and translated from scratch in certain segments throughout the task although they were only paid for post-editing. It is well known that the quality of the MT output is a key element in post-editing as it affects the productivity gain (Garcia 2011).  The majority of participants clearly stated that post-editing rates were not currently adequate to the effort it entailed (66%) (Q17) because it was more tiring than translating (38.78%) and than revising human translations (14.29%) (Q18). Regarding the estimation method to calculate the rate they were paid (Q19), nearly half of them preferred being paid by word (44%) although 26% felt comfortable with both payment methods. Regarding the tools they used, all translators who answered the survey used the same tool to translate (Q20) and to post-edit (Q21). They mainly used SDL Trados Studio (55%), followed by MemoQ (16%) (Q22). As it can be seen in Figure 9, most of them explained the tool they were currently using was the best suited for post-editing (74%) (Q23). However, Moorkens and O'Brien (2017) concluded after an extensive survey that currently post-editing was not well-supported by existing tools and there was a need to study new specifications for user interfaces (UIs) that better supported the post-editing task. To get the participants insight, we asked them if they would like to add any additional functionality for post-editing to the tool they were currently using (Q24). When asked about any additional element they would incorporate to the UI (Q25), participants mainly suggested adding more shortcuts for tag insertions, providing automatic corrections and including tools to help rearrange words in a sentence.
Another important element when post-editing is the MT system used to produce the output. Post-editors who answered the survey did not usually (38%) or never (26%) receive any information regarding the MT system (Q26). However, this could be a useful piece of information as recent research has shown that different MT models produce different types of errors (Klubicka et al. 2018). In fact, most respondents (67%) believed that if translators were trained on understanding how MT works they would feel more confident post-editing (Q27).
In the following questions, we enquired participants to rate their satisfaction level with the translation tasks (Q28) and post-editing tasks (Q29) giving a mark from 0 ("Very bad") to 100 ("Excellent"). For translation, the mean rating was 83, while postediting obtained a 56. Additionally, we asked them to explain the main reasons for the previous rating of translating (Q30) and post-editing (Q31). They thought translating boosted creativity and gave translators the chance to work with different text types.
However, respondents showed more concerns about post-editing. They found it was more boring and repetitive. They believed that having to correct computer-generated errors tended to be tedious, as they usually "have to correct as little as possible to be profitable, so we do not aim for the best quality". One respondent even suggested that "the hardest is to remember what is genuine in the language." Finally, we asked them if they thought their voices were heard in industrial workflows (Q32) and what suggestions would they make to improve this workflow (Q33). Except in two cases, they all thought their voices were never listened to, which correlates to previous surveys analyzing the current working conditions of professional translators (Vieira and Alonso 2018). They mainly stressed the importance of increasing post-editing rates, which were considered low, and improving the quality of the MT output. Post-editors also made reference to the tight deadlines in the current translation marketplace and the possibility of correcting formal or repetitive mistakes in the text before post-editing began. Translators who post-edited for the first time showed prejudices and a general negative attitude before post-editing. In part it was due to the characteristics of the task but also because of other external elements such as rates and the future of the translation profession. However, once they had finished post-editing, their opinions were not so negative and most of them would be willing to post-edit in a regular basis even though they all enjoyed more translating. The main challenges of post-editing were related to the constraints it imposes, mainly to creativity. Another important problem was the unpredictable errors in the MT output, which were sometimes difficult to spot.

Conclusions and further discussion
Regarding productivity, post-editing reduced in half the time spent by word.
Experienced post-editors also considered this task to be more repetitive, more tiring and less paid than translating from scratch. However, they highlighted post-editing productivity increases with experience and proper training. Moreover, post-editing reduces typing, which usually helps to increase productivity. In general, translators are less satisfied with post-editing than with translation from scratch.
According to the opinions of the participants, training and experience are key elements to post-editing with more confidence. There is also a need for a more fluent communication throughout the translation workflow, mainly related to information regarding the origin of the MT output. As Vieira and Alonso explain (2018), one of the main problems for translators is the lack of communication from project managers regarding what they are asked to do and how to do it. The focus of the translation workflow should be the human translator and MT should be used as a way to improve a human-centered process. Finally, as post-editors believe it is a more tiring task, one possibility would be to limit the time spent post-editing and always combine it with translation from scratch as part of the industrial workflow.
All in all, post-editing is a task that offers a new specialization for current translators. However, current market practices have to be improved and tuned in order to make it more attractive and enjoyable for professional translators.