The use of MT by undergraduate translation students for different learning tasks

This study investigates the use of MT (Machine Translation) by first, second-and third-year translation students for different learning tasks. The results of a self-report survey suggest that most students make regular use of MT, although the frequency of use decreases as they progress through their studies, particularly for reading tasks. Our results highlight the need for a comprehensive approach to the development of MT knowledge and PE (Post-Editing) skills among undergraduate students, using the concept of MT literacy as a stepping stone


INTRODUCTION
Ever since Google's introduction of neural Machine Translation (MT) in 2016 [Wu et al., 2016], we have seen an increase in the quality of MT solutions and a wider acceptance of its use by both the general population globally and the translation industry [Blagodarna, 2018].As was to be expected, these developments have also had a considerable effect on translator training programmes.
Translation scholars striving to address the question of how these developments should be incorporated into translator training, tend to favour an approach that makes it possible for students to gain the necessary experience and knowledge in realistic settings, i.e., (simulated) situations that resemble actual professional situations in the translation industry.This applies to teaching CAT tools and translation project management, but also to teaching MT and postediting (PE), the subject of this article.As Mellinger [2017, 284] states: Rather than viewing machine translation as a solitary task or tool that should be treated solely in a single course or module, MT can be positioned as a tool used in the service of a specific goal […].By integrating and embedding machine translation across the curriculum, trainers can model expert behaviour and encourage students to engage in best practices, which will position them well for current industry practices.
An interesting new addition to this discussion comes from the field of language studies.Defined by O'Brien and Ehrensberger-Dow [2020, 146] as " […] knowing how MT works, how it can be useful in a particular context, and what the implications are of using MT for specific communicative needs", the concept of 'MT literacy' was introduced to help language students and teachers deal more effectively with MT in their language learning process.The concept is recognized by many scholars as a necessary addition to the students' and teachers' toolkit, as MT is widely used by students in language learning, but mostly without training [Loock and Léchauguette, 2021].
A similar point can be made for undergraduate students in translation, as anecdotal evidence suggests that they also make extensive use of MT without having received prior training.For instance, the Maastricht School of Translation and Interpreting, the bachelor programme in translation offered at our university, does not provide explicit training in MT and PE until the fourth semester.At that stage, however, the students are likely to have used MT for several years and for a variety of purposes.
The goal of this study is to investigate how often students use MT for different learning tasks, and how their use of MT changes as they progress through their study programme.This knowledge is necessary to develop a comprehensive approach to teaching MT and PE that not only enhances students' MT literacy, but also takes into account the knowledge and experience that they already have acquired; an approach that should ultimately help translation students make more effective use of MT, both in the language learning process and in developing professional PE skills.

I LITERATURE REVIEW
When it comes to research on the use of MT by students, most studies have focused on foreign language learners.A few examples from research in the past decade illustrate clearly that the majority of foreign language learners use MT: • Clifford et al. [2013] show that 88% of their sample of undergraduate students learning French, Italian, Spanish and Portuguese as a foreign language use MT; • Larson-Guenette [2013] shows that 68% of her sample of university students learning German as a foreign language use MT or other online resources; • Alhaisoni and Alhaysony [2017] show that 96% of their sample of Saudi university students learning English as a foreign language use MT; • Briggs [2018] shows that 85% of his sample of Korean-speaking university students learning English as a foreign language use MT; • O'Neill [2016] shows that 88% of his sample of learners of Spanish or French as a foreign language use MT; • Bourdais and Guichon [2020] show that 89% of their sample of French pupils in secondary education learning English as a foreign language use MT.
Moreover, the aforementioned articles show that most learners use MT regularly or even frequently.For instance, Briggs [2018, 11] finds that out of 81 students using MT, 31 students use it daily, often or frequently, 36 students use it a few times a week/occasionally and 14 students use it less than once a week or seldom, both at school and out of school.Clifford et al. [2013, 111] shows that 36% of the students using MT use it often, while 44% use it sometimes and 19% use it rarely.
In both O'Neill [2019] and Alhaisoni and Alhaysony [2017] a distinction in MT use is made between always, often, sometimes and rarely/seldom.These studies report similar results: out of the 160 students using MT in O'Neill [2019,160], 20% of the students indicated that they make use of it always, while 37% use MT often, 32% sometimes and 11% rarely.Out of the students using MT in Alhaisoni and Alhaysony [2017, 77], 26% indicated that they use it always, 33% often, 26% sometimes and 15% use it seldom.
Another interesting finding is that students tend to use MT primarily to look up vocabulary [Alhaisoni and Alhaysony, 2017;Bourdais and Guichon, 2020;Briggs, 2018;Clifford et al., 2013;Larson-Guenette, 2013;Loock and Léchauguette, 2021;O'Neill, 2016].Finally, an interesting element to add is that students will use MT also if its use is explicitly prohibited by the course instructor [O'Neill, 2016;White and Heidrich, 2013].
In comparison to foreign language learners, data concerning the 'uncontrolled' use of MT by translation students are hard to come by.Alsalem [2019, 47] refers to "the vast majority of students" that use MT for their assignments, but his finding relies on anecdotal evidence.Koletnik Korošec [2011] shows in her survey among Slovenian students (third-year bachelor students in Inter-lingual Studies) that 90% report using MT.
A more recent survey by Loock and Léchauguette [2021] shows that out of the 89 students enrolled in translation courses at the University of Lille, 83% used online MT tools for their homework assignments.However, it is important to stress that these students were not necessarily translation students, but students following classes in which they practised 'pedagogical translation', which is primarily aimed at increasing language skills.
Our research differs from earlier studies in that our data will provide insight into the evolution of MT use for specific learning tasks throughout the first, second and third year of an undergraduate translation programmein this case, the programme offered by the Maastricht School of Translation and Interpreting at Zuyd University of Applied Sciences.

Hypotheses
Based on prior literature, two conflicting hypotheses concerning the change in students' use of MT throughout the first, second and third year of translator training can be formulated.
The first hypothesis is that the data will report a decline in MT use for the different learning tasks (from year 1 to year 3), as students may resort less to MT as they progress in mastering the foreign language.This hypothesis is derived from the observation that students with a lower proficiency in a foreign language tend to resort to MT more often [Bourdais and Guichon, 2020].This is also implied by Larson-Guenette [2013], who mentions that students with a higher proficiency in a foreign language, i.e., students in fifth-and sixth-semester language courses, are more concerned with becoming dependent on online resources and thus resort less often to MT.For the purposes of this paper, this hypothesis will be called the 'language proficiency hypothesis'.
The second hypothesis is that the data will report a rise in MT use for the different learning tasks (from year 1 to year 3).One could argue that translation students gain more (implicit and explicit) know-how with respect to using MT as they progress through their studies, and therefore use it more effectively and maybe even more intensively in the later stages of the programme, particularly for translation assignments.For the purposes of this paper, this hypothesis will be called the 'MT literacy hypothesis'.

II METHODOLOGY
In order to investigate students' use of MT for different learning tasks, we conducted two selfadministered internet surveys: one at the end of 2020, aimed at first-year students at the Maastricht School of Translation and Interpreting, and one in 2021, aimed at second and thirdyear students of the same school.

Participants and procedure
Of the 112 first-year students officially enrolled in the programme at the time the survey was administered, 61 took part in the survey (54%).The students were personally invited to participate by the first author during an online class he was teaching, in which he explained the goal and the scope of the survey.Students did not receive course credit nor any kind of remuneration for participation.Furthermore, they were reassured that their decision (not) to participate would not influence their course grade, as the survey was completely anonymous.
The students could access the survey via a web link, which redirected them to an online survey environment created using the online survey tool Microsoft Forms.At the time this survey was administered, our university did not yet have a standard survey tool, which is why we chose to use a tool that the first author knew and had used extensively the past.
The response rate for the second-year students (n = 8) and third-year students (n = 13) was considerably lower (34% overall), presumably because they were invited via email and not during a class session.Their participation was also voluntary, and they did not receive course credit.As an incentive, two gift vouchers were raffled off amongst the participants who provided us with their email address.The students could access the survey via a web link, which redirected them to an online survey environment created using Questback, a feedback platform that had in the meantime become the standard tool for administering surveys at our university.
The survey contained two introductory slides explaining the goal and scope of the survey and allowing the students to indicate whether they consented to the use of their data for these specific research purposes.

Instrumentation
The two survey versions (first-year and second / third-year) had many elements in common, but also differed in several aspects.What they had in common was that the majority of the questions required students to report their behaviour, attitude or opinion on a Likert-type scale that was comparable to the scales used by other studies [Alhaisoni and Alhaysony, 2017;Bourdais and Guichon, 2020;Briggs, 2018;Clifford et al., 2013;O'Neill, 2016].Furthermore, both surveys contained questions about variables that fell outside of the scope of this study (e.g., the degree to which MT output is verified and corrected or the use of other online translation resources).
For this study, we focused specifically on questions regarding the frequency of MT use for different learning tasks (writing in a foreign language, reading in a foreign language and translating from and into a foreign language).In the first-year survey, students indicated how often (Never, Sometimes, Often or Very often) they use MT for different learning tasks in their foreign language courses (English and French, German or Spanish), distinguishing 'Writing an essay or longer piece of text in a foreign language', 'Short written assignments in a foreign language', 'Translation assignments from a foreign language to Dutch or the other way around' and 'Reading assignments in a foreign language'.
Export files of all three surveys (year 1 to year 3), containing both the questions and answer options and the individual responses, are available via an online repository1 .
Since the structure of the second / third-year survey was different, the way in which these questions were phrased also differed.For each learning task ('writing in a foreign language', 'translation assignments' and 'reading in a foreign language'), we asked students to indicate on a seven-point scale how often they used MT for a set of eight specific sub-items: to understand or translate words (1), sentences (2), paragraphs (3) and entire texts (4) from a foreign language into Dutch and to understand or translate words (5), sentences (6), paragraphs (7) and entire texts (8) from Dutch into a foreign language.
To be able to compare the answers of the first-year students to those of the second and thirdyear students, we performed a number of statistical operations that are described in the next section.

Analysis
The data were analysed using IBM SPSS version 28.The data and syntax logs are available via an online repository.Three preparatory operations were necessary before the data of the two survey versions could be compared.First of all, the four-point scales on which the first-year students indicated their answers were transformed into seven-point scales following the procedure outlined in Lewis and Sauro [2020] for semantic differentials and IBM [2020] for ordinal scales.
Secondly, the fact that 'writing in a foreign language' was operationalized in two separate items in the first-year survey but not in the second/third-year survey had to be taken into account.Since these two items ('Writing an essay or longer piece of text in a foreign language' and 'Short written assignments in a foreign language') were highly intercorrelated (Cronbach's α = .95),we used the average of these items as a measure for the frequency of MT use while writing in a foreign language.
Similarly, the second / third-year survey contained separate questions for eight sub-activities (see above) falling under the main learning activities of writing, translating and reading, while the first-year survey did not.As with the first-year survey, we treated these questions as items of a scale, as the values for Cronbach's α suggested that they were highly intercorrelated (all α's between .92 and .95).Consequently, we used the averages of these items as measures for the frequency of MT use while writing, reading and translating, respectively.
To gain insight into the frequency of MT use by the students in our sample, we first generated descriptive statistics for the sample as a whole (across student groups and learning tasks) and for the three student groups separately (across learning tasks).Subsequently, we compared the use of MT in different learning tasks (a within-subjects factor) and the use of MT by the three student groups for each learning task separately (a between-subjects factor).Because of the methodological differences between the two survey versions, the non-normal distribution of the frequency estimates and the significant differences in their variances between student groups, we decided to use non-parametric tests for these comparisons.Significant differences were further explored by means of a stepwise stepdown procedure [Field, 2017].

III FINDINGS
In Table 1, you can find the mean, standard deviation and median for the frequency of MT use across learning tasks for the total sample, as well as for the three student groups separately.The data show that students in all years report using MT, and that there is considerable variation in how often they use it.This can also be seen when looking at the distribution of the frequency of MT use across student groups and learning tasks, which is shown in Figure 1.However, when we visualize the differences between student groups and learning tasks (see Figure 2), a number of interesting patterns emerge.Firstly, students seem to use MT most frequently while working on translation tasks, as evidenced by the high position of the blue line compared to the other two lines.Secondly, we see a slight increase in MT use among secondyear students compared to first-year students.Finally, third-year students resort to MT less often for all learning tasks, most notably for reading in a foreign language, as can be seen by the sharp downward bend in the red line.These patterns were partly supported by the outcomes of the statistical tests.First of all, a Friedman's ANOVA showed that the differences between the learning tasks were significant (p < 0.05), with the stepwise stepdown procedure revealing that reading differed significantly from translating (mean rankreading = 1.88; mean ranktranslating = 2.20; p < 0.05), while writing (mean rank = 1.92) did not differ significantly from the other two activities.For writing and translating, the Kruskal-Wallis test indicated that there were no significant differences between student groups (p-values of 0.25 and 0.29, respectively), but for reading, a significant difference was observed (p < 0.05).More specifically, third-year students (mean rank = 27.77)used MT significantly less often in reading tasks than the other two student groups (p < 0.05), while the first-year and second-year students did not differ from each other (mean rank1st year = 43.38;mean rank2nd year = 49.50;p = 0.46).
To summarize, the students in our sample used MT most often in translation tasks and least often in reading tasks, and the third-year students used MT less frequently in reading tasks than the first and second-year students.For the other two learning tasks, no differences between student groups were observed.

IV DISCUSSION
Looking back at our hypotheses, our data provide support for the language proficiency hypothesis: undergraduate translation students use MT more often during the early stages of their studies, and less often during the later stagesmost notably for reading in a foreign language.This suggests that students rely on MT more when they need help in understanding foreign languages, and less as they develop stronger language skills.
Given that the participants were translation students who are preparing to work as professional translators or post-editors in a rapidly automating industry, it could be expected that educators might prefer to see students use MT more effectively and increasingly for translation tasks as they progress through their studies (as predicted by the MT literacy hypothesis); however, this did not show from our data.Alternatively, it could be argued that the observed decrease in MT use in year 3 highlights that MT literacy does not develop automatically but needs to be trained explicitly.
With respect to the timing of this training, it is important to acknowledge that students will use MT anyway, regardless of whether you allow them to use it or provide them with prior training [O'Neill, 2016;White and Heidrich, 2013].For translator training curricula, this suggests a need for the repositioning of MT and PE skills.If MT literacy is considered an essential competence for language learning as well as translator training, MT and PE should be integrated into the curriculum from the earliest possible stage, rather than being addressed as separate entities in translation technology modules.
With regards to integration in the early stages of the programme, the focus could lie on topics that are useful for both language learning and translator training, such as estimating whether using MT is beneficial for a particular assignment, selecting which MT tool to use, finding the errors in MT output, or comparing MT output to other information sources.Other possible subjects for early MT training could be privacy, academic integrity, or the potential for algorithmic bias [Bowker, 2020].Finally, slightly more advanced activities could include postediting together and discussing different PE solutions in the classroom.
In the later stages of the programme, students can use MT not only to work on the development of professional PE skills (e.g., being able to post-edit a given text to a predefined quality level within a certain timeframe, according to particular instructions, such as the TAUS PE guidelines), but also to learn about (localization) project management skills in relation to MT, such as being able to assess which MT solution works best for a particular assignment, setting up a workflow with MT and PE in mind, or even managing collaborative MT projects in a (simulated) translation agency [Buysschaert et al., 2017].Ideally, the material relating to these subjects should be offered to students in realistic settings, or even 'just in time', allowing students to directly apply what they have learned [Gavrin, 2006].

Limitations and further research
Admittedly, there are a number of limitations to this study.The sample was limited to students from one Dutch undergraduate programme, which limits the generalizability of the outcomes to other populations of language learners, and only a small number of second and third-year students participated in the study.The small sample size could also be a partial explanation for the absence of significant effects of MT for translation and writing tasks.Finally, in order to obtain a thorough understanding of students' actual degree of MT literacy and to evaluate how effectively they are using MT, more research is necessary.For example, it could be worthwhile to follow one single cohort throughout the curriculum or conduct a detailed observation of different groups of undergraduate translation students working on different learning tasks.

CONCLUSION
In this article, we provide evidence for the necessity of an integrated approach to developing MT and PE skills among undergraduate translation students.This need is highlighted by our finding that third-year students resort to MT less often than first and second-year students, most notably for reading in a foreign language, even though it would be expected that students might be using MT more effectively at this point in their studies.The overall decreasing trend in MT use suggests that MT literacy does not develop automatically, and that educators need to pay explicit attention to the advantages and disadvantages of using MT.By doing this from the very beginning of the curriculum, they can help students develop a professional attitude toward MT and PE, using the concept of MT literacy as a stepping stone.

Figure 1 .
Figure 1.Distribution of frequency of MT use across student groups and learning tasks.

Table 1 .
Means, standard deviations and medians of frequency of MT use across learning tasks (7-point scale).