AUTOMATED WRITING EVALUATION REVIEW

Advances through

AI systems

Significant breakthroughs:

AI systems

• Natural language processing (NLP)

• Natural language understanding

Corrective feedback (CF)

• Providing useful CF = a difficult task

• CF, if appropriately applied, is beneficial.

Corrective feedback

- Grammarly = an AWCF tool that

(1) can work as standalone tool

(2) is integrated into existing writing tools (Microsoft Word or Google Docs)

(3) can work as a web browser extension or as a virtual keyboard in smartphones.

- Grammarly has both strengths and weaknesses that affect users’ experience.

- Typically, L2 students use MT to look up words or phrases, not the whole text.

Convenience

Speed

- Recent studies found that virtually all students surveyed used MT (mostly Google Translate) for learning tasks in instructed language learning.

- Along with Grammarly, Google Translate seems to have become a ubiquitous helper for students writing in an L2.

Availability

Free cost

Advantages

Disadvantages

- Reductionist perception of language in which human language can be simply

- Improved writing quality through the use of MT integrated into learning tasks (Fredholm, 2014, 2015, 2019; E. M. O’Neill, 2016, 2019)

reencoded based on a one-to-one correspondence between languages.

- Training in post-editing machine-translated texts = helpful in correcting raw MT output and in gaining insight into MT limitations.

- Students may see Google Translate as an “answer key,” pointing to a simplistic view of language (Ryu, 2022).

- Instrumentalist view of language “fails to acknowledge the richness and complexity of human interaction, identity, and culture” (Urlaub & Dessein, 2022)

- Developing the skills to post-edit requires focused attention and advanced reading ability, valuable both in language learning and in professional translation (H. Zhang and Torres-Hostench, 2022).

.

.

.

.

.

.

.

.

.

.

- MT captures the semantic dimension of language but misses the nuances and the context-dependent essence of human communication.

- MT into the classroom = leveraging students’ L1 in the learning process and contrasting its usage patterns with the L2.

Implication

- The use of MT for views on the nature of language may lead to a reconsideration of the goals of language education generally (Vinall & Hellmich, 2022).

- The availability of advanced writing and translation tools, linguistic accuracy “can no longer be viewed as a synonym of learning and excellence” (Klekovkina & Denié-Higney, 2022)

Topic

- If MT were capable of capturing the total essence of a language, that will reduce language to an instrumental role.

- Machine Translation will perhaps suffice for transactional language needs, but will hardly be a substitute for genuine person-to-person encounters (Godwin-Jones, 2019).

Language models

AI systems built from collections of data that are analyzed by machine learning

--> An ability to deal with human language in many effective ways

Language modeling involves predicting the next word in a text given the previous words (Ruder, 2018)

Predictive text technology

The ability of AI systems to write on their own

AWCF/AWE tools

- Have learners examine their own compositions for a particular type of error.

- Let them run their writing through an AWCF tool to see if it flags errors in that grammatical category.

(John and Woll, 2020)

Topic

- Have students perform a review of AWE feedback, critically examining its effectiveness and usefulness.

- Supply students with a text to proof, identifying errors and providing corrections.

(Ranalli, 2021)

Google Translation

- Assign writing assignments which incorporate specific materials currently or previously studied in the course.

- Ask students to identify and label the examples, using a checklist of specific grammar or vocabulary.

Note: Grading rubrics used in that study are made with the use of Google Translate, as they include identification of grammar and vocabulary.

(Knowles, 2022)

Topic

- Asks students to discuss the sociopragmatic issues involved in the translation of a given text (i.e., “what’s your name?”).

(Pellet and Myers, 2022)

Explicit instruction and guidance in the use of AI tools

- Guidance:

+ How AI-based systems work

+ What kind of writing they are best used for

+ What kind of performance one is likely to expect.

--> Building familiarity & confidence

Topic

- Students should develop realistic expectations

of utility in tool choice and use.

--> Help with appropriate use of language tools

Teacher feedback

Researchers emphasize the importance of teachers’ feedback on students writing (Y. Huang & Wilson, 2021; Z. Li, 2021). Link et al. (2020)

Teacher feedback

An ideal hybrid situation of having both the AWE tool and the teacher comment on students’ performance.

An AWE tool will provide sentence-level feedback while the teacher attends to higher-order writing issues.

--> Optimize the time teacher spends on correcting each writing

Peer feedback

Combining AWE with peer review is an additional option (Hockly, 2019).

Peer feedback

Mechanisms for including peer review: Criterion and MI Write

Peer-to-peer interactions: Google Translate

Learning environment

The same tool used in different environments is likely to have widely different results.

Learning environment

Some aspects that should be taken into considerations:

Individual Classroom

Department Institution

Time

Individuals

Individual teacher differences that can play a major role (Link et al., 2020) as well as student characteristics (Y. Huang & Wilson, 2021; Ranalli, 2021)

Individuals

The use of AWE systems shows widely different patterns of emergence through the individual students followed.

Human-machine relationship

There is likely to be a range of attitudes and reactions to the technology.

- Enthusiastic acceptance

Human-machine relationship

- Utter rejection

- Many stages in between.