Recent years have witnessed the creation of a plethora of writing assistants. These range from well-known commercial products aimed at users of general English (e.g., Grammarly (Shevchenko et al., 2009)) including those aimed at users of specific language pairs (e.g., Spanish-English, Write Assistant (Fuertes-Olivera & Tarp, 2020)), to those aimed at users of languages for specific purposes with specific language pairs (e.g., Academic Portuguese-Academic English, Scrido (Zomer, 2023)). Evaluations of the usability of writing assistants generally report positive results (e.g., Frankenberg-Garcia, Rees, et al., 2019). However, this may be in part due to the conditions in which these evaluations typically take place. For example, existing studies of the usability of writing assistants are predominantly lab- or classroom-based. While this approach allows for a high degree of internal validity, this often comes at the price of external validity. In other words, experimental conditions often differ from the typical everyday writing conditions of many users. Similarly, the types of writing tasks undertaken in a lab or class differ from real-world writing. Historically, administering studies in realistic contexts has proved complex, time consuming, and expensive.

Taking advantage of the affordances of cloud computing and webapps, the diary study reported here attempts to overcome these impediments. In this study, twelve writers from a range of L1 backgrounds recorded their experiences with the academic writing assistant ColloCaid (Frankenberg-Garcia, Lew, et al., 2019; Frankenberg-Garcia et al., 2021) for ten days. Four of these writers later elaborated on the experience in interviews.

The findings have both methodological and practical implications. From a methodological perspective, they demonstrate how technological developments such as cloud computing and webapps can provide an ecologically valid means of evaluating usability in an efficient way. From a practical perspective, they not only reaffirm the positive impression that users have of writing assistants, but also suggest that to have widespread impact on practice, writing assistants must be closely integrated with popular productivity tools such as word processors and email clients.

---

Abstract Summary
There are a plethora of writing assistants available (e.g., Grammarly (Shevchenko et al., 2009), Write Assistant (Fuertes-Olivera & Tarp, 2020, Scrido (Zomer, 2023)). Usability evaluations are usually positive (Frankenberg-Garcia, Rees, et al., 2019). However, they are predominantly lab- or classroom-based. This allows for a high internal validity but comes at the price of external validity. Similarly, the types of writing undertaken differ from real-world academic writing. Historically, realistic studies often proved unfeasible.

Using the affordances of cloud-computing and webapps, this diary study overcomes these impediments. Twelve writers recorded experiences with ColloCaid (Frankenberg-Garcia, Lew, et al., 2019) for ten days. This was followed by interviews.

Findings demonstrate how cloud-computing and webapps can facilitate an ecologically valid and efficient means of evaluating usability. Moreover, they not only reaffirm the positive impression of writing assistants, but also suggest the need for their closer integration with productivity tools such as word-processors and email clients.

Authors:
Geraint Paul Rees, Universitat Rovira i Virgili, Spain

About the Presenter(s)
Dr Geraint Paul Rees is a University Assistant Professor/Lecturer at Universitat Rovira i Virgili in Spain

See this presentation on the full schedule – Monday Schedule

Bookmark