Thursday, September 22, 2016

Can computers replace teachers? Cambridge "Write and Improve" automated writing assessment - does it work in practice?

It's been well over twenty years since people started debating whether computers would ever replace teachers.  Of course, there will always be a need for teachers and the real human interaction that is part of learning, so the question is not so much about computer replacing teachers, but what kind of relationship will teachers (and their students) form with the emerging artificial intelligent entities to provide enhanced learning opportunities.



In my quest to discover such artificially intelligent entities, I came across the Cambridge Write and Improve site, in BETA, which lets students submit their writing and receive 'feedback' from an algorithm that analyses the student's writing using the knowledge it has acquired by examining millions of words written by other learners of English taking various Cambridge proficiency exams.


I was curious to see how useful AI feedback would be to a students, so I found a writing from one of my students in a pre-intermediate class, and typed it in to the Write and Improve site.  In fact, this was the final draft of a writing task done in class, so the student had gone through two previous drafts.  In the  Cambridge English "Write and Improve" site, there are options to write about pre-defined topics, aligned to the typical types of topics given in the Cambridge exams. Presumably, the feedback would be more focused and detailed for these. However, there is an  ANY OTHER OPTION for writing about any topic, so I used that.   Here is the text of the final draft which the student had written about the similarities between high school and an English preparatory school:

There are three similarities between high school and preparatory school.  The first similarity is absenteeism.  Both high school and preparatory school have a limited absenteeism.  Therefore, students who are studying in high school and in preparatory school must go regularly.  If students don't go to school, they might be dismissed. Absenteeism is important for not only high school but also preparatory school.  The second similarity is lesson time.  In high schools the duration of lessons is six hours per day.  Likewise, in preparatory schools the duration of lessons is six hours per day. Both high schools and preparatory schools have ten minutes break time. Like in high schools, preparatory schools start lessons at 8:40.  The final similarity is having a fixed syllabus.  Like in high schools, teachers in preparatory schools follow a fixed programme because the students will pass an exam in the last semester.  Moreover, both of them give six exams, and a lot of quizzes.  To sum up, high school and preparatory school are similar in terms of absenteeism, lesson times, and having a fixed syllabus.

I pasted this into the ANY OTHER TOPIC option and asked for an analysis.  After thinking for a few moments, the AI assessment of B1 level of proficiency evident in the writing was proclaimed.  This intuitively seems about what I would have given it, but it took the AI only a few seconds to deliberate.

Inline image 2

Below is the actual feedback -- it just colours the sentences - green is good, yellow is okay but could be improved.  Orange means needs improvement, and red would mean errors.  There are no detailed errors or suggestions for improvements, most likely because this was the final draft and most of the mechanical errors had already been dealt with.  Also, the use of the ANY OTHER OPTION probably limited the AI to really pick out more specific issues.


Inline image 5

My opinion - if the student had typed in their first draft (which would have had more errors), perhaps it would give some useful feedback.  The colouring does give students an idea of where their writing could be developed on a sentence level.  The nice thing is that students get an approximate idea of their CEFR writing level.  At METU, they should be writing at a B2+ level of proficiency, so having this general level feedback may be an incentive for them to develop their writing more.

I like the idea of the colour coding for giving feedback, rather than the usual distinct error marking which we are recommended to do in the prep school, e.g., verb-tense agreement, missing word, wrong word, tense, wrong form, etc.  In fact, I actually use highlighting pens to give feedback to my students, and avoid the use of error codes.  However, I use the colour coding for different types of errors, e.g., lexical, grammatical, meaning and word choice.  I might try this method of colour coding method next time.  It makes the student try to figure out what is good or wrong with a sentence, rather than just focusing on trying to correct individual specific marked errors.

Being a naturally curious person, I put on my hat as a student and wondered if such feedback would actually give me any help in improving the text.  So, I went to the editing window with the paragraph and I made an effort to retain the content and organization, but tried to make the feedback highlighting mostly green.  Since I didn't change the content or the  organization, my sole task was to embellish the text with more 'native like' phrases and constructs-like our students could do if they used a corpus-assisted writing tool like http://wordandphrase.info.  I didn't change the fundamental content or organization.  Instead, I looked at the colour coding, and tried primarily to improve the weaker sentences to make everything come up shades of green.  Here is what I came up with:

Despite the fact that high schools and preparatory schools appear to be different in many ways, they are similar in several aspects. The first aspect in which these two types of school are similar is the way in which they deal with absenteeism.  Both types of school have a policy to allow only a limited amount of absenteeism.  Therefore, students who are studying in high school and in preparatory school must attend regularly and punctually.  If students don't attend, they might run the risk of being dismissed, which demonstrates why an absenteeism policy is important for not only high schools but also preparatory schools. Aside from absenteeism, high schools and preparatory schools also have a similar daily schedule.  The daily routine in high schools comprises six one-hour lessons.  Likewise, the daily schedule in preparatory schools consists of six one-hour lessons. In both types of school, there is a mandatory ten-minute break between lessons. Like in high schools, preparatory schools start lessons at 8:40. These two types of school are also similar in that they both follow a fixed syllabus.  Like in high schools, teachers in preparatory schools follow a fixed programme because the students will pass an exam in the last semester.  Moreover, both of them give six exams, and administer a lot of quizzes throughout the terms. To sum up, high schools and preparatory schools are similar in terms of absenteeism, lesson times, and having a fixed syllabus.

So, the content is basically the same, but I've increased the number of words and although I've left the organization basically the same, I've introduced some different sentence constructions or phrases.

I this way, the system assessed the writing as one band higher - it is now above that magical B2 proficiency rating which is the threshold for 60% on the METU English proficiency test.

Inline image 6

I haven't been able to make the entire essay green, but it is now mostly in shades of green.  

Inline image 7

I think this is a good case in point to illustrate the kind of relationship that teachers and students can foster with artificial intelligence entities.  Students are free to get feedback whenever they want, and gain an awareness of how their writing measures up to a standardized CEFR assessment.  The AI never tires of giving feedback, never sighs at the sight of obvious or repeated errors, and goes to the trouble of highlighting each sentence to show the good and the bad.  Over and over again, night or day, rain or shine...internet permitting.  It could, to some extent, help students identify basic sentence level problems in the drafting process.  However, the teacher would still need to step in to give higher levels of feedback in terms of issues like redundancy and overall organization or more complex issues.

To me, this illustrates the potential of developing a good relationship between AI, teachers and students.  Let the students use the AI to work out the basic problems with sentence level constructions, leaving the teacher to step in and spend more time on higher level feedback. Certainly this will be worth a closer look when it moves out of the BETA stage.

5 comments:

  1. Steve,

    I'm Paul Butcher, co-founder and CTO of English Language iTutoring (the company behind Write and Improve). Thank you so much for taking the time to write this, and for your insightful comments.

    As it happens, your timing is perfect: Write and Improve is just now out of beta, and the full production version is now available:

    https://writeandimprove.com/

    As you will see, the UI has had a *lot* of work and is significantly slicker. Not only that, but the underlying engine has also improved, and should give even more accurate and detailed feedback.

    I hope that you enjoy using it, and would be very interested to hear your feedback.

    ReplyDelete
    Replies
    1. Thanks for the heads up about the update. I'll certainly give the new version a try. I'm hoping to introduce it to my students this semester, so I'll be curious to see how they get on with it.

      Delete
  2. Hi Steve, it's great to have your feedback on Write & Improve (Beta) and I really hope you'll take the chance to look at the full version, which we at ELiT are very proud of. www.writeandimprove.com Thank you!
    I just wanted to say that replacing teachers couldn't be further from our minds! In fact, what you say in your closing paragraph hits the nail on the head as far as our hopes for teachers are concerned.
    We hope Write & Improve, as well as giving learners a place to practise their writing and learn good reviewing and self-correction skills, and hopefully making them aware of their fossilized errors too, could at the same time take some of the sighing and déjà vu out of marking for teachers, freeing them up to focus on those things that most definitely require a human!

    Thanks again for your interest!

    ReplyDelete
    Replies
    1. I'll be curious to see to what extent students will embrace such a tool for self-correction. Many of my students are used to the classic teacher-directed correction/feedback, so they may not have the autonomous learning skills and attitude necessary to really take advantage of this tool. We'll see.

      Delete
  3. We'd love to have your feedback on that, Steve. Please keep in touch.

    ReplyDelete