“Shouldn’t there be a dot on that ‘i’?” – Can machines help humans mark MOOCs?

The issue of how a mark MOOCs is a moot point at the moment.

As Europeans race to play catch up with their U.S. counterparts, (no educational body left behind!?), two glaring questions rise to the fore in many articles about these massive open online courses.

These are:

  • How do institutions make money from them?
  • Will MOOC students be able to gain credits for offline courses?

The answer to both these questions, in my view, ultimately rests on how the courses are evaluated.

To date, I have completed two MOOCs – Edinburgh University’s E-Learning and Digital Cultures (edcmooc), and the Open University’s Open Education (#h817open).

For the first, the assessment was done by peer evaluation of a digital artefact with students asked to mark their fellows on a scale of zero to two (0 = “You have got to be kidding me”; 2 = “I bow to your awesomeness”).

Effectively, anyone who submitted a video, Prezi, blog or other clickable virtual product gained a ‘Statement of Accomplishment’; but, only those awarded a grade of 1.5 or 2.0 earned the prestigious ‘with distinction’ wording on their downloadable certificate.

Before those in the latter category rushed to celebrate, they would have been well-advised to read the small print on the certificate which read :
“Please note: The online offering of this class does not reflect the entire curriculum offered to students enrolled at the University of Edinburgh. It does not affirm that the student was enrolled at the University of Edinburgh or confer a University of Edinburgh degree, grade or credit. The course did not verify the identity of the student”.

In plain English, this tells us: Don’t pretend you are on a par with real Edinburgh University academics; hell, we can’t even prove you are who you say you are.

Martin Weller

For the OU course, there were no certificates; the only way you could prove to doubters that you weren’t one of the dropouts was to apply for, receive and display a badge to show  a) Understanding of OERs (Open Educational Resources) b) Understanding of MOOCs c) Course completion.

The decision as to who had earned these badges fell to course guru, and all round decent guy,  Martin Weller.

Needless to say, no-one could delude themselves into thinking that these awards (created in Cloudworks) were equivalent to an Open University degree, grade or credit.

All this is fine and dandy for ageing lifelong learners, like yours truly, who are seeking greater wisdom rather than looking for ways to sex up their CV.

Nevertheless, I have a sneaking suspicion that the novelty value of MOOCs will quickly wear off for many participants when it dawns on them that there is no meaningful feedback or tangible reward for all the heavy-duty brainpower they have expended.

How can MOOCs boost credibility and maintain their appeal on this thorny issue of assessment?

Given that those registered for these free courses runs into the thousands rather than the hundreds, the potential workload for even the most dedicated academic is daunting and, arguably, beyond the realm of possibility.

Peer assessment is one alternative but, as this tends to be a bit of a lottery, it is not always popular with students and is unlikely to cut much ice with most educational institutions either.

Another possibility is to devise some automated computer-assisted marking scheme.

Prolific Melbourne-based teacher, writer and freelance journalist, Stephen Downes, has just pondered upon these very questions on his illuminating ‘Half an Hour blog’. He confesses at the outset “I can’t see myself marking 5,000 term papers, and a similar number of exams” , and who can blame him?

As a self-declared “specialist in online learning and new media technologies”, Downes is a man who should be able to shed some light on this topic, but, although he offers some interesting examples of methods that have been tried, he doesn’t say much more than to observe that there are risks whatever approach you take.

Let's hear it for the old-school methods!

Let’s hear it for the old-school method!

If you look at the recent Position Statement on Machine Scoring by the National Council of Teachers of English (NCTE), you are left in no doubt that schemes that propose a non-human grading of written work ruffle quite a few feathers.

The opening paragraph of this statement sets the tone:  “Writing is a highly complex ability developed over years of practice, across a wide range of tasks and contexts, and with copious, meaningful feedback. Students must have this kind of sustained experience to meet the demands of higher education, the needs of a 21st-century workforce, the challenges of civic participation, and the realization of full, meaningful lives”.

After reading this, you will not be surprised to learn that they totally reject the notion that any artificial intelligence could be up to the job of grading what they describe as “rich, multifaceted” tasks.

NCTE dismiss the idea that machines can gauge the subtleties of written texts, and state that the process would merely reward grammatical accuracy/lexical complexity. This may therefore penalize the type of innovative or creative styles that  good university-level students should be aspiring to. If, for example, someone uses repetition to hammer home a point, would a computer be able to pick up the fact that this was done deliberately.

NCTE also foresee that test-takers would soon devise “machine-tricking strategies”, and it’s easy to see how such techniques could be rapidly circulated on the net.

You begin to see what a thorny issue this whole marking business is.

I think it is inevitable that, sooner or later, participants will be asked to pay for some, or all, of the ‘expert’ support offered on these massive courses. When that day arrives, the honeymoon period of MOOCs will be well and truly over.

At this point, we are likely to see an escalation of learner-generated remixes of open educational resources that by-pass institutionalised credit-based systems entirely.

Of course, you wouldn’t have any paperwork at the end of such ‘courses’ but they would fulfill the goal of improving world knowledge and, correct me if I’m mistaken, isn’t that what education is meant to be for?