How AI Scores English Language Tests, and What Human Examiners Still Do

Every year, millions of people take an English test to get into a university, obtain a visa or register for a profession. For most of the history of those tests, a trained human examiner read each essay and listened to each spoken answer. That is no longer the general rule. Several of the most widely accepted tests now rely heavily on automated scoring, and in one major test a computer produces the first score on every speaking and writing response.

For test takers, this raises practical questions. Can a machine judge what someone means, or only how they say it? Does it treat speakers of different first languages fairly? Can it be gamed with memorised templates? And who is accountable when a score that decides someone's future is produced by software?

This article explains how the main English tests score speaking and writing, what research shows about the strengths and limits of automated scoring, and how the rules governing the technology are developing.

Why tests turned to machines

Scoring open-ended responses is the most expensive and slowest part of language testing. Each essay or spoken answer must be judged against detailed criteria by trained raters. To ensure consistency, high-stakes tests often have responses scored by two raters, with a third called in when they disagree. Recruiting, training and monitoring raters across time zones is costly, and it slows down results.

Automated scoring promised faster results, lower costs and, in principle, greater consistency. A scoring engine applies the same model to every response, at any time of day, without fatigue. Research on automated essay scoring dates back to the 1960s, and large testing organisations began using it operationally from the late 1990s.

Two kinds of systems now dominate. The first are engines built on features of language that can be measured: vocabulary range, grammatical accuracy, sentence complexity, organisation, pronunciation, fluency and pace of speech. Statistical models learn how those features relate to scores given by trained human raters, then predict scores for new responses. The second, more recent kind uses large language models and other neural networks that learn representations of text and speech directly. Testing organisations are researching these models, but are cautious about using them for high-stakes decisions.

In both cases the engine does not "understand" a response the way a person does. It predicts the score a trained human would probably give, based on patterns in large sets of human-scored responses.

How the major tests score speaking and writing

The tests most widely used for international study differ substantially in how much they rely on machines. The table below summarises the arrangements the providers describe publicly.

html

<table>
  <thead>
    <tr>
      <th>Test</th>
      <th>Writing</th>
      <th>Speaking</th>
      <th>Role of human judgement</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>IELTS</td>
      <td>Scored by certified human examiners</td>
      <td>Face-to-face or video interview with a certified human examiner, who also scores it</td>
      <td>Human examiners score all productive skills</td>
    </tr>
    <tr>
      <td>TOEFL iBT</td>
      <td>Combination of trained human raters and the e-rater automated scoring engine</td>
      <td>Human raters, supported by the SpeechRater automated engine</td>
      <td>Human and automated scores combined</td>
    </tr>
    <tr>
      <td>PTE Academic</td>
      <td>Automated scoring of every response; human experts also mark content on specified tasks</td>
      <td>Automated scoring of every response; pronunciation and oral fluency scored by AI only; human review of content on specified tasks</td>
      <td>Human double-marking on 7 of 22 task types since 2025, with a second human resolving disagreements with the AI</td>
    </tr>
    <tr>
      <td>Duolingo English Test</td>
      <td>Automated scoring</td>
      <td>Automated scoring</td>
      <td>Humans are mainly involved in reviewing test security, developing and validating the scoring models</td>
    </tr>
  </tbody>
</table>

Based on the providers' own descriptions of their scoring, as of October 2026.

IELTS has kept human examiners for writing and speaking. Its speaking test is a live interview with an examiner, either in person or by video call, and certified examiners rate writing against published band descriptors. IELTS has made human assessment a central part of how it presents the test.

TOEFL iBT combines human and automated scoring. ETS describes its writing section as scored by both human raters and its e-rater engine. For speaking, ETS has stated that its SpeechRater engine evaluates aspects of delivery and language use. Human raters assess a fuller range of features, including topic development, which ETS says the automated system is more limited in judging.

PTE Academic is the clearest example of AI-first scoring. Pearson describes every response as receiving an AI score. Since a format update announced in July 2025, human experts also double-mark content on seven of the test's 22 task types, including essay writing and summarising. Markers see only the response, not the test taker's identity. Where the human and the AI disagree, a second human makes the final decision. Pronunciation and oral fluency remain scored by AI alone.

The Duolingo English Test, taken online at home, uses automated scoring for its tasks. Its developers describe a system in which test sessions are recorded and reviewed for security, with human proctors involved in that review.

These arrangements are not fixed. Test providers adjust their scoring as their engines and research change, and the details can differ between versions of the same test.

What automated scoring does well

Automated scoring engines have several well-documented strengths.

Consistency. An engine applies the same model to every response. It does not tire, drift over a long scoring session or change its standards from one day to the next. Human raters, however well trained, vary.

Agreement with humans at the aggregate level. Testing organisations validate engines by comparing their scores with those of trained human raters on large samples. For well-defined tasks such as standard essays, published validation studies by test developers report agreement between machine and human scores comparable to agreement between two human raters. That is the main benchmark the industry uses.

Measurable features of speech. Engines can measure some features of speech, such as speaking rate, pauses and pronunciation patterns, with a precision humans cannot easily match. Pearson's decision to leave pronunciation and fluency entirely to AI reflects its view that these are areas where machines are strongest.

Speed and scale. Automated scoring allows results to be delivered within days, sometimes hours, which matters for applicants facing deadlines.

Where automated scoring is weakest

The limitations are equally well documented, and they explain why most test providers still keep humans involved.

Meaning and argument. Engines are strongest at form and weakest at substance. Judging whether an essay actually answers the question, whether an argument is coherent, or whether a spoken summary captures the main points of a lecture requires understanding content. ETS has acknowledged that SpeechRater is limited in evaluating topic development. Pearson's addition of human review in 2025 targeted content in tasks where meaning matters most.

Vulnerability to gaming. Because engines respond to measurable features, test takers and coaches can learn which features are rewarded. The best-known demonstration came from Les Perelman, a former director of writing at the Massachusetts Institute of Technology. In 2014 he and students built a program, the Basic Automatic BS Essay Language Generator, or BABEL. It produced grammatically complex but meaningless essays, which some automated scoring engines rated highly. In the PTE preparation market, memorised response templates are widely taught. They are designed to hit features the scoring engine rewards while saying little.

Fairness across groups. An engine trained on human-scored responses can behave differently for different groups of test takers. In a 2012 study published in Applied Measurement in Education, ETS researchers Brent Bridgeman, Catherine Trapani and Yigal Attali compared e-rater scores with human scores on GRE and TOEFL essays across gender, ethnic and national groups. They found that the machine-human differences were not uniform. Notably, e-rater tended to give higher scores than human raters to essays by test takers from mainland China. Such differences do not establish which score is "right". But they show why automated scoring needs to be checked for each population that takes a test.

Accents and speech recognition. Speech scoring depends on recognising what was said. Automatic speech recognition has historically performed less well on some accents and speech patterns than others. Test developers train their systems on speech from many first-language backgrounds to reduce this, but the extent of remaining differences is rarely published in detail.

Opacity. Test takers usually cannot see how an engine arrived at a score. When an outcome is disputed, explaining why a response received a particular score is harder with a statistical model than with a human rater's notes against published criteria.

html

<table>
  <thead>
    <tr>
      <th>Aspect of performance</th>
      <th>Automated scoring</th>
      <th>Trained human raters</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Consistency across responses</td>
      <td>Very high</td>
      <td>Variable; managed through training, monitoring and double-marking</td>
    </tr>
    <tr>
      <td>Grammar, vocabulary and fluency features</td>
      <td>Strong</td>
      <td>Strong</td>
    </tr>
    <tr>
      <td>Pronunciation and speech timing</td>
      <td>Strong on measurable features; depends on speech recognition across accents</td>
      <td>Strong on intelligibility; less precise on fine timing</td>
    </tr>
    <tr>
      <td>Relevance, argument and content</td>
      <td>Limited</td>
      <td>Strong</td>
    </tr>
    <tr>
      <td>Resistance to templates and gaming</td>
      <td>Weaker</td>
      <td>Stronger</td>
    </tr>
    <tr>
      <td>Speed and cost</td>
      <td>Fast and low marginal cost</td>
      <td>Slower and more expensive</td>
    </tr>
    <tr>
      <td>Explainability of individual scores</td>
      <td>Low</td>
      <td>Higher, through criteria-based judgements</td>
    </tr>
  </tbody>
</table>

A general summary drawn from published research and test-provider documentation. Performance varies by engine, task and population.

Why hybrid scoring has become the norm

The pattern across major tests is not a choice between humans and machines but a division of labour. Automated engines handle what they measure well, such as fluency, pronunciation and surface features of writing, and provide consistency. Humans handle meaning and argument, and act as a check on the engine.

Pearson's 2025 changes illustrate the logic. The test remained AI-first, but human double-marking was added for tasks in which content and meaning carry the most weight. Disagreements between human and machine were resolved by humans. ETS's long-standing combination of e-rater and human raters for writing reflects similar reasoning.

Hybrid systems also produce useful data. Where human and machine scores diverge, the divergence can flag unusual responses, possible template use or weaknesses in the engine, which developers can then investigate.

Generative AI on both sides of the test

Generative AI affects language testing in two directions.

On the scoring side, large language models can produce detailed feedback and apparently plausible scores for essays and transcribed speech. Testing organisations and researchers are studying whether such models can score reliably and fairly in high-stakes settings. The difficulty is not whether a model can produce a score, but whether that score is consistent, valid for the construct the test measures, fair across populations and stable as the model is updated. Those properties have to be demonstrated through validation research before a provider can rely on them for decisions about admission or immigration.

On the test-taker side, generative AI makes cheating easier in unsupervised settings and makes practice materials far more plentiful. Concern about remote testing security is one reason the IELTS partnership withdrew from the UK Home Office's tender for a remotely delivered immigration English test in 2026. It cited the difficulty of maintaining security and integrity in fully remote high-stakes testing.

How regulation is catching up

Automated scoring of language tests has so far been governed mainly by testing standards and institutional practice rather than specific law. Professional standards in educational measurement expect test developers to provide evidence that scores are valid and fair for their intended uses, including when automated scoring is involved. Universities and immigration authorities decide which tests to accept, generally without examining how each test is scored in detail.

That is beginning to change in the European Union. The EU Artificial Intelligence Act lists certain uses of AI in education as high-risk in its Annex III. They include AI systems intended to determine access or admission to educational institutions and systems intended to evaluate learning outcomes. High-risk systems will face requirements on risk management, data quality, documentation, human oversight, accuracy and robustness. According to the European Commission's implementation timeline, as amended by the 2025–2026 Digital Omnibus changes, those obligations for Annex III systems apply from 2 December 2027.

How exactly these provisions will apply to commercial language tests used for admission to EU universities remains to be clarified in guidance and practice. The direction is nonetheless clear. Where automated systems influence access to education, regulators expect documented accuracy, monitoring for bias and meaningful human oversight. Those are the same issues that research on automated scoring has long highlighted.

Outside the EU, there is no comparable binding framework specific to automated scoring in education in the major English-speaking study destinations. Oversight relies on test providers' own validation research, on the decisions of institutions and governments about which tests to accept, and on general laws governing data protection and discrimination.

What test takers and institutions should take from this

For test takers, the most useful lesson is to understand what is being measured. In automated scoring, features such as fluency, pronunciation and grammatical range carry substantial weight. In tests that include human review of content, relevance and coherence matter as well. Memorised templates may appear efficient, but tests with human review are designed to detect content that does not actually address the task.

For universities and other institutions accepting test scores, the method of scoring is part of what a score means. A score produced largely by a machine, one produced largely by humans and one produced by a combination may all be valid. But they reflect different strengths and different vulnerabilities, and institutions setting requirements across several tests should be aware of them.

For policymakers, the evidence points to a balanced position. Automated scoring is neither inherently unreliable nor inherently objective. Its quality depends on the engine, the task, the population and the checks built around it. The most robust systems combine automated consistency with human judgement on meaning, publish validation evidence, and monitor performance across the groups that actually take the test.