WER vs. CER: How to Measure Arabic ASR Accuracy

How is accuracy measured in Automatic Speech Recognition (ASR)? The two most common metrics are Word Error Rate (WER) and Character Error Rate (CER). For a language like English, these metrics are relatively straightforward. For Arabic, their interpretation depends heavily on linguistic and evaluation choices.

Choosing an ASR vendor based on a single, misleading accuracy score can lead to deploying a system that fails in the real world. This article breaks down what WER and CER are, explains why standard metrics fall short for the Arabic language, and provides a framework for a more intelligent and accurate assessment of Arabic ASR performance.

The Mechanics of Measurement: WER and CER

At their core, both WER and CER are based on the Levenshtein distance, a formula that calculates the minimum number of edits required to change one sequence into another. The formula is:

Error Rate = (Substitutions + Deletions + Insertions) / Total Number of Units

• Substitutions (S): A word/character is replaced (e.g., reference is "thus," ASR output is "this").
• Deletions (D): A word/character is missed (e.g., reference is "this is a test," ASR output is "is a test").
• Insertions (I): A word/character is added (e.g., reference is "this is a test," ASR output is "this is a the test").

The only difference is the unit of measurement: WER uses words, and CER uses characters. A lower score is better.

The Arabic Challenge: Why WER Requires Careful Scoring

Applying WER to Arabic is not a simple matter of translation. The language’s unique structure presents three fundamental challenges that can distort accuracy measurements.

1. Morphological richness: one written token, several grammatical elements

Arabic is a morphologically rich language. Words are typically formed from a three-letter root that is combined with various patterns to create different meanings. Furthermore, Arabic uses a variety of clitics, which are functional particles like prepositions, conjunctions, and pronouns that attach to the beginning or end of a word. For example, the single written word "وسيكتبونها" (wasayaktubūnahā) translates to "and they will write it"." This single token in Arabic corresponds to five distinct words in English.

This structure creates a significant ambiguity in word segmentation.

• Should "وسيكتبونها" be treated as one word or as multiple morphemes?

Different ASR systems and annotation standards may adopt different tokenization schemes. An ASR system that separates clitics will produce a different word count from one that does not, leading to inconsistent WER calculations. A system might correctly identify all the component morphemes but still be heavily penalized by WER if the reference transcription treats the entire token as a single word.

2. The Diacritics Dilemma (The Vowel Blind Spot)

Standard written Arabic is typically undiacritized, meaning it omits the short vowel marks essential for pronunciation.

The word "كتب" can be read as

• kataba (he wrote),
• kutiba (it was written),
• or kutub (books).

Arabic reference transcripts may be diacritized or undiacritized. That choice materially changes the evaluation: scoring against undiacritized text ignores short-vowel distinctions, while scoring diacritics introduces character-level penalties that may not reflect the intended downstream use case.

Vendors should state clearly whether diacritics are retained, removed, or normalized before scoring.

3. Dialectal Variation (The "Which 'Now' Do You Mean?" Problem)

The Arab world is characterized by diglossia, the coexistence of Modern Standard Arabic (MSA) with dozens of regional dialects. A spoken utterance may have multiple valid transcriptions. For example, the concept of "now" can be the following:

• al-ʾān (MSA)
• dilwaʾti (Egyptian)
• hallaʾ (Levantine)

Dialectal variation creates substantial challenges because spoken forms, spelling conventions, and code-switching differ across regions. A system should be evaluated against references that reflect the target dialect and domain. If the business task permits semantic rather than verbatim equivalence, add task-specific evaluation (such as intent, entity, or keyword accuracy) rather than treating synonym substitutions as correct ASR output.

The Role of CER in Arabic ASR Evaluation

Character Error Rate (CER) is a valuable complementary metric for Arabic ASR because it is less sensitive to word-boundary and segmentation differences. It can help distinguish systems that produce near-correct character sequences from those that make broader transcription errors.

CER does not resolve every evaluation challenge. It remains affected by choices around diacritics and character normalization, and it may understate errors that change a complete word, a person or organization name, a number, or the meaning required by a downstream workflow. For that reason, CER should be reported alongside WER and task-specific measures where relevant.

How to Properly Evaluate an Arabic ASR Vendor

To get a meaningful assessment of an Arabic ASR system, you need to go beyond a single headline number. Here are four best practices:

1. Report Both WER and CER: Each metric captures different failure modes. Reporting both provides a more complete picture of system performance. A gap between WER and CER can help identify whether word boundaries, tokenization, or character-level errors are influencing the result. It should be interpreted alongside the scoring protocol and test-set characteristics.
2. Specify Normalization and Tokenization: Any published results must be accompanied by a detailed description of the pre-processing steps applied to both the reference and hypothesis texts. This includes the tokenization scheme (e.g., separating clitics), the handling of diacritics (e.g., stripping them), and the normalization of characters (e.g., unifying different forms of the letter alif).
3. Use Morpheme-Based Evaluation: For a more linguistically sound evaluation, consider decomposing words into their constituent morphemes before calculating the error rate. This provides a more granular assessment of performance and rewards systems that correctly identify morphemes even if the full word form is incorrect.
4. Account for Dialectal Variation: Whenever possible, use references and scoring rules that reflect the target dialect, accepted spelling variants, and code-switching conventions. If this is not feasible, performance should be reported separately for different dialects to avoid penalizing systems for dialect-specific accuracy.

Moving Beyond a Single Metric

Measuring Arabic speech recognition quality is not solved by a single headline percentage. WER and CER each reveal useful but incomplete information: WER reflects word-level transcription errors, while CER is less sensitive to tokenization and word-boundary differences.

A credible Arabic ASR evaluation should report both metrics, disclose text-normalization and tokenization rules, and break results down by dialect, domain, channel, noise level, and code-switching. For production use cases, add task-level measures such as named-entity accuracy, number and date accuracy, keyword recall, or intent accuracy.

The goal is not to choose between WER and CER. It is to measure the capabilities that matter for the audio, dialects, and workflow your organization actually uses.