Skip to content
All insights

Guide5 min readAahlan team

How to evaluate Arabic language models on your own work.

Public leaderboards tell you how a model does on someone else’s test. Here is how to build one from your own tasks — dialects, mixed scripts, dates and all — and keep using it as models change.

  • Arabic NLP
  • Evaluation
  • Model selection

Choosing a model for Arabic work usually starts with a leaderboard. That is reasonable: it narrows a long list quickly. But the model that ranks highest on a public benchmark is often not the one that best handles your customers’ WhatsApp messages, your scanned contracts or your staff’s half-Arabic, half-English e-mails. The only reliable way to know is to test candidates on your own work. This guide sets out how we build and run that kind of evaluation.

Why public leaderboards are not enough

  • They mostly test Modern Standard Arabic, often in multiple-choice or short-answer formats that look little like real requests.
  • Many Arabic benchmark items are translated from English, and carry English assumptions and phrasing with them.
  • Popular test sets leak into training data over time, so scores can reflect memory rather than ability.
  • They say nothing about your domain vocabulary, your document formats, your tone of voice or your cost per task.

Use them to shortlist two to four models. Then switch to your own evidence.

Build the evaluation set from real tasks

Start from the work the system will actually do. Pull real examples from each use case — customer messages, internal requests, documents, the questions staff ask — with personal data removed or masked. For each example, record the input, what a good answer looks like, and who decided that. Aim for coverage rather than volume: every task type, every channel, the easy cases and the awkward ones that currently go to your most experienced people.

Write the reference answers with the people who do the work today, not only with the technical team. They know which answer is technically correct but would upset a customer, and which shortcut is perfectly acceptable.

Test the Arabic people actually write

Arabic in a Saudi organisation is not one register. A useful set covers at least the categories below. The table gives one example of each and what a good response does with it.

Test categoryExample inputWhat good looks like
Modern Standard ArabicA formal policy question: “ما المدة المسموح بها لتقديم الاعتراض؟”A precise answer in formal Arabic, citing the right clause
Najdi dialect“وش صار على طلبي؟”Understands the question and replies in clear, polite Arabic, without a poor imitation of the dialect
Hijazi dialect“أبغى أغيّر موعدي لبكرة”Recognises a request to move the appointment to tomorrow
Gulf dialect“شلون أطلّع شهادة راتب؟”Explains how to get a salary certificate, rather than reading the words literally
Code-switching“ابي اسوي cancel للـbooking حقي”Treats the English words as part of the request and replies in the user’s main language
Arabizi“abi a3rf wsh sar 3la talabi”Reads Arabic written in Latin letters and digits; replies in Arabic script unless asked otherwise
Numerals and dates“الفاتورة ٤٥٠٠ ريال، تاريخها ١٥ رمضان ١٤٤٧”Handles Eastern Arabic digits and Hijri dates, and converts or keeps them exactly as your policy requires
Names and transliterationMohammed, Muhammad, محمد; Al-Qahtani, Alqahtani, القحطانيMatches the variants to one person and keeps spelling consistent in outputs
Right-to-left formattingA reply that includes an invoice number, an amount and an English product nameNumbers and Latin text appear in the right order; lists and tables render correctly

Include the small mechanics that break search and extraction, too: optional diacritics, letter variants such as أ, ا and إ or ى and ي, and taa marbuta written as ه. A model that stumbles on these will fail quietly in production.

Measure tokenisation cost, not just price

Model prices are quoted per token, but tokenisers differ. Many were built mainly for English and split Arabic words into more pieces; Arabic-first models usually have larger Arabic vocabularies. The same sentence can therefore cost noticeably more on one model than another, and use up more of its context window. Run your evaluation set through each candidate’s tokeniser and record tokens per task in each language. It changes cost comparisons more often than people expect. It matters for self-hosted models too: longer sequences mean slower responses and less throughput from the same hardware.

Score it three ways

No single method is enough. We combine three:

  • A scoring rubric. Define three to five criteria per task — correctness, completeness, faithfulness to sources, language quality, tone — each with a short scale and written examples of every score. Rubrics make reviewers consistent and results comparable between rounds.
  • Pairwise human review. Show reviewers two anonymised answers to the same input and ask which is better, and why. People judge comparisons more reliably than absolute scores, and comparisons surface differences a rubric misses. Use native speakers from the business, and have some items reviewed twice to check that reviewers agree.
  • Automated checks. Exact-match and field-level checks for extraction, format validation for JSON and dates, citation checks for retrieval answers, and language detection to catch replies in the wrong language. A model can also grade other models against the rubric, but calibrate it against human scores first and keep checking: automated judges have their own biases, in Arabic as elsewhere.

Keep testing when models change

An evaluation set is most valuable after launch. Models are updated, vendors retire versions, prompts are edited and retrieval indexes grow. Any of these can improve one task while quietly breaking another.

  1. 1Freeze a versioned copy of the set, with the scores of the model currently in production as the baseline.
  2. 2Rerun it automatically on every change to the model, prompt, tokeniser or retrieval settings.
  3. 3Set thresholds per category, so a drop in dialect handling is not hidden by a gain in MSA.
  4. 4Add every production failure users report to the set, once it has been reviewed and anonymised.
  5. 5Review the set itself each quarter, retiring stale items and adding new task types.

Done this way, model selection stops being a one-off bet and becomes a routine decision you can repeat whenever a better or cheaper option appears. Building and maintaining Arabic evaluation sets is part of how we run the models we deploy, and we can help your team set up its own.

Want to talk this through?

We are happy to discuss how this applies to your organisation — no sales script.

Newsletter

Practical AI, in your inbox.

Real use cases, Saudi regulations and what works in Arabic — once or twice a month. No spam; unsubscribe any time.

We’ll e-mail you to confirm. Unsubscribe any time — see our privacy policy.