Choosing a model for Arabic work usually starts with a leaderboard. That is reasonable: it narrows a long list quickly. But the model that ranks highest on a public benchmark is often not the one that best handles your customers’ WhatsApp messages, your scanned contracts or your staff’s half-Arabic, half-English e-mails. The only reliable way to know is to test candidates on your own work. This guide sets out how we build and run that kind of evaluation.
Why public leaderboards are not enough
- They mostly test Modern Standard Arabic, often in multiple-choice or short-answer formats that look little like real requests.
- Many Arabic benchmark items are translated from English, and carry English assumptions and phrasing with them.
- Popular test sets leak into training data over time, so scores can reflect memory rather than ability.
- They say nothing about your domain vocabulary, your document formats, your tone of voice or your cost per task.
Use them to shortlist two to four models. Then switch to your own evidence.
Build the evaluation set from real tasks
Start from the work the system will actually do. Pull real examples from each use case — customer messages, internal requests, documents, the questions staff ask — with personal data removed or masked. For each example, record the input, what a good answer looks like, and who decided that. Aim for coverage rather than volume: every task type, every channel, the easy cases and the awkward ones that currently go to your most experienced people.
Write the reference answers with the people who do the work today, not only with the technical team. They know which answer is technically correct but would upset a customer, and which shortcut is perfectly acceptable.
Test the Arabic people actually write
Arabic in a Saudi organisation is not one register. A useful set covers at least the categories below. The table gives one example of each and what a good response does with it.
| Test category | Example input | What good looks like |
|---|---|---|
| Modern Standard Arabic | A formal policy question: “ما المدة المسموح بها لتقديم الاعتراض؟” | A precise answer in formal Arabic, citing the right clause |
| Najdi dialect | “وش صار على طلبي؟” | Understands the question and replies in clear, polite Arabic, without a poor imitation of the dialect |
| Hijazi dialect | “أبغى أغيّر موعدي لبكرة” | Recognises a request to move the appointment to tomorrow |
| Gulf dialect | “شلون أطلّع شهادة راتب؟” | Explains how to get a salary certificate, rather than reading the words literally |
| Code-switching | “ابي اسوي cancel للـbooking حقي” | Treats the English words as part of the request and replies in the user’s main language |
| Arabizi | “abi a3rf wsh sar 3la talabi” | Reads Arabic written in Latin letters and digits; replies in Arabic script unless asked otherwise |
| Numerals and dates | “الفاتورة ٤٥٠٠ ريال، تاريخها ١٥ رمضان ١٤٤٧” | Handles Eastern Arabic digits and Hijri dates, and converts or keeps them exactly as your policy requires |
| Names and transliteration | Mohammed, Muhammad, محمد; Al-Qahtani, Alqahtani, القحطاني | Matches the variants to one person and keeps spelling consistent in outputs |
| Right-to-left formatting | A reply that includes an invoice number, an amount and an English product name | Numbers and Latin text appear in the right order; lists and tables render correctly |
Include the small mechanics that break search and extraction, too: optional diacritics, letter variants such as أ, ا and إ or ى and ي, and taa marbuta written as ه. A model that stumbles on these will fail quietly in production.
Measure tokenisation cost, not just price
Model prices are quoted per token, but tokenisers differ. Many were built mainly for English and split Arabic words into more pieces; Arabic-first models usually have larger Arabic vocabularies. The same sentence can therefore cost noticeably more on one model than another, and use up more of its context window. Run your evaluation set through each candidate’s tokeniser and record tokens per task in each language. It changes cost comparisons more often than people expect. It matters for self-hosted models too: longer sequences mean slower responses and less throughput from the same hardware.
Score it three ways
No single method is enough. We combine three:
- A scoring rubric. Define three to five criteria per task — correctness, completeness, faithfulness to sources, language quality, tone — each with a short scale and written examples of every score. Rubrics make reviewers consistent and results comparable between rounds.
- Pairwise human review. Show reviewers two anonymised answers to the same input and ask which is better, and why. People judge comparisons more reliably than absolute scores, and comparisons surface differences a rubric misses. Use native speakers from the business, and have some items reviewed twice to check that reviewers agree.
- Automated checks. Exact-match and field-level checks for extraction, format validation for JSON and dates, citation checks for retrieval answers, and language detection to catch replies in the wrong language. A model can also grade other models against the rubric, but calibrate it against human scores first and keep checking: automated judges have their own biases, in Arabic as elsewhere.
Keep testing when models change
An evaluation set is most valuable after launch. Models are updated, vendors retire versions, prompts are edited and retrieval indexes grow. Any of these can improve one task while quietly breaking another.
- 1Freeze a versioned copy of the set, with the scores of the model currently in production as the baseline.
- 2Rerun it automatically on every change to the model, prompt, tokeniser or retrieval settings.
- 3Set thresholds per category, so a drop in dialect handling is not hidden by a gain in MSA.
- 4Add every production failure users report to the set, once it has been reviewed and anonymised.
- 5Review the set itself each quarter, retiring stale items and adding new task types.
Done this way, model selection stops being a one-off bet and becomes a routine decision you can repeat whenever a better or cheaper option appears. Building and maintaining Arabic evaluation sets is part of how we run the models we deploy, and we can help your team set up its own.