Map of the Levant with the word قلتلك drawn large across it. Five dots mark the varieties, each linked to its pronunciation: Lebanese ʾeltillak; Syrian ʾəltillak; Palestinian Rural kultlak; Palestinian Urban ʾultillak; Jordanian (m) gultlak, (f) ʾultillak. — **FIGURE 1** SHAMS includes audio and text for the Levantine Arabic varieties shown in the map above. Above, we illustrate different pronunciations of the word قلتلك (colloquial spelling, meaning *I told you*) to show the importance of our phonetic annotations; these pronunciation differences are not indicated in Arabic orthography, even with conventional diacritics.

SHAMS comprises

1,300utterances
2+hoursof speech
15,506words
52.8/47.2male/female share of words (%)
93.2%diacritic agreement, 150-utterance subset

TL;DR

We introduce SHAMS, an evaluation benchmark for Arabic NLP and speech technology that ties diacritization, G2P, ASR, and A2P together over the same utterances, grounded in audio and stratified by LA variety.

Abstract

Levantine Arabic (LA) is spoken by tens of millions of people, creating a pressing need for shared benchmarks to evaluate LA speech-language technologies. Evaluating such technology is particularly challenging given LA’s internal diversity and its opaque and non-standardized orthography.

We present SHAMS (SHami Annotated Multi-dialect Speech), a benchmark comprising 1,300 utterances drawn from open audio corpora, balanced across five LA varieties (Urban and Rural Palestinian, and Urban Jordanian, Lebanese, and Syrian). Each utterance is represented across four aligned tiers: audio, unvocalized orthography, diacritized text, and phonetic transcription. This structure supports evaluation of various downstream tasks such as diacritization, grapheme-to-phoneme conversion, automatic speech recognition, and audio-to-phoneme, grounded in audio and stratified by variety. We benchmark open and proprietary models across these tasks to demonstrate the utility of this benchmark for measuring progress across LA.

Contributions

A balanced spoken corpus of 1,300 utterances drawn from open audio sources, covering five Levantine varieties
Multi-tier annotations aligned to each utterance consisting of diacritized Arabic and phonetic transcription, thus enabling joint evaluation of diacritization, G2P, ASR, and A2P.
Baseline results across open and proprietary models, stratified by variety, demonstrating the benchmark’s utility for distinguishing model performance across tasks and varieties.

Four aligned tiers

Each utterance in SHAMS includes four aligned tiers:

(i)The audio segment
(ii)Unvocalized Arabic orthography
(iii)Fully diacritized Arabic
(iv)Phonetic transcription

Four tasks

Diacritizationrestoring vowel and gemination marks
G2Pmapping orthography to phonetic transcription
ASRtranscribing audio to text
A2Ptranscribing audio to phonetic transcription

G2P: grapheme-to-phoneme conversion. ASR: automatic speech recognition. A2P: audio-to-phoneme conversion.

Composition

SHAMS comprises 1,300 utterances totaling roughly 15,500 words and just over two hours of speech, spread across five Levantine varieties. The benchmark is built on the Levantine portions of two open multi-dialectal Arabic speech corpora, Casablanca and LangMedia.

Table 1 · Utterances, words, and duration in minutes per variety. Palestinian Urban and Jordanian each draw on both source corpora; Palestinian Rural comes from Casablanca only, and Lebanese and Syrian from LangMedia only.
Variety	Source	Utt.	Words	Min.
Pal./Urban	both	230	3,175	25.0
Pal./Rural	Casablanca	234	3,153	21.8
Jordanian	both	305	3,173	25.5
Lebanese	LangMedia	316	3,008	29.1
Syrian	LangMedia	215	2,997	26.7
Total		1,300	15,506	128.2

Citation

@misc{sapirstein2026shams,
  title = {{SHAMS}: An Audio-Grounded Pronunciation
           Benchmark for {L}evantine {A}rabic},
  author = {Sapirstein, Ben and Mattar, Roy and
            Mor-Lan, Guy and Mohamed, Ahlam and
            Cerqueglini, Letizia and Alper, Morris},
  year = {2026},
  eprint = {2610.01427},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2610.01427}
}