What word segmentation means in Khmer
In English, spaces mark word boundaries. In Khmer, spaces are used more loosely, roughly where a writer would pause, between phrases or clauses. A sentence such as "I want to open a bank account" is written as one unbroken string of letters, and the reader splits it into words from knowledge of the language.
Word segmentation is the task of making those boundaries explicit, so software can work with words instead of an undivided stream of characters. It sounds like a small pre-processing step. In practice it is one of the main reasons Khmer is harder for AI than many larger languages.
Why it matters for AI systems
Search and information retrieval
A search engine that cannot find word boundaries either matches fragments in the middle of unrelated words or misses the right document entirely. Good segmentation is the difference between a Khmer search box that works and one users give up on.
Classic NLP tasks
Named entity recognition, part-of-speech tagging, sentiment analysis and keyword extraction all assume you know what the words are. A segmentation error at the start of a pipeline becomes a wrong entity or a wrong label at the end.
Speech recognition
Speech models output text, so their transcripts need consistent word boundaries too. If the training transcripts are segmented one way and your evaluation data another, word error rate (WER) changes even when the model has not. That is why Khmer ASR teams usually also report character error rate (see how to build a Khmer ASR dataset).
Large language models
Modern LLMs use subword tokenizers rather than a word list, so they do not need explicit spaces to work. But segmentation still matters: it is used to clean and deduplicate training corpora, to count and filter text, to build evaluation sets, and to analyse why a model performs badly on Khmer. Many general tokenizers also split Khmer into far more tokens than an equivalent English sentence, which raises cost and shortens the usable context.
Why Khmer segmentation is hard
- Compounds. Many Khmer words are built from smaller words. Whether a compound is one word or two is often a matter of convention, and different dictionaries and annotators disagree.
- Loanwords and names. Pali, Sanskrit, French and English loanwords, and people's and place names, may not be in any dictionary and can be spelled several ways.
- Stacked consonants and Unicode order. Khmer writes some consonants as subscripts below others. The same visible word can be stored in Unicode in different character orders, so a word "missing" from a dictionary may simply be encoded differently.
- Hidden characters. Web text often contains zero-width spaces that some writers add to help line breaking, inconsistently. They can look like word boundaries but are not reliable ones.
- Legacy fonts. Older documents typed with pre-Unicode fonts must be converted before any tool can read them.
- Informal text. Social media mixes Khmer with English, uses Romanized Khmer, and skips the spelling conventions found in formal text.
How segmentation tools work, and where they fail
There are three broad approaches:
- Dictionary-based methods, such as the one built into the ICU library used by many operating systems and browsers, match text against a word list. They are fast and predictable, but fail on words that are not in the list.
- Statistical and neural models, such as conditional random fields or small neural networks (open-source examples include khmer-nltk), learn boundaries from labelled text and cope better with unknown words.
- Subword tokenizers in LLMs learn frequent character sequences from data without any linguistic rules.
All of them depend on data. Dictionary methods need a good word list; learned models need a corpus segmented by people. Most public Khmer text is news and social media, so tools trained on it do well on news and noticeably worse on medical, legal, financial or conversational text. If your product works in one of those domains, general tools will make mistakes on exactly the words that matter most to you.
What good segmentation training data looks like
A written standard
Before annotation starts, write a guideline that answers the hard questions: how to treat compounds, reduplication, particles, numbers, names, abbreviations and English words. Include examples for each rule. Without this, every annotator follows their own instinct and the model learns noise.
Clean, normalised text
Convert legacy fonts, fix Unicode ordering, and remove or standardise zero-width spaces before annotation, so the annotators see the same characters the model will see.
Domain coverage
Include text from the domains you care about, not only news. A few thousand well-chosen sentences from your domain can matter more than a much larger general corpus.
Double annotation and agreement
Have two trained Khmer annotators segment the same sample independently and measure how often they agree. Low agreement points to unclear rules. A reviewer resolves disagreements and updates the guideline.
Useful extras
The same pass can add part-of-speech tags or named entities (people, places, organisations, dates, amounts), which makes the corpus useful for more than one model.
How to measure segmentation quality
Compare the tool's output with a held-out, human-segmented test set and report precision, recall and F1 on word boundaries. Check results per domain and on sentences containing names and loanwords, where errors cluster. A small, carefully reviewed test set from your own data is worth more than a high score on someone else's benchmark.
Getting segmentation data for your project
Our Khmer word segmentation and NER corpus is a ready-made starting point. For your own domain, our Khmer data annotation team segments and labels text to a written standard with double review, and our text and LLM data service prepares clean Khmer corpora for language-model training. We start every project with a free pilot so you can check the labels before you commit.