First, decide what you are fixing
"Make the model better at Khmer" is too broad to plan data for. Start by testing a few candidate models on the tasks your product needs and writing down how they fail. Common Khmer problems include:
- Grammatically correct but unnatural sentences that read like a translation from English.
- Wrong register: too formal for a chat app, too casual for government information, or the wrong vocabulary when speaking about monks or royalty.
- Weak local knowledge: Cambodian places, institutions, holidays, laws and everyday practices.
- Poor handling of numbers, dates, Khmer digits and the Buddhist calendar.
- Switching to English, or mixing scripts, in the middle of an answer.
- Spelling and Unicode errors in generated text.
Each problem points to a different kind of data. Fixing tone and format calls for instruction data; fixing knowledge may call for more Khmer text or retrieval; fixing safety calls for refusal and policy examples.
The four kinds of data
1. Clean Khmer text for continued pre-training
If the base model has seen little Khmer, continued pre-training on a large Khmer corpus improves fluency before any instruction tuning. The text must be clean: legacy fonts converted, Unicode ordering fixed, zero-width spaces handled consistently, duplicates and boilerplate removed, and machine-translated pages filtered out where possible. Most available Khmer web text is news and social media, so add other genres (education, health, government, business, conversation) if your users need them. Segmentation tools help with cleaning and deduplication (see Khmer word segmentation).
2. Instruction pairs for supervised fine-tuning (SFT)
SFT data is a set of prompts and ideal responses that show the model how to behave. For Khmer, the pairs should be written by native speakers in natural Khmer, not translated from English datasets. Translated data teaches the model translated-sounding Khmer and Western examples that do not fit Cambodian users.
A good Khmer SFT set covers:
- The tasks your product performs: answering questions, summarising, drafting messages, explaining forms, customer support, and so on.
- A range of lengths, difficulty levels and formats (lists, step-by-step instructions, short answers).
- Different registers and audiences, with clear guidance on which to use when.
- Realistic user input, including spelling mistakes, Romanized Khmer and mixed Khmer–English prompts.
- Multi-turn conversations, not only single questions.
Quality matters more than size. A few thousand carefully written and reviewed pairs often help more than a much larger set of mediocre ones.
3. Preference data for RLHF or DPO
Preference data shows the model which of two answers is better. Native Khmer raters compare model outputs and choose, ideally with a short reason, based on a written rubric: accuracy, fluency, register, helpfulness and safety. This is how you teach subtle qualities that are hard to write as rules, such as when a reply sounds polite but natural. Raters must be fluent enough to notice unnatural phrasing, not just factual errors.
4. Safety and policy data
Your model needs examples of what to refuse and how to refuse politely in Khmer, covering local sensitivities as well as general harms. Include borderline prompts so the model does not refuse harmless requests.
Build a Khmer evaluation set before you train
Without a fixed test set you cannot tell whether fine-tuning helped. Build one first, from prompts the model will never train on, that mirrors real use:
- Prompts for each target task, written by native speakers.
- Reference answers or a scoring rubric for each.
- Specific checks for the weaknesses you found: register, numbers, local knowledge, staying in Khmer.
Score with native Khmer reviewers, at least on a sample. Automatic metrics and LLM-as-judge methods are useful for speed, but they are not yet reliable judges of Khmer fluency on their own. Our evaluation and benchmarking service builds and scores Khmer test sets of this kind.
Don't forget the tokenizer
Many tokenizers were built mostly from English and other large languages and split Khmer text into many small pieces. That makes Khmer more expensive to process, shortens the effective context window, and can slow learning. When choosing a base model, compare how many tokens the same Khmer paragraph takes in each. For a large project, extending the tokenizer with Khmer tokens may be worth it, but it also requires more training data to learn the new tokens well.
Practical tips
- Write a style guide for SFT writers and raters: tone, register, how to write numbers and English terms, and how long answers should be.
- Normalise all text the same way before training, including prompts, responses and evaluation data.
- Review everything twice. A second Khmer reviewer catches errors and drift between writers.
- Keep a held-out set and never let it leak into training data.
- Iterate. Train, evaluate, find the remaining failures, then write data that targets them.
- Consider retrieval for facts that change, such as prices, laws or schedules, rather than training them into the model.
Where to get Khmer LLM data
Our Khmer LLM training data service provides native-written SFT pairs, preference ratings and safety data with double review. For cleaner corpora and translation work, our Khmer–English parallel text and segmentation corpus are ready-made options, and Khmer AI engineers and prompters can work inside your team on prompts and evaluation. We start with a free pilot, so you can test the data on your own model first.