AI translation is now a normal part of localization. Most teams are no longer asking whether to use machine translation or LLMs. They are asking a harder question: how do we know which translations are good enough to publish?
Reading every segment defeats the point of automating. Publishing everything unchecked is a risk to your brand. Quality estimation, usually shortened to QE, sits between those two options. This guide explains what QE is, how it compares with other quality measures, and how to put it to work in a real workflow.
What is translation quality estimation?
Quality estimation predicts how good a translation is without a reference translation. A QE model reads the source sentence and the machine-translated output, then returns a score (and, in some models, the likely error spans) that approximates how a human reviewer would judge it.
That “no reference” detail is what makes QE useful in production. When you translate a new product page or support article, there is no perfect human translation to compare against. QE gives you a signal anyway, segment by segment, as the translation happens.
Why traditional metrics aren’t enough
Teams often start with metrics such as BLEU or chrF. These compare the machine output with a human reference translation and reward overlap in words or characters. They are useful for benchmarking engines in a controlled test, but they have limits in daily work:
- They need a reference. You only have one after a human has already translated the content, which is the work you wanted to avoid.
- They penalize valid alternatives. A correct translation that uses different wording from the reference can score poorly.
- They work on test sets, not live content. They tell you which engine did better last month on sample data, not whether this paragraph is safe to publish.
Newer reference-based neural metrics such as COMET correlate better with human judgment, but they still depend on having a reference. For live content, you need a reference-free approach.
The three layers of translation quality measurement
Strong programs do not pick one method. They combine three layers, each answering a different question.
| Layer | Method | Question it answers | Needs a human? |
| 1. Automated rules | Checks for numbers, tags, placeholders, glossary terms, formatting | Is anything mechanically broken? | No |
| 2. Quality estimation | QE model scores each segment | How likely is this segment to be good? | No |
| 3. Human evaluation | Error-based review such as MQM, or post-editing effort | How good is it, really, and where does it fail? | Yes |
QE is the middle layer. It does not replace rule-based checks, and it does not replace human review. It tells you where to spend human attention.
How QE works in practice
Most current QE approaches fall into two groups.
1. Trained QE models. Models such as the CometKiwi family, xCOMET and MetricX in reference-free mode are trained on human quality judgments. Given a source and a translation, they output a score, and some can also highlight suspect words or phrases. They are fast and cheap enough to run on every segment.
2. LLM-based judges. A large language model is prompted to rate a translation or list its errors using a defined rubric. This can be flexible and can explain its reasoning, but it adds cost and latency, and results can vary with the prompt and the model. If you use this approach, fix the prompt, test it against human labels, and re-test whenever the model changes.
Either way, treat the output as an estimate, not a measurement. QE models are strongest at catching fluency problems and obvious meaning errors. They are weaker at subtle issues such as wrong terminology, small omissions, tone, and domain-specific meaning.
A practical QE workflow, step by step
Step 1: Define what “good enough” means for each content type
A legal notice, a medical leaflet, a game dialogue line, and an internal wiki page do not need the same quality bar. Group your content by risk and decide the acceptable outcome for each: publish as is, light review, or full human review.
Step 2: Run deterministic checks first
Before any scoring, catch what rules can catch: missing or changed numbers, broken placeholders and tags, glossary violations, inconsistent product names, and length limits for UI strings. These errors are cheap to detect and expensive to publish.
Step 3: Score every segment with QE
Run the QE model across all machine-translated segments. Keep the score alongside the segment in your translation management system so reviewers and reports can use it later.
Step 4: Calibrate against human judgment
This step is the one most teams skip. QE scores are not comparable across models or language pairs, so a “0.85” means different things in different setups. Take a sample of segments per language pair and content type, have linguists rate them, then check how well the QE score predicts those ratings. Use that data to choose your thresholds instead of guessing.
Step 5: Route segments by score
Once calibrated, build simple routing bands:
- High score: auto-approve, or approve with spot checks.
- Medium score: send for light post-editing.
- Low score: send for full human translation or a different engine.
Routing is where QE pays for itself. Human effort goes to the segments that need it, and review time on clean output drops.
Step 6: Audit the high-scoring segments
Every QE model has blind spots. Pull a random sample from the auto-approved bucket on a regular schedule and have a linguist review it. If errors show up in that sample, tighten your thresholds or adjust the model.
Step 7: Close the loop
Feed reviewer corrections back into your process. Track which error types QE missed, update glossaries and translation memories, and revisit thresholds when you change engines, add a language, or move into a new domain.
Metrics worth tracking
QE is most useful when it connects to numbers your team and your clients already care about:
- Post-editing effort: how much reviewers change the machine output, often measured as edit distance or time per segment.
- Error rate by severity: minor, major, and critical errors per thousand words, using an error-typology approach such as MQM.
- QE-to-human agreement: how often the QE band matches the linguist’s verdict. If this drops, recalibrate.
- Escape rate: errors found in content that QE approved, whether from audits or customer reports.
- Cost and turnaround by band: how much time and money each routing band consumes.
Common mistakes to avoid
- Trusting a single score as truth. A high QE score lowers risk. It does not remove it.
- Using one threshold for every language. Models perform differently across language pairs, so calibrate each pair separately.
- Ignoring content risk. A score that is fine for a help article may not be fine for safety instructions or regulated content.
- Skipping rule-based checks. QE models can miss a changed number or a broken tag that a simple script would catch.
- Never re-validating. When you swap an MT engine or an LLM, your old thresholds may no longer hold.
Where QE fits in a modern localization stack
QE works best when it is built into the workflow instead of run as a separate, manual step. In a translation management system, that means scores attach to segments automatically, routing rules assign work by band, and reviewers see where to focus. It also pairs naturally with managed AI translation, where project managers and linguists oversee machine output instead of translating from scratch. In that model, QE is the dial that decides how much human involvement each piece of content receives.
At MarsHub, this is the idea behind combining AI-powered QA with human expertise: automation handles the volume, and linguists handle the judgment calls.
Quick checklist
- ☐ Content grouped by risk level, with a quality target for each
- ☐ Rule-based checks running before scoring
- ☐ QE scores stored per segment
- ☐ Thresholds calibrated with linguist ratings, per language pair
- ☐ Routing bands defined and automated
- ☐ Regular audits of auto-approved content
- ☐ Feedback loop to glossaries, TMs and engine choices
Final thoughts
Measuring AI translation quality is not about finding one perfect number. It is about building a system that estimates quality cheaply, verifies it with people where it matters, and improves over time. Start small: calibrate on one language pair and one content type, set simple routing bands, and audit the results. Once you can show that QE sends the right segments to the right reviewers, you can expand with confidence.
Want help deciding how much of your AI translation needs human review? Talk to the MarsHub team about a pilot.
contact@marshub.com 
