evaluate
Evaluate a trained model against a held-out gold corpus and print quality
metrics. Unlike the in-sample metrics printed by train, this measures
quality on text the model has never seen.
A gold corpus is a text file containing the correct answers: one
sentence per line, already segmented into the correct tokens by human
annotation (the same file formats used for training corpora — space- or
tab-separated tokens, or word/POS with --pos). “Gold” refers to the
gold standard the model’s output is judged against. For a meaningful
held-out evaluation it must contain sentences that were not used to
train the model — the bundled files in resources/eval/ are the UD GSD
test splits, while the bundled models are trained on the train splits.
Usage
litsea evaluate [OPTIONS] <MODEL_URI> <GOLD_FILE>
Arguments
| Argument | Description |
|---|---|
MODEL_URI | Path or URL to the trained model file. Supports: local file paths, file://, http://, https:// |
GOLD_FILE | Path to the gold corpus (one sentence per line) |
Options
| Option | Default | Description |
|---|---|---|
-l, --language <LANGUAGE> | japanese | Language of the model and gold corpus. Accepts: japanese / ja, chinese / zh, korean / ko, english / en |
--pos | off | Evaluate segmentation + POS tagging. Requires a two-stage model (train --pos). Combines with --format below to select the gold format |
--format <FORMAT> | space | Gold corpus format. Without --pos: space (space-separated tokens) or tsv (tab-separated tokens; a token may be a literal space, as in the Korean/English space-preserving corpus). With --pos: space selects "word/POS word/POS ..." (the two-stage training corpus format, unspaced) and tsv selects tab-separated "word/POS" tokens where a token may also be a literal space (issues #196/#198; the space-preserving format the Korean and English two-stage models are now trained on, so for those languages this is the protocol that matches both training and real input) |
Metrics
Two token sequences are compared for every sentence:
- Gold tokens – the reference segmentation from the gold corpus: the human-annotated correct answer (here, the tokenization of the UD GSD treebank test split). The evaluated sentence text is reconstructed by concatenating them.
- Predicted tokens – what the model produces when the reconstructed
sentence text is fed to
segment(orsegment --pos), exactly as a user would at inference time.
Predicted and gold tokens are matched by exact character-offset spans over the reconstructed sentence. Pure-whitespace tokens are excluded from scoring, so the Korean/English space-preserving protocol does not inflate the numbers.
| Metric | Measures | A low value means |
|---|---|---|
| Word Precision | Of the predicted words, the fraction that exactly matches a gold word (both ends correct) | many spurious words: over-segmentation or wrongly merged words |
| Word Recall | Of the gold words, the fraction recovered exactly | many gold words missed |
| Word F1 | Harmonic mean of word precision and recall | overall segmentation quality |
| Boundary Precision | Of the predicted word-start positions, the fraction that is a gold boundary | many false boundaries (over-segmentation) |
| Boundary Recall | Of the gold word-start positions, the fraction found | many missed boundaries (under-segmentation) |
| Boundary F1 | Harmonic mean of boundary precision and recall | overall boundary quality |
Tagged Word Precision / Recall / F1 (--pos) | Like the word metrics, but the predicted POS tag must also match | correct spans carrying wrong tags |
A word counts as correct only when both of its boundaries are correct,
so word metrics are always at least as strict as boundary metrics — a
single misplaced boundary invalidates the two words on either side of it.
Sentences is the number of evaluated (non-empty) gold sentences.
Examples
Reproduce the documented held-out figures with the bundled gold data
(resources/eval/, converted from the UD GSD test splits):
litsea evaluate -l japanese models/japanese.model resources/eval/japanese_gsd_test.txt
litsea evaluate -l korean --format tsv models/korean.model resources/eval/korean_gsd_test.tsv
litsea evaluate -l chinese models/chinese.model resources/eval/chinese_gsd_test.txt
litsea evaluate -l english --format tsv models/english.model resources/eval/english_ewt_test.tsv
litsea evaluate --pos -l japanese models/japanese_pos.model resources/eval/japanese_gsd_test_pos.txt
The Korean and English two-stage models are trained on the
space-preserving corpus (issue #198), so evaluate them with --pos --format tsv against the *_pos_spaced.tsv gold – that protocol matches
both their training and the spaced text segment --pos receives in
practice:
litsea evaluate --pos --format tsv -l korean models/korean_pos.model resources/eval/korean_gsd_test_pos_spaced.tsv
litsea evaluate --pos --format tsv -l english models/english_pos.model resources/eval/english_ewt_test_pos_spaced.tsv
The *_pos.txt gold for those two languages measures the unspaced
protocol they used before #198; it is kept only for reproducing the older
published numbers (see
Pre-trained Models).
Output:
Evaluation Metrics:
Sentences: 543
Word Precision: 96.73%
Word Recall: 96.66%
Word F1: 96.70%
Boundary Precision: 98.63%
Boundary Recall: 98.56%
Boundary F1: 98.59%