extract

Extract features from a corpus file for model training.

Usage

litsea extract [OPTIONS] <CORPUS_FILE> <FEATURES_FILE>

Arguments

Options

Corpus Format

The input corpus must have words separated by spaces, one sentence per line:

Litsea は TinySegmenter を 参考 に 開発 さ れ た 。
Rust で 実装 さ れ た コンパクト な 単語 分割 ソフトウェア です 。

TSV Corpus Format (--format tsv)

With --format tsv, tokens are separated by tab characters and a token may be a literal space " ". This preserves the original spacing of the sentence in the training text, which is essential for languages like Korean and English where spaces mark most word boundaries (see Korean and English). Generate such a corpus from a UD Treebank with corpus_udtreebank.sh -s:

litsea extract -l korean --format tsv ./ko_corpus.tsv ./ko_features.txt
litsea extract -l english --format tsv --tag-free ./en_corpus.tsv ./en_features.txt

Output Format

The features file contains one line per character position. For the corpus line これ は テスト です 。, the first two lines are:

-1	BC1:OI	BC2:II	BC3:II	BP1:UU	BP2:UU	BQ1:UOI	BQ2:UII	BQ3:UOI	BQ4:UII	BW1:B1こ	BW2:これ	...
1	BC1:II	BC2:II	BC3:IK	BP1:UU	BP2:UO	BQ1:UII	BQ2:UII	BQ3:OII	BQ4:OII	BW1:これ	BW2:れは	...

• 1 = word boundary
• -1 = non-boundary
• Features are written tab-separated in alphabetically sorted order, so each line starts with the BC1: feature rather than following the template definition order

Examples

# Japanese
litsea extract -l japanese ./corpus.txt ./features.txt

# Chinese
litsea extract -l zh ./corpus_zh.txt ./features_zh.txt

# Korean
litsea extract -l ko ./corpus_ko.txt ./features_ko.txt

# English
litsea extract -l en ./corpus_en.txt ./features_en.txt

Output to stderr on success:

Feature extraction completed successfully.

Two-Stage Feature Extraction

When the --pos flag is specified, extract expects a POS corpus instead of a plain word-separated corpus. Each line contains words annotated with UPOS tags in the format word/POS:

POS Corpus Format

これ/PRON は/ADP テスト/NOUN です/AUX 。/PUNCT
今日/NOUN は/ADP いい/ADJ 天気/NOUN です/AUX ね/PART 。/PUNCT

extract --pos writes three files derived from FEATURES_FILE as a prefix, for the two-stage segmentation + POS tagging architecture:

Pass the same prefix to litsea train --pos:

litsea extract --pos -l japanese ./pos_corpus.txt ./pos_features
# writes ./pos_features.stage1, .stage2, .lexicon

Space-Preserving POS Corpus (--pos --format tsv)

For a space-delimited language, combine --pos with --format tsv (issue #198). The corpus is then a tab-separated list of word/POS tokens in which a token may be a literal space carrying no /POS suffix — the format corpus_udtreebank.sh -p -s emits:

I/PRON	 	do/AUX	n't/PART	 	know/VERB	./PUNCT

bash scripts/corpus_udtreebank.sh -p -s "$conllu_file" ./pos_corpus.tsv
litsea extract --pos --format tsv -l english --stage2-features full ./pos_corpus.tsv ./pos_features

This is how the bundled korean_pos.model and english_pos.model are trained. Training on the unspaced corpus instead costs Korean ~5.9pt and English ~20.8pt of held-out Word F1, because stage 1 never sees the spaces that mark most word boundaries and stage 2’s context features see different neighbours than they will at inference.

Whitespace tokens get no stage-2 row — they are ~43% of tokens in a spaced corpus and would train one degenerate X class — but they do get a lexicon entry, which makes them single-candidate and therefore tagged deterministically through the model’s fixed-tag path, skipping the classifier entirely.

Japanese and Chinese should use plain --pos: their text has no spaces, so there is no spacing to preserve.