Introduction

Litsea is an extremely compact word segmentation library implemented in Rust, inspired by TinySegmenter and TinySegmenterMaker.

Unlike traditional morphological analyzers such as MeCab and Lindera, Litsea does not rely on large-scale dictionaries. Instead, it performs word segmentation using a compact pre-trained model based on the AdaBoost binary classification algorithm. Litsea also supports word segmentation and POS (Part-of-Speech) tagging with the Universal POS (UPOS) tagset via a two-stage architecture.

Key Features

• Fast and safe Rust implementation – built with Rust’s safety guarantees and performance
• Compact pre-trained models – the legacy RWCP.model / JEITA_Genpaku_ChaSen_IPAdic.model files are kilobyte-scale; the quality-optimized japanese/chinese/korean/english.model files are ~86 KB-2.0 MB, still small enough to embed directly in applications or serve over HTTP
• No dictionary dependency – segmentation is driven entirely by a statistical model
• Two-stage POS tagging – segments with a binary boundary classifier and tags each word via a candidate-tag lexicon plus a word-level tagger, adding little cost over plain segmentation
• Multilingual support – Japanese, Chinese (Simplified/Traditional), Korean, and English
• Model training capabilities – train custom models using AdaBoost or Averaged Perceptron with your own corpora
• Remote model loading – load models from HTTP/HTTPS URLs (opt-in remote_model feature) or local files
• Simple and extensible API – easy to integrate into Rust projects as a library

How It Works

Litsea treats word segmentation as a binary classification problem: for each character position in a sentence, the model predicts whether it is a word boundary (+1) or not a boundary (-1). The classifier uses character n-gram features and character type information specific to each language.

Input:  "これはテストです。"
         こ れ は テ ス ト で す 。
         B  O  B  B  O  O  B  O  B   ← word-start predictions (RWCP.model)
Output: ["これ", "は", "テスト", "です", "。"]

POS Tagging

Litsea also supports POS (Part-of-Speech) tagging in addition to word segmentation, through the two-stage architecture: the sentence is segmented by a binary boundary classifier, then each word is tagged through a candidate-tag lexicon plus a word-level tagger.

For each character position, the model predicts one of 18 SegmentLabel classes:

• B-NOUN, B-VERB, …, B-X (boundary labels for 17 POS tags)
• O (non-boundary = continuation of the current word)

The POS tags follow the Universal Dependencies UPOS tagset (17 POS tags).

Input:  "今日はいい天気ですね。"
Output: 今日/NOUN は/ADP いい/ADJ 天気/NOUN です/AUX ね/PART 。/PUNCT

Name Origin

There is a small plant called Litsea cubeba (Aomoji) in the same Lauraceae family as Lindera (Kuromoji). This is the origin of the name Litsea.

Current Version

Litsea v0.13.0 – Rust Edition 2024, minimum Rust version 1.87.

Links

• GitHub Repository
• crates.io
• API Documentation (docs.rs)
• Japanese Documentation (日本語)

Getting Started

Welcome to Litsea! This section will help you get up and running quickly.

Litsea is a compact word segmentation library in Rust that supports word segmentation (AdaBoost) and two-stage POS tagging (a binary boundary classifier plus a word-level tagger).

Next Steps

• Installation – install Litsea from source or crates.io
• Quick Start – segment your first sentence in minutes

Installation

Prerequisites

• Rust 1.87 or later (stable channel) from rust-lang.org
• Cargo (Rust’s package manager, included with Rust)

Installing the CLI Tool

From crates.io

cargo install litsea-cli

From Source

git clone https://github.com/mosuka/litsea.git
cd litsea
cargo build --release

The binary will be available at ./target/release/litsea.

Verify the installation:

./target/release/litsea --help

Using as a Library

Add Litsea to your project’s Cargo.toml:

[dependencies]
litsea = "0.13.0"

Remote model loading over http(s) is opt-in; enable the remote_model feature if you need it:

litsea = { version = "0.13.0", features = ["remote_model"] }

Note: Loading models from local files (load_model_from_path) is synchronous, so no async runtime is needed. An async runtime such as tokio is only required if you load models over HTTP/HTTPS with the async load_model method (available only when the opt-in remote_model feature is enabled).

Supported Platforms

Litsea is tested on the following platforms:

Quick Start

CLI Quick Start

Segmenting Text

Litsea ships with pre-trained models in the models/ directory. Pipe text into the segment command:

Japanese (using the bundled RWCP.model, the original TinySegmenter model):

echo "LitseaはTinySegmenterを参考に開発された、Rustで実装された極めてコンパクトな単語分割ソフトウェアです。" \
  | litsea segment -l japanese ./models/RWCP.model

Output:

Litsea は TinySegmenter を 参考 に 開発 さ れ た 、Rust で 実装 さ れ た 極めて コンパクト な 単語 分割 ソフトウェア です 。

Chinese:

echo "中文分词测试。" | litsea segment -l chinese ./models/chinese.model

Korean:

echo "한국어 단어 분할 테스트입니다." | litsea segment -l korean ./models/korean.model

English:

echo "I don't know." | litsea segment -l english ./models/english.model

POS Tagging

Litsea can perform word segmentation and POS tagging using a two-stage model. Add the --pos flag to the segment command:

echo "今日はいい天気ですね。" \
  | litsea segment --pos -l japanese ./models/japanese_pos.model

Output:

今日/NOUN は/ADP いい/ADJ 天気/NOUN です/AUX ね/PART 。/PUNCT

Each token is annotated with a Universal POS (UPOS) tag.

Library Quick Start

Here is a minimal Rust program that loads a model and segments text:

use std::path::Path;

use litsea::adaboost::AdaBoost;
use litsea::language::Language;
use litsea::segmenter::Segmenter;

fn main() -> litsea::Result<()> {
    // Load the pre-trained model
    let mut learner = AdaBoost::new(0.01, 100);
    learner.load_model_from_path(Path::new("./models/RWCP.model"))?;

    // Create a segmenter
    let segmenter = Segmenter::with_learner(Language::Japanese, learner);

    // Segment text
    let tokens = segmenter.segment("これはテストです。");
    println!("{}", tokens.join(" "));
    // Output: これ は テスト です 。

    Ok(())
}

POS Tagging with the Library

Here is a minimal Rust program that loads a POS model and segments text with POS tags:

use std::path::Path;

use litsea::language::Language;
use litsea::segmenter::Segmenter;
use litsea::two_stage::TwoStageLearner;

fn main() -> litsea::Result<()> {
    // Load the pre-trained two-stage POS model
    let mut learner = TwoStageLearner::new();
    learner.load_model_from_path(Path::new("./models/japanese_pos.model"))?;

    // Create a segmenter with POS support
    let segmenter = Segmenter::with_two_stage_learner(Language::Japanese, learner);

    // Segment text with POS tags
    let tokens = segmenter.segment_with_pos("今日はいい天気ですね。")?;
    for (word, pos) in &tokens {
        print!("{}/{} ", word, pos);
    }
    // Output: 今日/NOUN は/ADP いい/ADJ 天気/NOUN です/AUX ね/PART 。/PUNCT

    Ok(())
}

What’s Next

• CLI Reference – learn all CLI commands and options
• Training Guide – train your own models
• Architecture – understand how Litsea works internally

Architecture Overview

Litsea is designed as a compact, dictionary-free word segmentation system. It treats word segmentation as a binary classification problem and uses AdaBoost to learn word boundary patterns from character-level features.

High-Level Data Flow

Litsea has two main workflows: training and segmentation.

Training Pipeline

flowchart LR
    A["Corpus (text)"] --> B["Extractor"]
    B --> C["Features File (.txt)"]
    C --> D["Trainer (AdaBoost)"]
    D --> E["Model File (.model)"]

1. Corpus preparation – Prepare text with words separated by spaces
2. Feature extraction – The Extractor reads the corpus, classifies characters by type, and outputs labeled feature vectors
3. Model training – The Trainer feeds features into AdaBoost, which iteratively selects the most informative features and produces a compact model

Segmentation Pipeline

flowchart LR
    F["Raw text"] --> G["Segmenter (AdaBoost)"]
    H["Model file"] --> G
    G --> I["Segmented words"]

1. Model loading – Load a pre-trained model (from file or URL)
2. Character classification – For each character in the input, determine its type code based on language-specific patterns
3. Feature extraction – Stream character n-gram features for each position through a reused buffer using a sliding window
4. Prediction – AdaBoost predicts whether each position is a word boundary

Design Principles

• No dictionary dependency – Unlike MeCab or Lindera, Litsea relies solely on a statistical model learned from character patterns
• Compact models – Legacy word-segmentation models (RWCP.model, JEITA_Genpaku_ChaSen_IPAdic.model) are ~16-22 KB; the retrained japanese/chinese/korean/english.model are ~86 KB-2.0 MB, trading file size for the quality gains documented in Pre-trained Models; two-stage POS models are ~3.6-8 MB – all still small enough to embed directly in applications, containing only the feature weights that matter
• Language-agnostic framework – The core algorithm is the same for all languages; only the character type patterns differ
• Simple extensibility – Adding a new language requires only defining character type patterns and training a model

Module Design

The litsea library crate is organized into focused modules, each with a clear responsibility.

Module Dependency Graph

graph TD
    language["language.rs<br/>Character classification"]
    segmenter["segmenter.rs<br/>Segmentation + POS tagging"]
    adaboost["adaboost.rs<br/>AdaBoost (boundaries)"]
    perceptron["perceptron.rs<br/>Averaged Perceptron (POS)"]
    upos["upos.rs<br/>UPOS tags and labels"]
    extractor["extractor.rs<br/>Feature extraction"]
    trainer["trainer.rs<br/>Training orchestration"]
    two_stage["two_stage.rs<br/>Two-stage model container"]
    word_features["word_features.rs (private)<br/>Stage-2 word feature templates"]
    packed_model["packed_model.rs (private)<br/>Feature templates + packed AdaBoost tables"]
    packed_two_stage["packed_two_stage.rs (private)<br/>Packed two-stage tagging tables"]
    model_io["model_io.rs (private)<br/>Model URI loading"]
    error["error.rs<br/>LitseaError / Result"]
    metrics["metrics.rs<br/>Evaluation metrics (in-sample)"]
    evaluation["evaluation.rs<br/>Held-out quality metrics"]

    language --> segmenter
    upos --> segmenter
    adaboost --> segmenter
    perceptron --> segmenter
    packed_model --> segmenter
    packed_two_stage --> segmenter
    two_stage --> segmenter
    segmenter --> extractor
    two_stage --> extractor
    word_features --> extractor
    evaluation --> extractor
    adaboost --> trainer
    perceptron --> trainer
    two_stage --> trainer
    adaboost --> two_stage
    perceptron --> two_stage
    upos --> two_stage
    language --> word_features
    word_features --> packed_two_stage
    language --> packed_two_stage
    perceptron --> packed_two_stage
    upos --> packed_two_stage
    model_io --> adaboost
    model_io --> perceptron
    error --> adaboost
    error --> perceptron
    metrics --> trainer
    segmenter --> evaluation
    upos --> evaluation

Module Details

language.rs – Language Definitions

Defines the Language enum and character type classification.

• Language – Enum with variants Japanese, Chinese, Korean, English
  • Implements FromStr (parses "japanese", "ja", "chinese", "zh", "korean", "ko", "english", "en")
  • Implements Display (outputs lowercase name)
  • char_type(c: char) -> &'static str – Classifies a character as a table lookup over the numeric type id returned by the private char_type_id(), which dispatches to a per-language function (japanese_char_type_id, etc.) implemented as a direct match on character ranges (allocation-free; no regex). The language-specific functions share a punct_latin_digit() helper for the common "P"/"A"/"N" classes.

segmenter.rs – Word Segmentation and POS Tagging

The main user-facing module.

• Segmenter – Holds a Language and an AdaBoost learner (fields are private; use language(), learner(), learner_mut()), plus an internal cache for the compiled scoring tables (packed) that back segment(), and an optional compiled two-stage tagging model (set by with_two_stage_learner, backing segment_with_pos(); unlike the cache it is the stage-2 model itself – the raw learner parts are dropped after compilation)
  • new(language) – Create a segmenter with a default (empty) AdaBoost learner
  • with_learner(language, learner) – Create a segmenter with a pre-configured AdaBoost learner (e.g. one that has loaded a pre-trained model)
  • with_two_stage_learner(language, learner) – Create a segmenter for two-stage segmentation + POS tagging from a TwoStageLearner
  • segment(sentence) – Segment text into words, returns Vec<String>
  • segment_into(sentence, buf) – Allocation-free variant (#184): returns token byte ranges borrowed from a reusable SegmentBuffer
  • segment_with_pos(sentence) – Segment and tag, returns Result<Vec<(String, Upos)>> (PosLearnerNotSet unless a two-stage learner is set)
  • char_type(ch) – Classify a single character into its type code
  • add_corpus(corpus) / add_corpus_tsv(corpus) – Add training data (space-separated, or tab-separated/space-preserving; the latter is used for Korean and English, see issue #152)
  • add_corpus_with_writer(corpus, callback) / add_corpus_with_pos_writer(corpus, callback) / add_corpus_tsv_with_writer(corpus, callback) – Process a corpus with a custom callback (the POS-writer variant feeds two-stage stage-1 feature extraction)

adaboost.rs – AdaBoost Algorithm

The binary classifier used for word boundary decisions.

• AdaBoost
  • new(threshold, num_iterations) – Create with training parameters
  • initialize_features(path) / initialize_instances(path) – Load training data
  • train(running) – Run the AdaBoost training loop
  • predict(&attributes) – Predict boundary (+1) or non-boundary (-1)
  • load_model(uri) (async) / load_model_from_path(path) / load_model_from_reader(reader) – Load model weights
  • save_model(path) – Save model weights to a file
  • metrics() – Calculate accuracy, precision, and recall (BinaryMetrics)
  • bias() – Get the model’s bias term

perceptron.rs – Averaged Perceptron

The multiclass classifier behind two-stage training (both stages) and the bundled segmentation models’ collapse recipe.

• AveragedPerceptron
  • add_instance(features, label) – Add a training instance
  • train(num_epochs, running) – Train with weight averaging (running: &AtomicBool)
  • predict(&features) – Predict the best class label
  • load_model(uri) (async) / load_model_from_path(path) / load_model_from_reader(reader) – Load model weights
  • save_model(path) – Save model weights
  • metrics() – Macro-averaged evaluation (MulticlassMetrics)
• Weights are stored in a feature → per-class vector layout for fast inference.

upos.rs – Universal POS Tags

• Upos – The 17 Universal Dependencies POS tags (NOUN, VERB, …)
• SegmentLabel – Combined segmentation + POS label per character position (B(Upos) or O), with Display/FromStr for the "B-NOUN" / "O" string form

extractor.rs – Feature Extraction

Extracts features from a corpus for model training.

• Extractor – Wraps a Segmenter to process corpus files
  • new(language) – Create an extractor for a specific language
  • extract(corpus_path, features_path) – Read a corpus, write a features file
  • extract_tsv(corpus_path, features_path) – Same for tab-separated, space-preserving corpora (issue #152, used for Korean and English)
  • extract_two_stage(corpus_path, output_prefix, feature_set) – Extract two-stage training features (issue #147) from a POS-tagged corpus: writes {output_prefix}.stage1 (boundary features), .stage2 (word-level features), and .lexicon

trainer.rs – Training Orchestration

High-level training workflows.

• Trainer – Segmentation model training (AdaBoost)
  • new(threshold, num_iterations, features_path) – Initialize from a features file
  • load_model(uri) – Optionally load an existing model for incremental training (async)
  • train(running, model_path) – Train and save, returns BinaryMetrics
• PerceptronTrainer – Generic Averaged Perceptron training over opaque string labels (the training step of the bundled segmentation models’ collapse recipe)
  • new(num_epochs, features_path) / load_model(uri) / train(running, model_path) returning MulticlassMetrics
• TwoStageTrainer – Two-stage model training (issue #147): trains a stage-1 boundary AveragedPerceptron and a stage-2 word tagger from the files Extractor::extract_two_stage writes, then collapses stage 1 to AdaBoost format and assembles a TwoStageLearner
  • new(num_epochs, dominance, features_prefix) / train(running, model_path) returning TwoStageMetrics (see Trainer for the full API)
• TwoStageMetrics – One MulticlassMetrics per stage of a TwoStageTrainer::train run (stage1, stage2)

two_stage.rs – Two-Stage Model Container

Defines the litsea-two-stage v1 file format (see Model File Format) and the types that hold a two-stage model in memory (issue #147).

• TwoStageLearner – Bundles a stage-1 boundary AdaBoost model, a stage-2 AveragedPerceptron word tagger, and a candidate-tag lexicon; new() / from_parts(...) / load_model_from_path(path) / save_model(path) mirror the single-learner types’ API
• TwoStageFeatureSet – Enum selecting the stage-2 word-level template subset (Fast, Balanced, Full)
• ModelKind – Detect a model file’s format from its first line (AdaBoost, standalone AveragedPerceptron, or TwoStage); used to give wrong-kind files precise loader errors

error.rs – Error Handling

• LitseaError – Error enum (Io, InvalidData, InvalidInput, Unsupported, PosLearnerNotSet, and Download with the remote_model feature). Marked #[non_exhaustive], so downstream match expressions need a wildcard arm
• Result<T> – Alias used by every fallible API

metrics.rs – Evaluation Metrics

• BinaryMetrics – Accuracy, precision, recall, confusion matrix (AdaBoost)
• MulticlassMetrics – Accuracy and macro-averaged precision/recall (Averaged Perceptron)

evaluation.rs – Held-Out Evaluation Metrics

While metrics.rs reports in-sample quality (measured on the training data itself, as printed by train), this module computes held-out quality: it compares a Segmenter’s output against a gold corpus using character-offset spans, so predicted and gold tokens can be matched exactly regardless of tokenization differences elsewhere in the sentence.

• SegmentationMetrics – Word and boundary precision/recall/F1 for word segmentation, produced by evaluate_segmentation(segmenter, gold)
• PosMetrics – Wraps a SegmentationMetrics plus tagged-word precision/recall/F1, produced by evaluate_pos(segmenter, gold) (fallible: propagates segment_with_pos errors)
• parse_gold_line(line, tsv) / parse_gold_pos_line(line) – Parse a gold corpus line into tokens (plain or POS-tagged); also used by the two-stage extractor and trainer
• Backs the CLI’s litsea evaluate subcommand

packed_model.rs – Feature Templates and Packed AdaBoost Tables (private)

Internal module holding the declarative feature-template table (TEMPLATES, the single source of truth for all feature consumers), the load-time parser that converts model feature strings into packed integer keys, and PackedModel – the AdaBoost weights compiled into the merged/dense tables read by segment()’s two-pass scorer. Not part of the public API.

packed_two_stage.rs – Packed Two-Stage Tagging Tables (private)

Internal module that compiles a TwoStageLearner’s stage-2 tagger and lexicon into the dense/sparse scoring tables segment_with_pos() reads, mirroring what packed_model.rs does for the AdaBoost learner: a surface map covering the lexicon and dominance-skip tags, sparse per-class rows for the char-valued word-feature templates (from word_features.rs), and dense tables for the type-valued and word-length templates. Not part of the public API.

word_features.rs – Stage-2 Word Feature Templates (private)

Internal module defining the word-level feature templates used by the two-stage tagger’s stage 2 (surface, word length, first/last char and type, context chars/types/bigrams, …). It is the single source of truth for the template set: the training extractor (via extract_two_stage) writes feature strings with write_word_features, and packed_two_stage.rs compiles the same strings back into integer keys with parse_word_feature, pinned against each other by a round-trip test. Not part of the public API.

model_io.rs – Model Loading I/O (private)

Internal module that resolves a model URI (plain path, file://, or http(s):// with the remote_model feature) and returns the raw model bytes. Not part of the public API.

Public Exports

The library’s lib.rs exposes the public modules and re-exports the main types:

#![allow(unused)]
fn main() {
pub mod adaboost;
pub mod error;
pub mod evaluation;
pub mod extractor;
pub mod language;
pub mod metrics;
mod model_io;
mod packed_model;
mod packed_two_stage;
pub mod perceptron;
pub mod segmenter;
pub mod trainer;
pub mod two_stage;
pub mod upos;
mod word_features;

pub use adaboost::AdaBoost;
pub use error::{LitseaError, Result};
pub use evaluation::{PosMetrics, SegmentationMetrics};
pub use extractor::Extractor;
pub use language::{Language, ParseLanguageError};
pub use metrics::{BinaryMetrics, MulticlassMetrics};
pub use perceptron::AveragedPerceptron;
pub use segmenter::{SegmentBuffer, Segmenter};
pub use trainer::{PerceptronTrainer, Trainer, TwoStageMetrics, TwoStageTrainer};
pub use two_stage::{
    ModelKind, ParseTwoStageFeatureSetError, TwoStageFeatureSet, TwoStageLearner,
};
pub use upos::{ParseSegmentLabelError, ParseUposError, SegmentLabel, Upos};

pub fn version() -> &'static str { ... }
}

AdaBoost Binary Classification

Litsea uses the AdaBoost (Adaptive Boosting) algorithm for binary classification to determine word boundaries. This chapter explains the algorithm as implemented in Litsea.

Overview

AdaBoost combines many weak learners (simple classifiers) into a strong ensemble classifier. In Litsea:

• Positive label (+1) = word boundary
• Negative label (-1) = non-boundary (continuation of the current word)
• Weak learners = individual features (each feature is a binary “stump” – present or absent)

Training Algorithm

The training loop in AdaBoost::train() works as follows:

Initialization

1. Load features and instances from the training file
2. Initialize instance weights uniformly (later adjusted based on initial score)
3. All model weights start at zero

Iterative Boosting

For each iteration t (up to num_iterations):

Step 1: Calculate weighted errors

For each feature h, compute its weighted error over all instances:

error[h] -= D[i] * y[i]   (for each instance i that has feature h)

where D[i] is the instance weight and y[i] is the true label.

Step 2: Select the best weak learner

Find the feature with the lowest weighted error rate:

error_rate(h) = (error[h] + positive_weight_sum) / instance_weight_sum
h_best = argmax_h |0.5 - error_rate(h)|

The baseline competitor is the “all-negative” classifier (always predicts -1), whose error rate equals the fraction of positive instances. Any real feature must beat this baseline.

Step 3: Check convergence

If |0.5 - best_error_rate| < threshold, stop early – no feature can significantly improve the model.

Step 4: Compute the weak learner weight

alpha = 0.5 * ln((1 - error_rate) / error_rate)
model[h_best] += alpha

A lower error rate produces a higher alpha, giving more influence to better features.

Step 5: Update instance weights

For each instance i:
    prediction = +1 if h_best in features(i), else -1

    if y[i] * prediction < 0:  (misclassified)
        D[i] *= exp(alpha)     (increase weight)
    else:                       (correctly classified)
        D[i] /= exp(alpha)     (decrease weight)

Normalize: D[i] /= sum(D)

This ensures subsequent iterations focus on the instances that are still difficult to classify.

Prediction

Given an input set of features (attributes), the prediction is:

score = bias + sum(model[feature] for each feature in attributes)
prediction = +1 if score >= 0, else -1

Bias Term

The bias is computed as:

bias = -sum(all model weights) / 2.0

The value is cached in the learner and kept in sync by every weight-mutating path, so reading it during inference is O(1).

This centers the decision boundary. The empty-string feature ("") serves as the bias bucket during training. It is registered at feature index 0 on every construction path (new(), feature-file initialization, and model loading), so models trained through add_instance() train, save, and reload correctly.

Compiled Scoring in the Segmenter

predict() itself looks features up by string, and its semantics are unchanged. The Segmenter, however, does not call it on the hot path: after a model (re)load it compiles the learner’s (feature, weight) pairs into integer-indexed tables and scores each sentence in two passes (see Prediction Pipeline). The compiled tables mirror the model exactly; only the floating-point accumulation order differs, and the differential test suite pins that the segmentation output stays identical in practice.

Model File Format

The trained model is saved as a simple text file:

feature1\tweight1
feature2\tweight2
...
bias_value

• Each line contains a feature name and its weight (tab-separated)
• Zero-weight features are omitted
• The last line contains the bias term (a single number)

Malformed files (empty, truncated before the bias line, duplicate bias lines or features, non-finite weights) are rejected at load time. See Model File Format for details.

This page describes the AdaBoost training algorithm. The file format above is also how litsea stores the four bundled segmentation models (japanese.model, chinese.model, korean.model, english.model) – but as of issue #165, those are no longer produced by boosting. They are trained as a 2-class Averaged Perceptron and losslessly collapsed into this same scalar-weight format (see Pre-trained Models for the collapse procedure). Any file AdaBoost::load_model can read is a valid instance of this format, but that does not mean it was actually produced by the boosting loop described above.

Hyperparameters

Averaged Perceptron

Litsea uses the Averaged Perceptron algorithm for multiclass classification as the training-side learner behind both stages of the two-stage POS architecture and the bundled segmentation models’ collapse recipe. This chapter explains the algorithm as implemented in Litsea.

Overview

While AdaBoost performs binary classification (boundary vs. non-boundary), the Averaged Perceptron performs multiclass classification – predicting one of 18 segment labels for each character position:

• 17 boundary labels: B-ADJ, B-ADP, B-ADV, B-AUX, B-CCONJ, B-DET, B-INTJ, B-NOUN, B-NUM, B-PART, B-PRON, B-PROPN, B-PUNCT, B-SCONJ, B-SYM, B-VERB, B-X
• 1 non-boundary label: O (continuation of the current word)

These labels correspond to the 17 Universal POS (UPOS) tags from the Universal Dependencies project, prefixed with B- to indicate a word boundary. This enables simultaneous word boundary detection and POS estimation in a single classification step.

The Averaged Perceptron in 2-class mode (B/O labels only) is what actually trains the bundled japanese.model, chinese.model, korean.model, and english.model segmentation models before they are losslessly collapsed to the AdaBoost model format for inference – see Pre-trained Models: Training Procedure for the full derivation. The same collapse trains the two-stage architecture’s stage-1 boundary classifier, and the multiclass form is the algorithm behind its stage-2 word-level tagger.

Algorithm

Weight Representation

The perceptron maintains a per-class weight vector for each feature, stored in a single sparse map so that scoring and weight updates need one hashed lookup per feature (not one per feature x class):

slots: FxHashMap<Feature, FeatureSlot>
// FeatureSlot { w: Vec<f64>, acc: Vec<f64>, ts: Vec<usize> } -- one entry per class

For example:

weights["UW4:猫"]["B-NOUN"] = 2.5
weights["UC4:H"]["B-NOUN"]  = 1.8
weights["UW4:猫"]["O"]      = -0.3
...

For a given feature set, the score for each class is the sum of its feature weights:

score(class) = sum(weights[feature][class] for each feature in input)
prediction = argmax(score(class) for all classes)

Update Rule

When the perceptron makes a misclassification:

For each training instance (features, truth):
    guess = predict(features)

    if guess != truth:
        For each feature f in features:
            weights[f][truth] += 1.0   # increase weight for correct class
            weights[f][guess] -= 1.0   # decrease weight for predicted class

This increases the weights for the correct class and decreases them for the incorrectly predicted class, making the correct prediction more likely for similar inputs in the future.

Averaging

A key improvement over the basic perceptron is weight averaging. Rather than using the final weights (which can be unstable and tend to overfit to the tail of the training data), the model averages all weight vectors seen during training. This improves generalization to unseen data.

The implementation uses a cumulative sum approach for efficiency:

cumulative[feature][class] += weights[feature][class] * elapsed_steps

At the end of training:
    averaged[feature][class] = cumulative[feature][class] / total_steps

This avoids storing all intermediate weight vectors while producing the same result. The averaging reduces dependence on the order of training data and improves generalization performance.

The accumulator and timestamp vectors are materialized lazily, the first time training touches a feature – a model loaded for inference only carries live weights and pays nothing for averaging state.

Training with Epochs

Training iterates over the data multiple times (epochs). Each epoch processes all training instances in order:

For each epoch (1 to num_epochs):
    For each instance in training data:
        features = extract_features(instance)
        predicted = argmax(score(class) for all classes)
        if predicted != correct_label:
            update weights
        accumulate weights for averaging

Training supports graceful interruption via AtomicBool – a Ctrl+C signal stops training and saves the model at its current state.

#![allow(unused)]
fn main() {
use std::sync::atomic::AtomicBool;
use litsea::perceptron::AveragedPerceptron;

let mut perceptron = AveragedPerceptron::new();
// ... add instances ...
let running = AtomicBool::new(true);
perceptron.train(10, &running);  // 10 epochs
}

Model File Format

The Averaged Perceptron model is saved as a text file with the following structure:

18
O
B-ADJ
B-ADP
...
B-X
feature1\tclass1\tweight1
feature2\tclass2\tweight2
...

• Line 1: Number of classes (18)
• Lines 2 to N+1: Class names, one per line
• Remaining lines: Feature weights, tab-separated as feature\tclass\tweight
• Zero-weight entries are omitted
• Weight lines are written in sorted feature order, so saving the same model always produces byte-identical files; loading does not depend on the order

Comparison with AdaBoost

Hyperparameters

Feature Extraction

Litsea uses character n-gram features to capture the local context around each potential word boundary. This chapter catalogs all feature types: the character-level boundary templates shared by the AdaBoost and two-stage stage-1 pipelines, and the word-level templates used by two-stage’s stage-2 tagger.

Feature Categories

For each character position i in the input, the segmenter extracts features from a sliding window of characters, their type codes, and previous boundary decisions.

Base Features (38 features)

Language-Specific Features (4 features, Japanese and Chinese only)

• WC1: character at i-1 + type code at i
• WC2: type code at i-1 + character at i
• WC3: character at i-1 + type code at i-1
• WC4: character at i + type code at i

Why no WC for Korean and English? Korean Hangul syllables are classified into only two types (SN and SF), so WC features would add noise rather than useful signal. English measured the same outcome directly: a dev-split comparison scored 98.68% Word F1 with the 38 base templates versus 98.65% with all 42 (WC included) – see English.

Total Feature Count

Single Source of Truth

The whole template above is defined once as a declarative table (packed_model::TEMPLATES – prefix plus an ordered list of tag/char/type slots, in a fixed emission order). Both feature representations derive from it:

• the string form below, used for training data extraction, corpus processing, and model files;
• the integer-indexed scoring tables used by the two-pass scorers of segment() and segment_with_pos(), into which model files are compiled at load time (see Prediction Pipeline).

Adding or reordering a template therefore changes every consumer consistently. The table order defines the string writer’s emission sequence, which model files and training data depend on.

Feature Format

Each feature is represented as a string in the format PREFIX:VALUE:

UW4:は        ← The character at position i is "は"
UC4:I         ← The type code at position i is "I" (Hiragana)
BW2:はテ      ← The bigram at position i-1..i is "はテ"
BC2:IK        ← The type bigram is Hiragana + Katakana
UP3:B         ← The previous boundary decision was "B" (boundary)
WC1:はK       ← Character "は" combined with type "K"

Sliding Window Layout

The segmenter pads the input with sentinel characters:

Index:   0    1    2    3    4    5    ...  n+2  n+3  n+4  n+5
Chars:   B3   B2   B1   c1   c2   c3  ...  cn   E1   E2   E3
Types:   O    O    O    t1   t2   t3  ...  tn   O    O    O
Tags:    U    U    U    U    ?    ?   ...  ?

• B3, B2, B1 – Begin sentinels (padding)
• E1, E2, E3 – End sentinels (padding)
• O – “Other” type for padding positions
• U – “Unknown” tag for initial positions
• B – “Boundary” tag (word start)
• O – “Other” tag (continuation)

Features are extracted for positions 4 through len-4 (inclusive) for the boundary (AdaBoost) pipeline; the POS pipeline also emits position 3, the first real character, because segment_with_pos predicts there to derive the first word’s POS (#100). Positions run 4 (or 3) through len-4 (inclusive), where the full window of i-3 to i+2 is available.

Training Data Format

The extract command writes features to a file in this format (real output for the corpus line これ は テスト です 。):

-1	BC1:OI	BC2:II	BC3:II	BP1:UU	BP2:UU	BQ1:UOI	BQ2:UII	BQ3:UOI	BQ4:UII	BW1:B1こ	...
1	BC1:II	BC2:II	BC3:IK	BP1:UU	BP2:UO	BQ1:UII	BQ2:UII	BQ3:OII	BQ4:OII	BW1:これ	...

Each line contains:

1. A label (1 for boundary, -1 for non-boundary)
2. Tab-separated feature strings, written in alphabetically sorted order (not template emission order), so each line starts with the BC1: feature

Word-Level Feature Templates (Two-Stage)

Everything above is the character-level template set scored at each boundary decision. Two-stage POS tagging’s stage-2 word tagger scores a separate, unrelated template set defined in litsea::word_features (N_WORD_TEMPLATES = 23) – one row of features per already-segmented word, not per character position. It is a different declarative table from packed_model::TEMPLATES above, compiled into its own runtime, packed_two_stage::PackedTwoStageModel (see Prediction Pipeline).

For a word spanning [start, end) of a sentence (w = surface, n = end - start):

That is 23 templates in total. Context positions beyond the sentence use begin/end sentinel characters, analogous to the character-level pipeline’s B1-B3 / E1-E3 padding above.

Not every template is written on every extraction: which subset lands in the .stage2 feature file is controlled by TwoStageFeatureSet (full / balanced / fast) at extraction time – see Extracting Features for the CLI flag and what each variant includes.

Character Type Classification

Each language in Litsea defines a set of character type patterns that classify individual characters into linguistically meaningful categories. These type codes are used as features for the AdaBoost classifier.

How It Works

Language::char_type(c: char) -> &'static str classifies a character with a direct match expression on Unicode character ranges — no regex, no allocation. Match arms are tried top to bottom, so the first matching arm determines the type code. If no arm matches, the character is classified as "O" (Other).

Each language has its own classification function returning a numeric type id (japanese_char_type_id, chinese_char_type_id, korean_char_type_id); char_type is a table lookup over the returned id, so string codes and numeric ids stay consistent by construction. The classes shared by all languages — "P" (punctuation), "A" (Latin), "N" (digits) — live in a common punct_latin_digit() helper that is checked after the language-specific classes. Logic beyond plain ranges is expressed with extra code inside an arm body (e.g., Korean’s codepoint test for Hangul syllable structure).

Japanese Character Types

Note: “M” (Kanji numbers) is checked before “H” (general Kanji), so characters like 一 and 百 are classified as numbers rather than generic ideographs.

Chinese Character Types

Chinese function words include:

• Structural particles: 的, 地, 得
• Aspect/modal particles: 了, 着, 过, 吗, 呢, 吧, 啊, 嘛
• Conjunctions: 和, 与, 或, 但, 而, 且, 及
• Prepositions: 在, 从, 到, 把, 被, 对, 向, 给
• Common grammatical verbs/adverbs: 是, 有, 不, 也, 都, 就, 要, 会, 能, 可

Korean Character Types

Korean Hangul Syllable Detection

Korean uses a range arm with a codepoint test for the SN and SF types. This leverages Unicode’s systematic Hangul encoding:

• Hangul Syllables occupy U+AC00–U+D7AF
• Each syllable is encoded as: (initial * 21 + medial) * 28 + final + 0xAC00
• If (codepoint - 0xAC00) % 28 == 0, the syllable has no final consonant (SN)
• Otherwise, it has a final consonant (SF, “받침”)

This distinction is important because the presence of a final consonant (받침) affects Korean word boundary patterns and particle attachment.

English Character Types

Unlike the other three languages, English classifies ASCII punctuation (minus the apostrophe) as "P" rather than leaving it as "O" — a deliberate, English-specific difference (char_type('@') is "O" elsewhere but "P" for English). The apostrophe is split out into its own type ("Q") because it is the character-level marker of contractions and possessives (do + n't, Google + 's), and uppercase gets its own type ("U") because it correlates with sentence-initial and proper-noun boundaries. See English for the full rationale, including why the hyphen stays "P" rather than getting an eighth type.

Cross-Language Comparison

Prediction Pipeline

This chapter provides a step-by-step walkthrough of how Segmenter::segment() processes input text.

Example: Segmenting “これはテストです。”

Step 1: Initialize Arrays with Padding

chars: ["B3", "B2", "B1"]
types: ["O",  "O",  "O" ]
tags:  ["U",  "U",  "U", "U"]

The tags array gets one extra “U” because tags[3] represents the first real character’s tag (set to “Unknown” since there is no prior boundary decision).

Step 2: Scan Input Characters

For each character in the input, determine its type using language-specific patterns and append to the arrays:

chars: ["B3","B2","B1", "こ","れ","は","テ","ス","ト","で","す","。"]
types: ["O", "O", "O",  "I", "I", "I", "K", "K", "K", "I", "I", "P"]

Step 3: Append End Sentinels

chars: [..., "。", "E1", "E2", "E3"]
types: [..., "P",  "O",  "O",  "O" ]

Step 4: Iterate and Predict

For each position i from 4 through len(chars) - 4 (inclusive):

i=4 (れ): Extract features → predict → label=-1 (O) → word="これ"
i=5 (は): Extract features → predict → label=+1 (B) → push "これ", word="は"
i=6 (テ): Extract features → predict → label=+1 (B) → push "は", word="テ"
i=7 (ス): Extract features → predict → label=-1 (O) → word="テス"
i=8 (ト): Extract features → predict → label=-1 (O) → word="テスト"
i=9 (で): Extract features → predict → label=+1 (B) → push "テスト", word="で"
i=10(す): Extract features → predict → label=-1 (O) → word="です"
i=11(。): Extract features → predict → label=+1 (B) → push "です", word="。"

Step 5: Push Final Word

Push the remaining word “。” to the result.

Result

["これ", "は", "テスト", "です", "。"]

How Prediction Works: Two Passes

segment() never builds feature strings. When a model is loaded, the segmenter compiles the learner’s string-keyed weights into integer-indexed tables once (see below), and each sentence is then scored in two passes, exploiting the fact that only 16 of the 38–42 features depend on earlier boundary decisions:

1. Static pass – every tag-free feature is accumulated into a per-position score buffer in one sweep over the sentence:

   • Each character position makes one merged UW probe (char code -> [UW1..UW6] weights) and one direct UC vector load (type id -> [UC1..UC6]), scatter-adding the six values to the six neighboring decision positions they feed.
   • Each adjacent pair makes one merged BW probe and one direct BC vector load (three values each), and each triple one direct TC vector load (four values).
   • For Japanese/Chinese, each character makes one merged WC row probe (the row is direct-indexed by type id and scatter-added to the two decision positions the character feeds); the block is skipped entirely for models without WC features.

2. Sequential pass – at each position i, the score starts from the bias plus the precomputed static score, adds the 16 tag-dependent weights (UP*, BP*, UQ*, BQ*, TQ* – all direct-indexed dense array loads, no hashing), and decides: if score >= 0, the character starts a new word; otherwise it continues the current one. The decision is pushed to the tags array and feeds the next positions’ lookups.

   score = bias + static[i] + sum(dense[tag-dependent template][mixed-radix index])
   

   Pointwise fast path (issue #183): if the model has no tag-dependent features at all (e.g. it was trained with litsea extract --tag-free, like the bundled korean.model/english.model), the compiled model records this at load time and the sequential pass is skipped entirely – the decision reduces to bias + static[i] >= 0 with no tag bookkeeping. Output is identical either way (the skipped loads would each add 0.0); only the serial dependency between positions disappears.

The bias is a cached field (-sum(model) / 2.0, kept in sync by every weight-mutating path) and is read once per sentence. The packed context carries parallel u32 char-code and u8 type-id arrays plus per-character byte offsets (the boundary pipeline emits tokens as byte ranges of the input, #184; segment() materializes them as Strings, segment_into() returns them directly from a reusable buffer); the sentinel entries (B3…E3) map to code points just above the Unicode scalar range.

The Compiled Scoring Tables

The feature template is defined once as a declarative table (packed_model::TEMPLATES), from which three consumers derive: the string writer used by training and extraction, the load-time parser that converts each model feature string into an integer key, and the two-pass scorer’s tables. Compilation happens eagerly in Segmenter::with_learner and is invalidated whenever the learner is mutated (learner_mut() / add_corpus), then rebuilt lazily on the next segment() call.

The compiled model splits by key-space size and tag dependence:

• Merged-vector hash tables for char n-grams: UW1..6 collapse into one char -> [f64; 6] table and BW1..3 into one (char, char) -> [f64; 3] table, so a whole family costs one probe. WC1..4 likewise collapse into one char -> [slot][type_id] row table (one probe per character, type dimension direct-indexed).
• Dense arrays for tag/type-only templates: each of the 29 gets a direct-indexed table sized by the exact mixed-radix product (3 per tag slot, 7–10 per type slot; about 74 KB total for Japanese). The UC/BC/TC tables additionally get merged scatter-vector views for the static pass.

Model features that the segmenter’s language could never generate (for example Korean type codes in a Japanese segmenter) are omitted from all tables; they are unreachable at scoring time, so scores are unaffected.

This table and its three consumers cover the character-level (stage-1 / AdaBoost) feature set only. Two-stage POS tagging’s stage-2 word tagger is compiled from a separate, parallel declarative table (litsea::word_features, 23 word-level templates) into its own runtime, packed_two_stage::PackedTwoStageModel – see Feature Extraction for its template catalog and the section below for how it fits into segment_with_pos.

Output Equivalence

Scoring accumulates in two-pass order, which differs from the historical string-keyed accumulation order, so the f64 sums are no longer guaranteed bit-for-bit identical. In practice no output difference has been observed: the exact-equality differential tests (all bundled models, sentinel stress strings, and a real-text corpus) pass unchanged, and they remain in the test suite as the detection net for any knife-edge score flip a future model might expose.

Segmentation and POS Tagging (segment_with_pos)

segment_with_pos requires a segmenter built with with_two_stage_learner (issue #147). It segments with the ordinary segment() boundary path described above, then tags each resulting word through packed_two_stage::PackedTwoStageModel::tag_words – a candidate-tag lexicon lookup, with a masked-argmax fallback for ambiguous known words and a full-argmax fallback for unknown words. The POS layer therefore adds no per-character scoring at all; see Two-Stage Tagging for the full explanation of that pipeline and its cost model.

Training vs. Prediction

During training, tags are derived from the ground-truth corpus segmentation, so the model learns from correct boundary decisions. During prediction, tags are generated on-the-fly, meaning each decision depends on all previous predictions – this is a left-to-right greedy approach.

Performance Characteristics

The segmentation algorithm is linear in the length of the input:

• Each character position is visited once: O(n)
• Feature extraction at each position: O(1) (fixed number of templates, each packed into a u64 on the stack)
• Prediction at each position: O(f) where f is the number of active features (~38-42), but with families merged – the static pass costs ~2 hash probes (UW, BW) plus a handful of direct vector loads per character, and the sequential pass is 16 direct dense-array loads
• Total: O(n * f) which is effectively O(n)
• Allocation profile: the packed context borrows word slices from the input and carries flat u32/u8 arrays, the bias is cached, and no strings are built anywhere in the hot loop (the packed table itself is compiled once per model load, off the hot path)

Two-Stage Tagging

Litsea’s segment_with_pos() is backed by the two-stage model (--pos, Segmenter::with_two_stage_learner, issue #147). The model file’s litsea-two-stage v1 header identifies the format (see Model File Format), and the method returns Vec<(String, Upos)> word/tag pairs.

How it works

Two-stage tagging never scores POS classes at the character level:

1. Stage 1 segments with a binary boundary classifier – structurally the same scalar-weight AdaBoost format segment() uses, so it runs at the same speed.
2. Stage 2 tags each word stage 1 produces, using a candidate-tag lexicon built from the training corpus plus a word-level classifier (see Model File Format):
   • A surface with a single observed tag, or one that covers at least the dominance fraction of its training occurrences, is tagged with no classifier call at all.
   • An ambiguous known surface is scored only over its observed candidate tags (a masked argmax), not all ~18 classes.
   • An unknown surface falls back to the full-class argmax.

Since a word averages more characters than one, and most words are either unambiguous or skip the classifier entirely, stage 2’s total cost is a small fraction of per-character multiclass scoring. This is what makes the POS path nearly as fast as plain segmentation: the historical joint architecture (removed in favor of two-stage) scored every one of the ~18 UPOS classes at every character position and ran 1.8-2.8x slower on the same corpora.

A methodology note: use enough training epochs

An epoch sweep (10 to 150 epochs) during the two-stage rollout showed that model quality keeps improving well past the 10-epoch convention older models were trained with, and plateaus at around 50 epochs on the UD GSD training sets. In particular, stage 1’s boundary-only features need more epochs than richer per-character feature sets to converge on the same corpus. The bundled two-stage models use 50 epochs; when retraining, a one-shot low-epoch run will understate the quality the architecture can reach.

Measured quality and throughput

Held-out figures are from litsea evaluate --pos on the UD GSD/EWT test splits (see Pre-trained Models). Throughput is cargo bench -- external_corpus on this project’s development machine (not dedicated, idle hardware); run-to-run spread is noticeable – see Benchmarking for the methodology and its limits.

Two observations worth keeping in mind:

• Candidate-tag restriction is a real quality lever. Restricting a known word’s candidates to what was actually observed removes most opportunities to pick an implausible tag, which offsets scoring a smaller feature set per word.
• Throughput varies by lexicon coverage. Korean’s high held-out unknown-word rate means a larger share of its words pay stage 2’s full-class fallback rather than the cheap dominance-skip or candidate-masked paths. English sits at the other extreme since #198: ~43% of its tokens are spaces, every one of which is a single-candidate lexicon hit that skips the classifier entirely.

Corpus protocol matters more than anything else here (issue #198)

The stage-1 classifier is only as good as the text it trained on. Until issue #198 the two-stage extractor read a space-separated word/POS corpus and reconstructed each sentence by concatenating word forms with no separator — correct for Japanese and Chinese, whose real text has no spaces, but for Korean and English it threw away the single strongest boundary signal their input carries.

Training on the space-preserving corpus instead (extract --pos --format tsv) moved both languages onto their dedicated segmentation models’ level:

(The “before” column is the real-world/spaced measurement from issue #196, not the unspaced-protocol figure, so it is an apples-to-apples comparison of what a user actually got.) Korean now sits 0.03pt from korean.model’s 99.91% and English 0.01pt from english.model’s 98.31%.

The unspaced corpus caused two distinct train/inference mismatches, and both are now fixed:

1. Stage 1 never saw the space characters that mark most word boundaries in these languages.
2. Stage 2 was affected too, less obviously: its context features (L1-L3 / R1-R3 and cl1-cl3 / cr1-cr3) read the surrounding characters, and at inference a word’s neighbour is usually a space — but in unspaced training it was the next word’s character instead. Since spaces are ~43% of tokens, essentially every word was affected.

Whitespace tokens get no stage-2 training row (they would be ~43% of rows for one degenerate X class) but do get a lexicon entry, which makes them single-candidate and therefore tagged X through the fixed-tag path without invoking the classifier — deterministic, and cheaper than the full-argmax guess the previous models fell back to.

Japanese and Chinese are unaffected by all of this: their text has no spaces, so the space-separated corpus already matched their real input.

Choosing a stage-2 feature set

Stage 2’s word-level tagger can be extracted with three feature sets (--stage2-features on litsea extract --pos; see Extracting Features), trading tagging quality for throughput. Segmentation quality is unaffected – it is decided entirely by stage 1. The figures below are at 50 epochs, matching the bundled models.

(Korean and English were re-swept on the space-preserving corpus for issue #198; those two columns are dev-split figures from that sweep, while the Chinese column predates it. Korean’s winner moved from balanced to full as a result.)

For Japanese, fast alone reaches 92.95% tagged F1, so the bundled japanese_pos.model uses it. For Chinese, balanced gives most of full’s gain (82.29% vs. 82.96%) at meaningfully better throughput, so chinese_pos.model uses balanced rather than full. For Korean and English on the space-preserving corpus, full is the clear winner (Korean 93.33% vs. 92.95% for balanced; English 90.43% vs. ~88.7% for either alternative), so both korean_pos.model and english_pos.model use full. Retraining with a different set is a matter of re-running extract --pos --stage2-features <set> + train --pos; there is no need to change any other part of the pipeline.

Language Support Overview

Litsea supports word segmentation for four languages through a unified framework based on the Language enum.

Supported Languages

The Language Enum

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
#[non_exhaustive]
pub enum Language {
    #[default]
    Japanese,
    Chinese,
    Korean,
    English,
}
}

• Default is Japanese
• Marked #[non_exhaustive] – new languages can be added without a breaking change, so external match expressions need a wildcard arm
• Implements FromStr – parses from full name or ISO 639-1 code (case-insensitive)
• Implements Display – outputs the lowercase full name

Parsing Examples

#![allow(unused)]
fn main() {
use litsea::language::Language;

let ja: Language = "japanese".parse().unwrap();
let zh: Language = "zh".parse().unwrap();
let ko: Language = "Korean".parse().unwrap();   // case-insensitive
let err = "french".parse::<Language>();          // Err(...)
}

How Languages Differ

Each language defines its own character type patterns that classify characters into type codes. These type codes are used as features for the AdaBoost classifier.

Why Korean and English Have Fewer Features

Korean Hangul syllables are classified into only two types: SN (without 받침/final consonant) and SF (with 받침). This binary distinction means WC features (word + character-type combinations) would produce redundant information with little discriminative power. Excluding them reduces noise and keeps the model compact.

English shares the same conclusion for a related reason: the dominant boundary signal is whitespace, and a dev-split comparison measured the 38-template (no WC) tag-free model at 98.68% Word F1 versus 98.65% with all 42 templates — WC features measured worse, not just unhelpful. See English for the full comparison.

Japanese

Japanese is the default language in Litsea.

Character Types

Pattern Priority

Patterns are evaluated in order. Notably:

• M before H: Characters like 一 and 百 are classified as “Kanji Numbers” (M), not generic “Kanji” (H)
• This distinction helps the model learn number-specific boundary patterns

Pre-trained Models

japanese.model

• Training corpus: UD Japanese-GSD
• Training options: 50 epochs of Averaged Perceptron training, collapsed to AdaBoost scalar weights, then pruned to the top 40,000 features by |weight| – see Training Procedure for the full recipe
• Word F1 (held-out): 96.70%
• Boundary F1 (held-out): 98.59%

japanese_pos.model

• Algorithm: two-stage segmentation + POS tagging (a binary boundary classifier plus a word-level tagger with a candidate-tag lexicon)
• Details: see Pre-trained Models

RWCP.model

• Source: Extracted from the original TinySegmenter
• License: BSD 3-Clause (Taku Kudo)
• Size: ~22 KB

JEITA_Genpaku_ChaSen_IPAdic.model

• Training corpus: JEITA Project Sugita Genpaku corpus
• Tokenizer: ChaSen with IPAdic dictionary
• Size: ~16 KB

Example

echo "LitseaはTinySegmenterを参考に開発された、Rustで実装された極めてコンパクトな単語分割ソフトウェアです。" \
  | litsea segment -l japanese ./models/RWCP.model

Output:

Litsea は TinySegmenter を 参考 に 開発 さ れ た 、Rust で 実装 さ れ た 極めて コンパクト な 単語 分割 ソフトウェア です 。

Chinese

Litsea supports Chinese word segmentation covering both Simplified and Traditional Chinese.

Character Types

Chinese Function Words (虚词)

The “F” type captures high-frequency grammatical words that are critical for segmentation:

These characters appear overwhelmingly in grammatical roles and signal word boundaries differently from content words.

Pre-trained Models

chinese.model

• Training corpus: UD Chinese-GSD
• Training options: 100 epochs of Averaged Perceptron training, collapsed to AdaBoost scalar weights, then pruned to the top 70,000 features by |weight| – see Training Procedure for the full recipe
• Word F1 (held-out): 90.69%
• Boundary F1 (held-out): 95.64%

chinese_pos.model

• Algorithm: two-stage segmentation + POS tagging (a binary boundary classifier plus a word-level tagger with a candidate-tag lexicon)
• Details: see Pre-trained Models

Example

echo "中文分词测试。" | litsea segment -l chinese ./models/chinese.model

Korean

Litsea supports Korean word segmentation with specialized Hangul character type detection.

Character Types

Korean Particles (조사)

The “E” type captures six high-frequency grammatical particles:

These particles frequently appear at word boundaries and are given a distinct type code to improve segmentation accuracy.

Hangul Syllable Structure (받침 Detection)

Korean uses a range arm with a codepoint test in its body for the SN and SF types. This exploits the systematic Unicode Hangul encoding:

• Hangul Syllables: U+AC00–U+D7AF (11,172 syllables)
• Each syllable = (initial * 21 + medial) * 28 + final + 0xAC00
• SN (no 받침): (codepoint - 0xAC00) % 28 == 0
• SF (with 받침): (codepoint - 0xAC00) % 28 != 0

The 받침 (final consonant) distinction is linguistically significant because it affects how particles attach to words and where boundaries occur.

No WC Features

Korean does not use WC (word + character-type) features. Since most Hangul syllables fall into only two types (SN and SF), WC features would produce low-entropy, noisy combinations that hurt model accuracy.

Space-Preserving Training

Korean is written with spaces between eojeol (word phrases), and those spaces mark most word boundaries. The Korean model is therefore trained on a space-preserving TSV corpus: tokens are tab-separated and each inter-eojeol space is kept as its own token, so the training text contains the space characters of the original sentence and the model can use them as boundary context. Generate the corpus with corpus_udtreebank.sh -s (which reconstructs spacing from the treebank’s SpaceAfter annotations) and extract features with litsea extract --format tsv. At inference no special handling is needed: segment() receives the spaced text as-is and emits each space as its own token.

Because each space is its own single-character token, the character-level labeling (see AdaBoost) marks two separate boundaries around it: the space itself starts a new token (label B), and the character immediately following the space starts the next token (also label B). Both are deterministic given the corpus construction – a space is always exactly one token, and whatever follows it always begins the next token – so the model learns them as a near-trivial rule. Only the second of these (the boundary that starts the following real word) affects the held-out Word F1 score, since pure-whitespace tokens are excluded from scoring (see Evaluation).

Pre-trained Models

korean.model

• Training corpus: UD Korean-GSD (space-preserving TSV corpus)
• Training options: --format tsv --tag-free, 30 epochs of Averaged Perceptron training, collapsed to AdaBoost scalar weights, not pruned (3,132 features) – see Training Procedure for the full recipe
• Word F1 (held-out): 99.91%
• Boundary F1 (held-out): 99.96%

The model is trained without the 16 tag-dependent feature templates (--tag-free, issue #183): with the space signal available they measured as contributing nothing, and a pointwise model lets segment() skip its sequential scoring pass entirely. See Tag-Free (Pointwise) Models.

Held-out metrics are computed on the original spaced text with space tokens excluded from scoring.

korean_pos.model

• Algorithm: two-stage segmentation + POS tagging (a binary boundary classifier plus a word-level tagger with a candidate-tag lexicon)
• Word F1 (held-out): 99.88%
• Tagged Word F1 (held-out): 93.95%
• Note: this model is trained on the same space-preserving corpus as korean.model (issue #198), so its Word F1 sits 0.03pt from korean.model’s 99.91%. Until #198 the two-stage pipeline trained on the unspaced word/POS corpus and scored 94.01% on real spaced input; switching protocols gained +5.9pt Word F1 and +10.8pt tagged-word F1. Spaces are re-emitted as their own tokens tagged X, deterministically, via a single-candidate lexicon entry rather than a classifier guess. See English, where the same change was worth over 20pt because English orthography carries far less boundary signal without spaces
• Details: see Pre-trained Models

Example

echo "한국어 단어 분할 테스트입니다." | litsea segment -l korean ./models/korean.model

English

Litsea supports English word segmentation with a character type set tuned for Latin-script orthography: case, whitespace, and the apostrophe as distinct classes.

Character Types

Uppercase as a Distinct Class

Sentence-initial capitals, proper nouns, and acronyms correlate strongly with word boundaries in English, so uppercase Latin gets its own type (“U”) instead of collapsing into the same class as lowercase (“A”). This mirrors how the other languages carve out a linguistically distinctive subset (e.g. Korean’s particle characters) from a broader shared class.

Whitespace as a Distinct Class

The shared punct_latin_digit() helper used by every language does not classify ASCII punctuation or the ASCII space (U+0020) — both fall through to "O" for Japanese, Chinese, and Korean, since none of those languages’ corpora need horizontal whitespace to carry a boundary signal on its own. English does: the type table adds “W” for space, tab, and no-break space (only the plain space occurs in the training corpus; the other two share its id so pasted input inherits the same behavior instead of falling back to “O”).

Apostrophe as a Distinct Class

The apostrophe is the character-level signal that separates a contraction or possessive from ordinary punctuation: do + n't, Google + 's. It gets its own type (“Q”) covering both the ASCII apostrophe (U+0027) and the typographic right single quotation mark (U+2019), which is common in the training corpus’s source text. Q is deliberately excluded from P so the character-level feature templates can key on it directly.

Punctuation Is Uniform

Unlike the other three languages — where ASCII punctuation such as @ falls through to "O" and only CJK/full-width punctuation maps to "P" — English classifies essentially all ASCII punctuation (minus the apostrophe) as "P", alongside the same General Punctuation range (dashes, curly quotes, ellipsis) that covers non-ASCII editions of the training text. This is a deliberate, English-specific difference: char_type('@') returns "O" for Japanese/Chinese/Korean but "P" for English.

Hyphen is classified as "P" rather than given its own type. UD English-EWT tokenizes hyphenated compounds as separate tokens (e.g. search-engine), so a hyphen behaves like ordinary separator punctuation in the gold standard; the raw character is still visible to the character-level (UW*/BW*) templates, so hyphen-specific behavior remains learnable without an eighth type code, which would grow the dense feature tables by a further \((8/7)^3 \approx 1.49\times\).

No WC Features

English does not use WC (word + character-type) features, the same choice as Korean and for a related reason: the dominant boundary signal (whitespace) already resolves most positions, so the mixed char/type templates add little on top of it. This was verified empirically, not just by analogy — on a held-out dev split, the tag-free segmentation model scored 98.68% Word F1 with the 38 base templates versus 98.65% with all 42 templates (WC1–WC4 included), i.e. adding WC features made the model worse, not better.

Space-Preserving Training

English is written with spaces between words, and (outside contractions and a few punctuation cases) those spaces mark most word boundaries. Like Korean, the model is trained on a space-preserving TSV corpus: tokens are tab-separated and each space is kept as its own token, so the training text contains the space characters of the original sentence and the model can use them as boundary context. Generate the corpus with corpus_udtreebank.sh -s and extract features with litsea extract --format tsv. At inference no special handling is needed: segment() receives the spaced text as-is and emits each space as its own token.

Multiword tokens (contractions). UD English-EWT represents a contraction such as don't as a range line (e.g. ids 3-4) covering two word lines (do, n't) with no space between them. corpus_udtreebank.sh -s treats a range line specially: it emits no token of its own, suppresses space insertion between the range’s member words, and applies the range’s own SpaceAfter annotation after the last member word. Concretely, the sentence “I don’t know.” becomes the token sequence I, , do, n't, , know, . – matching english.model’s actual output (see the example below). This invariant — concatenating a range’s member word forms reproduces the range’s own surface form — holds for every multiword token in UD English-EWT.

Because each space is its own single-character token, the character-level labeling marks two separate boundaries around it, exactly as for Korean: see Korean’s explanation of why this is a near-trivial rule for the model and does not affect held-out Word F1 (pure-whitespace tokens are excluded from scoring).

Pre-trained Models

english.model

• Training corpus: UD English-EWT (space-preserving TSV corpus)
• Training options: --format tsv --tag-free, 20 epochs of Averaged Perceptron training (chosen by a dev-split epoch sweep over {10, 20, 30, 50}; quality peaked at epoch 20 and degraded slightly beyond it), collapsed to AdaBoost scalar weights, not pruned (4,794 features) — see Training Procedure for the full recipe
• Word F1 (held-out): 98.31%
• Boundary F1 (held-out): 99.18%
• File size: ~125 KB

The model is trained without the 16 tag-dependent feature templates (--tag-free, issue #183). A dev-split comparison confirmed tag features buy almost nothing for English (tagged 38-template best: 98.71% Word F1 at epoch 30, vs. tag-free 38-template best: 98.68% at epoch 20 — a 0.03pt difference), so the bundled model ships tag-free and lets segment() skip its sequential scoring pass entirely. See Tag-Free (Pointwise) Models.

Held-out metrics are computed on the original spaced text with space tokens excluded from scoring.

english_pos.model

• Algorithm: two-stage segmentation + POS tagging (a binary boundary classifier plus a word-level tagger with a candidate-tag lexicon)
• Stage-2 feature set: full (chosen by a dev-split sweep over fast/balanced/full; full gave the best tagged-word accuracy), 50 epochs
• Word F1 (held-out): 98.30%
• Tagged Word F1 (held-out): 90.55%
• File size: ~3.1 MB
• Details: see Pre-trained Models

This model is trained on the same space-preserving corpus as english.model (issue #198), so its Word F1 (98.30%) essentially matches english.model’s 98.31% — the two-stage stage-1 classifier is now as good at finding English word boundaries as the dedicated segmentation model.

That was not always true, and the history is worth knowing if you are comparing against older numbers. The two-stage pipeline originally trained on an unspaced concatenation of the word/POS corpus, throwing away the spaces English text actually contains. That model scored 70.33% on the same unspaced protocol and 77.55% on real spaced input — it merged a test into a single token, for instance. Switching the training corpus gained +20.8pt Word F1 and +20.7pt tagged-word F1, and made tagging ~3.6x faster (2.05M → 7.32M chars/s).

The unspaced corpus caused two separate train/inference mismatches. Stage 1 never saw the spaces that mark nearly every English word boundary. Stage 2’s context features (L*/R*/cl*/cr*) were hit too: at inference a word’s neighbour is usually a space, but during unspaced training it was the next word’s character. Both now match what segment --pos computes.

Spaces are re-emitted as their own tokens and tagged X — the corpus gives whitespace a single-candidate lexicon entry, so the packed model takes its fixed-tag path rather than guessing with the classifier (the pre-#198 model returned PUNCT/PART/AUX for different spaces in one sentence). The golden test in litsea/tests/golden.rs pins this.

Example

echo "I don't know." | litsea segment -l english ./models/english.model
# I   do n't   know .

Adding a New Language

Litsea’s multilingual framework is designed to be easily extensible. This guide explains how to add support for a new language, using the addition of English (issue #194) as the worked example throughout.

Steps Overview

1. Add a variant to the Language enum
2. Implement Display and FromStr match arms
3. Create a character classification function
4. Register the classification function
5. Decide on WC feature inclusion
6. Choose a corpus protocol (space-separated or space-preserving TSV)
7. Train the bundled segmentation model (binary-perceptron-collapse recipe)
8. Optionally train a two-stage POS model
9. Add held-out evaluation gold files
10. Add tests

Step 1: Add a Variant to Language

In litsea/src/language.rs, add a new variant to the Language enum:

#![allow(unused)]
fn main() {
#[non_exhaustive]
pub enum Language {
    #[default]
    Japanese,
    Chinese,
    Korean,
    English,
    Thai,       // ← new language
}
}

The enum is marked #[non_exhaustive] precisely because new languages are expected to be added, so adding a variant is not a breaking change for downstream crates.

Step 2: Implement Display and FromStr

Add match arms for the new language:

#![allow(unused)]
fn main() {
// In Display impl
Language::Thai => write!(f, "thai"),

// In FromStr impl
"thai" | "th" => Ok(Language::Thai),
}

Also update the ParseLanguageError message in language.rs: it enumerates the supported languages (Supported: japanese (ja), chinese (zh), korean (ko), english (en)) and is pinned by a unit test, so both the message and the test must include the new language.

Step 3: Create a Character Classification Function

Define a function that classifies a char into a type id for the new language. Ids are indices into the language’s ordered type_codes() table (Step 4): the shared classes occupy fixed indices (“O” = 0, “P” = 1, “A” = 2, “N” = 3) and language-specific classes follow from 4. Classification is a direct match on character ranges (no regex), so each class is an arm; the first matching arm wins:

#![allow(unused)]
fn main() {
fn thai_char_type_id(c: char) -> u8 {
    match c {
        // Thai consonants and sequential vowels (U+0E01-U+0E3A)
        '\u{0E01}'..='\u{0E3A}' => 4, // "T"
        // Thai vowels and tone marks (U+0E40-U+0E4E)
        '\u{0E40}'..='\u{0E4E}' => 5, // "V"
        // Thai digits (U+0E50-U+0E59)
        '\u{0E50}'..='\u{0E59}' => DIGIT_TYPE_ID, // "N"
        // Shared classes: "P" (punctuation), "A" (Latin), "N" (digits)
        _ => punct_latin_digit(c).unwrap_or(OTHER_TYPE_ID),
    }
}
}

English’s english_char_type_id is a real, in-tree example of the same pattern applied to a Latin-script language: it adds “U” (uppercase), “W” (whitespace), and “Q” (apostrophe) as dedicated classes, and additionally widens the shared “P” class to cover ASCII punctuation (which the other languages leave as “O”) – a language’s classification function is free to layer extra logic in front of punct_latin_digit(), not just append new classes after it.

Design Tips for Character Types

• Identify linguistically distinct categories that correlate with word boundary patterns
• Order matters – match arms are tried top to bottom, so put more specific classes before general ones
• Consider high-frequency function words as a separate type (as Chinese does with “F”), or, for a space-delimited language, whatever punctuation/case/diacritic distinctions actually correlate with boundaries (as English does with “U”/“W”/“Q”)
• Use extra logic inside an arm body when a plain range is not enough (as Korean does with a codepoint test to split syllables with/without 받침)
• Reuse the shared punct_latin_digit() helper for the common “P”/“A”/“N” classes
• Keep the code set prefix-free – no code may be a prefix of another (Korean’s SN/SF work because S alone is not a code; a bare "S" is therefore rejected for every language by a unit test, not just Korean). The model loader decodes concatenated codes left to right when compiling packed feature keys, and prefix-freeness is what makes that decoding unambiguous
• The type table needs at least 7 codes. A shared test context (packed_model.rs’s ctx_for) asserts codes.len() >= 7; English’s 7-code table is the current minimum. There is no fixed upper bound, but the dense feature tables (BC/UC/TC/BQ/TQ) scale roughly with type_count^2 to type_count^3, so a much larger table trades model size and load time for classification granularity

Step 4: Register the Type-Code Table and Classification Function

Add the language’s ordered code table to Language::type_codes() (index = type id; shared codes first) and a dispatch arm in Language::char_type_id(). char_type() itself is derived from these two, so string codes and numeric ids cannot drift apart:

#![allow(unused)]
fn main() {
pub(crate) fn type_codes(self) -> &'static [&'static str] {
    match self {
        // ...
        Language::Thai => &["O", "P", "A", "N", "T", "V"],    // ← new
    }
}

pub(crate) fn char_type_id(self, c: char) -> u8 {
    match self {
        // ...
        Language::Thai => thai_char_type_id(c),    // ← new
    }
}
}

Step 5: Decide on WC Feature Inclusion

The feature template is defined once in packed_model.rs (TEMPLATES), and templates_for() decides whether a language uses the trailing WC1–WC4 char/type mixed templates:

#![allow(unused)]
fn main() {
pub(crate) fn templates_for(language: Language) -> &'static [Template] {
    match language {
        Language::Japanese | Language::Chinese => &TEMPLATES[..],
        Language::Korean | Language::English => &TEMPLATES[..BASE_TEMPLATE_COUNT], // 38 base templates
    }
}
}

This match is deliberately exhaustive, with no wildcard arm – adding Thai without adding it to one of the two arms is a compile error, not a silent default. This is intentional: an earlier version of this match had a _ => &TEMPLATES[..BASE_TEMPLATE_COUNT] fallback, which meant a new language got the 38-template (no WC) configuration without anyone deciding that on purpose. Make the WC decision explicitly, and back it with a measurement rather than an assumption: train a tag-free model both with and without WC on a held-out dev split and compare Word F1 (English’s comparison, documented in English, found WC measured worse, not just unhelpful – 98.68% without vs. 98.65% with). As a starting heuristic: if your language’s character types have enough variety to make WC features informative, include them; if your type system is low-entropy (like Korean’s/English’s dominant “SN”/“A” or whitespace-dominated distribution), exclude them – but verify with numbers before committing bundled models to either choice.

Step 6: Choose a Corpus Protocol

Litsea supports two corpus protocols, and the right choice depends on whether the language is written with spaces:

• Space-separated (the default): words joined with a single space, one sentence per line. Used for languages written without spaces (Japanese, Chinese) or where spacing carries no boundary signal.
• Space-preserving TSV (--format tsv, issue #152): tab-separated tokens where a token may itself be a literal space " ", so the original spacing survives into training as a first-class feature. Used for languages where the space itself is the strongest boundary signal (Korean, English).

If your language is written with spaces between words (like English, unlike Korean’s inter-eojeol convention but the same underlying reasoning), use the space-preserving protocol:

conllu_file=$(bash scripts/download_udtreebank.sh -l en -o /tmp)
bash scripts/corpus_udtreebank.sh -s "$conllu_file" corpus.tsv
litsea extract -l english --format tsv --tag-free corpus.tsv features.txt

If the source treebank has multiword tokens (contractions, clitics), verify corpus_udtreebank.sh -s handles them correctly before trusting the corpus. UD CoNLL-U represents a contraction like English’s don't as a range line (e.g. 3-4 don't) covering two word lines (do, n't) that carry no space between them. corpus_udtreebank.sh -s treats a range line specially: it emits no token of its own, suppresses space insertion between the range’s member words, and applies the range’s own SpaceAfter annotation only after the last member word. The invariant to check for a new treebank is: concatenating a range’s member word forms must reproduce the range’s own surface form. This held for every multiword token in UD English-EWT (verified by scripting a comparison across the full corpus before training); if it does not hold for your treebank, the safe fallback is to emit the range’s own form as a single token instead of expanding its members. As a second, independent check, reconstruct each sentence by concatenating its TSV tokens (space tokens included) and diff the result against the CoNLL-U file’s # text = metadata line for that sentence – this catches any spacing bug the member-concatenation check alone would miss. When you change corpus_udtreebank.sh itself, also regenerate an existing space-preserving gold file (e.g. resources/eval/korean_gsd_test.tsv) and diff it against the committed version – it should come out byte-identical if your change is additive.

Step 7: Train the Bundled Segmentation Model

The bundled segmentation models are not trained with plain litsea train -t/-i (AdaBoost boosting). They are trained as a 2-class Averaged Perceptron and losslessly collapsed to the AdaBoost model format – see Pre-trained Models: Training Procedure for the full derivation and the exact five-step recipe (extract → relabel 1/-1 to B/O → train --perceptron → scripts/collapse_binary_perceptron.py → optionally prune). For a space-preserving-protocol language:

litsea extract -l english --format tsv --tag-free corpus.tsv features.txt
sed -i 's/^1\t/B\t/; s/^-1\t/O\t/' features.txt
litsea train --perceptron --num-epochs 20 features.txt perceptron.model
scripts/collapse_binary_perceptron.py perceptron.model models/english.model

Do not guess the epoch count or the tag-free/WC decisions – run a sweep on a held-out dev split (never the test split, which should be touched only once at the end) across a handful of epoch counts, and compare tag-free vs. tagged and (per Step 5) WC vs. no-WC at the best epoch count for each. English’s sweep, for example, found quality peaking at 20 epochs and degrading slightly beyond it – a one-shot low-epoch run would have understated the model’s real quality, and a one-shot high-epoch run would have looked like overfitting was the ceiling rather than a convergence point already passed.

Step 8: Optionally Train a Two-Stage POS Model

If UPOS-tagged data is available for the language, you can additionally train a two-stage model:

bash scripts/corpus_udtreebank.sh -p "$conllu_file" pos_corpus.txt
litsea extract --pos -l english --stage2-features full pos_corpus.txt pos_features
litsea train --pos --num-epochs 50 pos_features models/english_pos.model

Sweep --stage2-features (fast/balanced/full) on a dev split the same way as the epoch sweep above; the three bundled languages that use this path each picked a different winner (see Choosing a stage-2 feature set).

For a space-delimited language, train two-stage on the space-preserving corpus too (issue #198). Generate it with corpus_udtreebank.sh -p -s and extract with --pos --format tsv:

bash scripts/corpus_udtreebank.sh -p -s "$conllu_file" pos_corpus.tsv
litsea extract --pos --format tsv -l english --stage2-features full pos_corpus.tsv pos_features

This matters far more than it might look. Training on an unspaced concatenation costs Korean ~5.9pt of Word F1 and English ~20.8pt, because it creates two train/inference mismatches at once: stage 1 never sees the spaces that mark most word boundaries, and stage 2’s context features (L*/R*/cl*/cr*) see a different neighbour during training than at inference. Whitespace tokens get no stage-2 row (they would be ~43% of rows for one degenerate class) but do get a lexicon entry, which makes them single-candidate and therefore tagged deterministically through the packed model’s fixed-tag path — also skipping the classifier for ~43% of tokens, which is a throughput win.

For a language written without spaces (Japanese, Chinese), use the plain -p corpus: there is no spacing to preserve, and the space-separated format already matches the real input.

Step 9: Add Held-out Evaluation Gold Files

Generate held-out gold data from the treebank’s test split (touched only after your dev-split sweeps are done) and add it under resources/eval/, following the existing naming convention (<language>_<treebank>_test.{txt,tsv} for segmentation, <language>_<treebank>_test_pos.txt for POS):

bash scripts/corpus_udtreebank.sh -s "$conllu_file_test" resources/eval/english_ewt_test.tsv
bash scripts/corpus_udtreebank.sh -p "$conllu_file_test" resources/eval/english_ewt_test_pos.txt
litsea evaluate -l english --format tsv models/english.model resources/eval/english_ewt_test.tsv
litsea evaluate -l english --pos models/english_pos.model resources/eval/english_ewt_test_pos.txt

Update resources/eval/README.md with the new file(s) and their provenance/license (UD treebanks are typically CC BY-SA 4.0, distinct from the rest of the repository’s MIT/Apache-2.0 licensing). Record the held-out numbers you get from litsea evaluate – not numbers from litsea train’s in-sample printout – in the model’s documentation (Step 10 lists every doc page that needs the new numbers).

Step 10: Add Tests and Documentation

This is the step most likely to be under-scoped: a new language touches more test files than just language.rs/segmenter.rs. Work through this checklist:

Code tests:

• litsea/src/language.rs: ALL_LANGUAGES (bump the array size), test_language_from_str, test_parse_language_error_message (new full error string), test_language_display, and a new test_<language>_char_types covering every type code plus a couple of shared-class and “O” cases
• litsea/src/packed_model.rs: test_templates_for_language_gating (assert the new language’s template count), and the two hard-coded language-enumeration arrays in test_pack_parse_roundtrip_unique_and_injective and test_dense_index_consistent_with_key_decode
• litsea/src/segmenter.rs: a new test_char_type_<language> (mirroring the existing per-language ones)
• litsea/src/word_features.rs: add a case to the round-trip sample list (cheap, catches type-code encoding bugs early)
• litsea-cli/src/main.rs: the three --language help strings

Model-dependent tests (need a trained model first):

• litsea/tests/golden.rs: a new golden_segment_<language> (and, if you trained a POS model, golden_segment_with_pos_<language>_two_stage). Pin the model’s actual output, not an idealized one – write the test with a println! of the real output first, copy that into the assertion, then delete the debug print. Immediately after, sabotage-check it: temporarily change one expected value to something wrong, confirm the test goes RED, then restore the correct value. A golden test that never failed during its own creation has not proven it protects anything
• litsea/src/segmenter.rs: a differential test (test_segment_differential_<language>_model) comparing the packed scorer against the string-keyed reference, and, if the model is tag-free, a segment_into tiling/parity case
• litsea/benches/bench.rs: add the new language to all four per-language tuple lists (bench_segment_short, bench_external_corpus’s two case lists, bench_segment_into), which requires a corpus file under resources/ (a public-domain text of a similar size to the existing per-language corpora)
• litsea-cli/tests/cli.rs: at least one segmentation smoke test pinning CLI output end-to-end

Verification commands (run all of these before considering the language done):

cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo bench -- external_corpus   # sanity-check the new bench cases run; decide if pruning is needed
markdownlint-cli2 "docs/src/**/*.md"
markdownlint-cli2 "docs/ja/src/**/*.md"
mdbook build docs
mdbook build docs/ja

Documentation (English source first, then the Japanese mirror under docs/ja/src/, per this project’s documentation policy):

• A new language-support/<language>.md page (use English or Korean as the template) plus a docs/src/SUMMARY.md entry (and its docs/ja/src/SUMMARY.md mirror)
• language-support/overview.md and algorithm/character-type-classification.md – both have a per-language table/section to extend
• pre-trained-models.md – a model card per bundled model, plus the “Tag-Free (Pointwise) Models” and “Two-Stage POS Tagging Models” comparison tables if applicable
• Root README.md and the crate-level docs in litsea/src/lib.rs
• Before considering the sweep complete, run grep -rln "<an existing language's name>" docs/src docs/ja/src and check every file it finds – language-enumerating docs are easy to miss (this project’s own experience: issue #165 skipped this check once and left 14 stale mentions for a later PR to clean up)

Library API Overview

The litsea crate provides a Rust API for word segmentation, model training, and feature extraction.

Installation

[dependencies]
litsea = "0.13.0"

Loading models from local files is synchronous and needs no async runtime. An async runtime such as tokio is only required when loading models over HTTP/HTTPS with the async load_model method (for example TwoStageLearner::load_model, which always resolves the model URI through the same async path).

Module Map

graph LR
    A["litsea::segmenter"] --- B["Segmenter"]
    C["litsea::adaboost"] --- D["AdaBoost"]
    E["litsea::language"] --- F["Language"]
    G["litsea::extractor"] --- H["Extractor"]
    I["litsea::trainer"] --- J["Trainer, PerceptronTrainer, TwoStageTrainer, TwoStageMetrics"]
    K["litsea::error"] --- L["LitseaError, Result"]
    M["litsea::perceptron"] --- N["AveragedPerceptron"]
    O["litsea::upos"] --- P["Upos, SegmentLabel"]
    Q["litsea::metrics"] --- R["BinaryMetrics, MulticlassMetrics"]
    S["litsea::evaluation"] --- T["PosMetrics, SegmentationMetrics"]
    U["litsea::two_stage"] --- V["ModelKind, TwoStageFeatureSet, TwoStageLearner"]

All primary types are also re-exported at the crate root, so use litsea::Segmenter; works as a shorthand for use litsea::segmenter::Segmenter;.

The learners resolve their own URIs, so model_io::read_model_bytes is rarely needed directly. It is public for callers that must inspect a model before choosing a learner — litsea-binding-core reads the bytes once, detects the kind with ModelKind::detect, and feeds the same bytes to load_model_from_reader, which avoids downloading a remote model twice.

Quick Example

use std::path::Path;

use litsea::adaboost::AdaBoost;
use litsea::language::Language;
use litsea::segmenter::Segmenter;

fn main() -> litsea::Result<()> {
    let mut learner = AdaBoost::new(0.01, 100);
    learner.load_model_from_path(Path::new("./models/RWCP.model"))?;

    let segmenter = Segmenter::with_learner(Language::Japanese, learner);
    let tokens = segmenter.segment("これはテストです。");

    assert_eq!(tokens, vec!["これ", "は", "テスト", "です", "。"]);
    Ok(())
}

Quick Example (POS Tagging)

use std::path::Path;

use litsea::language::Language;
use litsea::segmenter::Segmenter;
use litsea::two_stage::TwoStageLearner;

fn main() -> litsea::Result<()> {
    let mut learner = TwoStageLearner::new();
    learner.load_model_from_path(Path::new("./models/japanese_pos.model"))?;

    let segmenter = Segmenter::with_two_stage_learner(Language::Japanese, learner);
    let tokens = segmenter.segment_with_pos("これはテストです。")?;

    for (word, pos) in &tokens {
        print!("{}/{} ", word, pos);
    }
    println!();

    Ok(())
}

Full API documentation is available on docs.rs/litsea.

Segmenter

The Segmenter struct is the primary interface for word segmentation.

Definition

#![allow(unused)]
fn main() {
pub struct Segmenter {
    // private: language: Language,
    // private: learner: AdaBoost,
    // private: two_stage: Option<PackedTwoStageModel> (compiled stage-2 model)
    // internal: packed cache (see below)
}
}

The fields are private; use the accessor methods language(), learner(), and learner_mut() to reach them.

Besides these, the struct also holds packed: a lazily-rebuilt cache of the learner’s weights compiled into the integer-indexed tables segment() scores against (see Prediction Pipeline). It is internal implementation detail with no accessor of its own, invalidated automatically whenever the learner is mutated. two_stage holds the compiled stage-2 tagging model set by with_two_stage_learner (see below); it is None unless the segmenter was built from a two-stage model. Unlike the cache it is not derived from a retained learner — the raw stage-2 parts are dropped after compilation and there is no mutation path for it.

Constructors

Segmenter::new

#![allow(unused)]
fn main() {
pub fn new(language: Language) -> Self
}

Creates a segmenter with a default (untrained) AdaBoost learner — suitable for training or feature extraction. Until a model is loaded or training data is added, segment returns one word per character. No two-stage model is set; segment_with_pos returns Err(LitseaError::PosLearnerNotSet) — use with_two_stage_learner for segmentation + POS tagging.

Segmenter::with_learner

#![allow(unused)]
fn main() {
pub fn with_learner(language: Language, learner: AdaBoost) -> Self
}

Creates a segmenter with the given learner, typically one that has loaded a pre-trained model.

#![allow(unused)]
fn main() {
use litsea::language::Language;
use litsea::segmenter::Segmenter;

// With a pre-trained model
let segmenter = Segmenter::with_learner(Language::Japanese, learner);

// Without a model (for training or feature extraction)
let segmenter = Segmenter::new(Language::Japanese);
}

Methods

segment

#![allow(unused)]
fn main() {
pub fn segment(&self, sentence: &str) -> Vec<String>
}

Segments a sentence into words. Returns an empty vector for empty input.

#![allow(unused)]
fn main() {
let tokens = segmenter.segment("これはテストです。");
// ["これ", "は", "テスト", "です", "。"]
}

Internally this is a thin wrapper over segment_into with a fresh buffer per call, materializing each range as an owned String — there is a single scoring implementation.

segment_into / SegmentBuffer

#![allow(unused)]
fn main() {
pub struct SegmentBuffer { /* internal scratch + output storage */ }

impl SegmentBuffer {
    pub fn new() -> Self
}

impl Segmenter {
    pub fn segment_into<'b>(
        &self,
        sentence: &str,
        buf: &'b mut SegmentBuffer,
    ) -> &'b [(usize, usize)]
}
}

The allocation-free variant of segment (issue #184). Each returned (start, end) pair is a byte range into sentence (&sentence[start..end] is the token), in order, tiling the sentence exactly. The buffer owns every per-call allocation (context arrays, score buffer, tag scratch, output ranges); reusing one buffer across a batch of sentences reaches a steady state where segmentation allocates nothing. Empty input yields an empty slice.

At the published throughputs this matters: segment allocates one String per token (millions per second in batch workloads) plus per-call scratch, which measured as roughly a quarter of the batch profile. The buffer holds plain data (no borrows), so it can be reused across sentences, models, and languages; for parallel processing use one buffer per thread.

#![allow(unused)]
fn main() {
use litsea::segmenter::{SegmentBuffer, Segmenter};

let mut buf = SegmentBuffer::new();
for line in lines {
    for &(start, end) in segmenter.segment_into(line, &mut buf) {
        let token: &str = &line[start..end];
        // write/inspect token without allocating
    }
}
}

char_type

#![allow(unused)]
fn main() {
pub fn char_type(&self, c: char) -> &'static str
}

Classifies a character into its language-specific type code (delegates to Language::char_type).

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Language::Japanese);
assert_eq!(segmenter.char_type('あ'), "I");  // Hiragana
assert_eq!(segmenter.char_type('漢'), "H");  // Kanji
assert_eq!(segmenter.char_type('A'), "A");   // ASCII
}

add_corpus

#![allow(unused)]
fn main() {
pub fn add_corpus(&mut self, corpus: &str)
}

Processes a space-separated corpus and adds instances to the internal AdaBoost learner.

#![allow(unused)]
fn main() {
let mut segmenter = Segmenter::new(Language::Japanese);
segmenter.add_corpus("テスト です");
}

add_corpus_with_writer

#![allow(unused)]
fn main() {
pub fn add_corpus_with_writer<F>(&self, corpus: &str, writer: F)
where
    F: FnMut(HashSet<String>, i8),
}

Processes a corpus and calls the callback for each character position with its feature set and label.

#![allow(unused)]
fn main() {
segmenter.add_corpus_with_writer("テスト です", |attrs, label| {
    println!("Features: {:?}, Label: {}", attrs, label);
});
}

add_corpus_tsv / add_corpus_tsv_with_writer

#![allow(unused)]
fn main() {
pub fn add_corpus_tsv(&mut self, corpus: &str)
pub fn add_corpus_tsv_with_writer<F>(&self, corpus: &str, writer: F)
where
    F: FnMut(HashSet<String>, i8),
}

Tab-separated variants of add_corpus / add_corpus_with_writer: tokens are separated by tab characters, and a token may be a literal space " ". This preserves the original spacing of the sentence in the training text so the model can learn from space characters as boundary context (used for the Korean and English models; issue #152).

#![allow(unused)]
fn main() {
let mut segmenter = Segmenter::new(Language::Korean);
segmenter.add_corpus_tsv("나는\t \t고양이");
}

Accessors

#![allow(unused)]
fn main() {
pub fn language(&self) -> Language
pub fn learner(&self) -> &AdaBoost
pub fn learner_mut(&mut self) -> &mut AdaBoost
}

Provide access to the segmenter’s language and its internal learner (for a two-stage segmenter, the stage-1 boundary classifier).

Feature extraction for a character position (38 features for Korean/English, 42 for Japanese/Chinese) is an internal detail; the former get_attributes method is now private.

POS-Mode API

The segmenter also supports word segmentation and POS tagging with a two-stage model (issue #147).

with_two_stage_learner

#![allow(unused)]
fn main() {
pub fn with_two_stage_learner(language: Language, learner: TwoStageLearner) -> Self
}

Creates a segmenter with a two-stage model (a litsea-two-stage v1 file loaded into a TwoStageLearner): the model’s stage-1 boundary classifier becomes the segmenter’s AdaBoost-path learner (so segment works naturally), and segment_with_pos tags each segmented word through the candidate-tag lexicon — single-candidate and dominant surfaces skip the classifier entirely — with the stage-2 word-level tagger deciding ambiguous surfaces (candidate-masked argmax) and unknown surfaces (full argmax over all classes). See Model File Format for the two-stage format.

segment_with_pos

#![allow(unused)]
fn main() {
pub fn segment_with_pos(&self, sentence: &str) -> Result<Vec<(String, Upos)>>
}

Segments a sentence with the stage-1 boundary classifier (exactly as segment) and tags each word with its UPOS tag through the two-stage tagging path. An empty sentence yields Ok with an empty vector.

Errors with LitseaError::PosLearnerNotSet if no two-stage learner is set — build the segmenter with with_two_stage_learner() first.

#![allow(unused)]
fn main() {
use std::path::Path;

use litsea::language::Language;
use litsea::segmenter::Segmenter;
use litsea::two_stage::TwoStageLearner;

let mut learner = TwoStageLearner::new();
learner.load_model_from_path(Path::new("./models/japanese_pos.model"))?;

let segmenter = Segmenter::with_two_stage_learner(Language::Japanese, learner);
let tokens = segmenter.segment_with_pos("これはテストです。")?;
// [("これ", Upos::PRON), ("は", Upos::ADP), ("テスト", Upos::NOUN),
//  ("です", Upos::AUX), ("。", Upos::PUNCT)]
}

add_corpus_with_pos_writer

#![allow(unused)]
fn main() {
pub fn add_corpus_with_pos_writer<F>(&self, corpus: &str, writer: F)
where
    F: FnMut(HashSet<String>, SegmentLabel)
}

Streams the character-level training features of a POS-tagged corpus (word/POS word/POS ...), including the first position, to a custom writer, without mutating the segmenter. This is what Extractor::extract_two_stage builds its stage-1 boundary features on.

Extractor

The Extractor struct extracts features from a corpus file for model training.

Definition

#![allow(unused)]
fn main() {
pub struct Extractor {
    segmenter: Segmenter,
}
}

Constructor

Extractor::new

#![allow(unused)]
fn main() {
pub fn new(language: Language) -> Self
}

Creates a new extractor for the specified language. Internally creates a Segmenter without a pre-trained model. Extractor also implements Default, which is equivalent to Extractor::new(Language::Japanese).

#![allow(unused)]
fn main() {
use litsea::extractor::Extractor;
use litsea::language::Language;

let extractor = Extractor::new(Language::Japanese);
}

The extraction methods take &self, so the binding does not need to be mutable.

Methods

extract

#![allow(unused)]
fn main() {
pub fn extract(
    &self,
    corpus_path: &Path,
    features_path: &Path,
) -> litsea::Result<()>
}

Reads a corpus file (space-separated words, one sentence per line) and writes the extracted features to the output file.

#![allow(unused)]
fn main() {
use std::path::Path;

extractor.extract(
    Path::new("./corpus.txt"),
    Path::new("./features.txt"),
)?;
}

Pipeline

flowchart LR
    A["corpus.txt<br/>(space-separated words)"] --> B["Extractor::extract()"]
    B --> C["features.txt<br/>(label + features per position)"]

The extractor:

1. Reads each line from the corpus file
2. Calls Segmenter::add_corpus_with_writer() to process each line
3. Writes the label and feature set for each character position to the output file

extract_tsv

#![allow(unused)]
fn main() {
pub fn extract_tsv(
    &self,
    corpus_path: &Path,
    features_path: &Path,
) -> litsea::Result<()>
}

Reads a tab-separated corpus file (tokens separated by tabs, one sentence per line; a token may be a literal space " ") and writes the extracted features. The preserved spaces let the model learn from space characters as boundary context — used to train the Korean and English models (issue #152). Output format is identical to extract.

#![allow(unused)]
fn main() {
use std::path::Path;

extractor.extract_tsv(
    Path::new("./ko_corpus.tsv"),
    Path::new("./ko_features.txt"),
)?;
}

extract_tag_free / extract_tsv_tag_free

#![allow(unused)]
fn main() {
pub fn extract_tag_free(
    &self,
    corpus_path: &Path,
    features_path: &Path,
) -> litsea::Result<()>

pub fn extract_tsv_tag_free(
    &self,
    corpus_path: &Path,
    features_path: &Path,
) -> litsea::Result<()>
}

The tag-free variants of extract / extract_tsv (issue #183): identical input and output formats, but the 16 tag-dependent templates (UP*/BP*/UQ*/BQ*/TQ*, which read the previous boundary decisions) are dropped from every row. A model trained on these features is pointwise, so segment() skips its sequential scoring pass entirely. The bundled korean.model/english.model are trained this way; see Tag-Free (Pointwise) Models for the measured per-language quality/speed trade-off. These back the CLI’s extract --tag-free.

extract_two_stage

#![allow(unused)]
fn main() {
pub fn extract_two_stage(
    &self,
    corpus_path: &Path,
    output_prefix: &Path,
    feature_set: TwoStageFeatureSet,
) -> litsea::Result<()>
}

Reads a POS-tagged corpus (word/POS word/POS ..., one sentence per line, POS tags from the UPOS tagset) in a single pass and writes the three files consumed by TwoStageTrainer, used to train a two-stage model, from output_prefix:

• {output_prefix}.stage1 – boundary features (label\tfeature1\t..., label B or O), using the same character-level feature templates as plain extraction, emitted at every position including the first
• {output_prefix}.stage2 – word-level features (label\tfeature1\t..., label a UPOS tag), using the templates selected by feature_set (see TwoStageFeatureSet below)
• {output_prefix}.lexicon – the candidate-tag lexicon (surface\tTAG:count[,TAG:count...], most-frequent-first)

TwoStageTrainer::new reads the same three paths back from the same prefix.

#![allow(unused)]
fn main() {
use std::path::Path;

use litsea::TwoStageFeatureSet;

extractor.extract_two_stage(
    Path::new("./pos_corpus.txt"),
    Path::new("./pos_features"),
    TwoStageFeatureSet::Fast,
)?;
}

In-memory extraction

Every extract* method has a *_to_writer twin that takes the corpus as a string and writes the feature rows to any Write, for callers with no filesystem (WebAssembly) or with the corpus already in memory. The output is byte-identical to the path-based method.

#![allow(unused)]
fn main() {
use litsea::{Extractor, Language};

let extractor = Extractor::new(Language::Japanese);
let corpus = "これ は テスト です 。\n";

let mut features = Vec::new();
extractor.extract_to_writer(corpus, &mut features)?;
}

The two-stage variant writes the three outputs that the path version puts in {prefix}.stage1, .stage2, and .lexicon:

#![allow(unused)]
fn main() {
let (mut stage1, mut stage2, mut lexicon) = (Vec::new(), Vec::new(), Vec::new());
extractor.extract_two_stage_to_writers(
    corpus,
    &mut stage1,
    &mut stage2,
    &mut lexicon,
    TwoStageFeatureSet::Fast,
)?;
}

Feed the result to TwoStageTrainer::from_features.

The path-based methods are unavailable on wasm32-unknown-unknown, which has no filesystem; the *_to_writer twins compile everywhere.

TwoStageFeatureSet

#![allow(unused)]
fn main() {
pub enum TwoStageFeatureSet {
    Full,
    Balanced,
    #[default]
    Fast,
}
}

Selects which stage-2 word-level templates extract_two_stage writes (see Word-Level Feature Templates for the full template catalog), trading tagging quality for throughput:

• Full – every word template (quality-leaning)
• Balanced – the Fast templates plus first/last char identity and the word type string
• Fast (default) – the minimal measured set: surface, word length, first/last char type, adjacent context char + type, 2-char prefix/suffix

Also implements Display (lowercase: "full", "balanced", "fast") and FromStr (returns ParseTwoStageFeatureSetError for invalid strings) – the same names the --stage2-features CLI flag accepts; see Extracting Features.

Trainer

The Trainer struct orchestrates the full model training pipeline.

Definition

#![allow(unused)]
fn main() {
pub struct Trainer {
    learner: AdaBoost,
}
}

Constructor

Trainer::new

#![allow(unused)]
fn main() {
pub fn new(
    threshold: f64,
    num_iterations: usize,
    features_path: &Path,
) -> litsea::Result<Self>
}

Creates a trainer and initializes it from a features file. This calls AdaBoost::initialize_features() and AdaBoost::initialize_instances().

#![allow(unused)]
fn main() {
use std::path::Path;
use litsea::trainer::Trainer;

let mut trainer = Trainer::new(
    0.0001,                          // threshold
    20000,                           // max iterations
    Path::new("./features.txt"),     // features file
)?;
}

Methods

load_model

#![allow(unused)]
fn main() {
pub async fn load_model(&mut self, uri: &str) -> litsea::Result<()>
}

Loads an existing model for retraining. Supports file paths, file://, and (with the remote_model feature) http:// and https:// URIs.

When called after Trainer::new, the loaded weights are merged into the freshly initialized training data by feature name, so incremental training starts from the existing model without corrupting the feature index.

#![allow(unused)]
fn main() {
trainer.load_model("./models/japanese.model").await?;
}

train

#![allow(unused)]
fn main() {
pub fn train(
    &mut self,
    running: &AtomicBool,
    model_path: &Path,
) -> litsea::Result<BinaryMetrics>
}

Trains the model and saves it to the specified path. Returns evaluation metrics.

The running flag enables graceful interruption – set it to false to stop training early.

#![allow(unused)]
fn main() {
use std::sync::atomic::AtomicBool;
use std::path::Path;

let running = AtomicBool::new(true);
let metrics = trainer.train(&running, Path::new("./model.model"))?;

println!("Accuracy: {:.2}%", metrics.accuracy);
}

Full Training Example

use std::sync::atomic::AtomicBool;
use std::path::Path;

use litsea::trainer::Trainer;

#[tokio::main]
async fn main() -> litsea::Result<()> {
    let mut trainer = Trainer::new(
        0.0001,
        20000,
        Path::new("./features.txt"),
    )?;

    // Optionally resume from an existing model
    // trainer.load_model("./models/japanese.model").await?;

    let running = AtomicBool::new(true);
    let metrics = trainer.train(&running, Path::new("./model.model"))?;

    println!("Accuracy:  {:.2}%", metrics.accuracy);
    println!("Precision: {:.2}%", metrics.precision);
    println!("Recall:    {:.2}%", metrics.recall);

    Ok(())
}

PerceptronTrainer

PerceptronTrainer is the generic Averaged Perceptron counterpart of Trainer: it trains a multiclass Averaged Perceptron over opaque string labels from a features file (litsea train --perceptron). Its main use is training the 2-class (B/O) boundary perceptron that the collapse recipe (see Pre-trained Models) turns into the bundled AdaBoost-format segmentation models.

PerceptronTrainer::new

#![allow(unused)]
fn main() {
pub fn new(num_epochs: usize, features_path: &Path) -> litsea::Result<Self>
}

Reads the features file (each line is label\tfeature1\tfeature2\t..., where labels are opaque strings, e.g. the boundary labels B/O) and registers the training instances.

PerceptronTrainer::load_model

#![allow(unused)]
fn main() {
pub async fn load_model(&mut self, model_uri: &str) -> litsea::Result<()>
}

Loads an existing perceptron model for incremental training. Classes already registered from the training data are merged with the model’s classes.

PerceptronTrainer::train

#![allow(unused)]
fn main() {
pub fn train(
    &mut self,
    running: &AtomicBool,
    model_path: &Path,
) -> litsea::Result<MulticlassMetrics>
}

Trains for the configured number of epochs, saves the model, and returns multiclass metrics (accuracy, macro precision, macro recall). The running flag enables graceful interruption, like Trainer::train.

use std::sync::atomic::AtomicBool;
use std::path::Path;

use litsea::trainer::PerceptronTrainer;

#[tokio::main]
async fn main() -> litsea::Result<()> {
    let mut trainer = PerceptronTrainer::new(10, Path::new("./features.txt"))?;
    let running = AtomicBool::new(true);
    let metrics = trainer.train(&running, Path::new("./perceptron.model"))?;
    println!("Accuracy: {:.2}%", metrics.accuracy);
    Ok(())
}

TwoStageTrainer

TwoStageTrainer trains the two-stage model (issue #147): a binary boundary classifier (stage 1) plus a word-level multiclass tagger (stage 2), both Averaged Perceptrons, assembled with a candidate-tag lexicon into a single litsea-two-stage v1 file. After training, stage 1 is collapsed to scalar per-feature weights in the existing AdaBoost format (a lossless transformation – see this module’s source docs for the derivation), so the runtime scores it exactly as it scores a plain segment() model. Both TwoStageTrainer and TwoStageMetrics are re-exported from the crate root as litsea::TwoStageTrainer / litsea::TwoStageMetrics.

TwoStageTrainer::new

#![allow(unused)]
fn main() {
pub fn new(
    num_epochs: usize,
    dominance: f64,
    features_prefix: &Path,
) -> litsea::Result<Self>
}

Reads the three files written by Extractor::extract_two_stage from features_prefix ({prefix}.stage1, {prefix}.stage2, {prefix}.lexicon) and registers the training instances for both stages.

dominance is the classifier-skip threshold of the assembled model: a known word whose most frequent tag covers at least this fraction of its training occurrences is tagged without invoking the stage-2 classifier. It must be in (0.5, 1.0] and is validated eagerly in new(), so an out-of-range value fails immediately rather than after training runs.

#![allow(unused)]
fn main() {
use std::path::Path;
use litsea::trainer::TwoStageTrainer;

let trainer = TwoStageTrainer::new(
    50,                            // num_epochs (both stages)
    0.99,                          // dominance
    Path::new("./features"),       // features prefix
)?;
}

TwoStageTrainer::train

#![allow(unused)]
fn main() {
pub fn train(
    mut self,
    running: &AtomicBool,
    model_path: &Path,
) -> litsea::Result<TwoStageMetrics>
}

Unlike Trainer::train and PerceptronTrainer::train, this method takes self by value (it consumes the trainer). It trains both stages as Averaged Perceptrons for num_epochs epochs each, collapses stage 1 to AdaBoost weights, assembles the two stages with the lexicon into a litsea-two-stage v1 model, saves it to model_path, and returns the in-sample metrics of both stages. The running flag enables graceful interruption, like the other trainers.

use std::sync::atomic::AtomicBool;
use std::path::Path;

use litsea::trainer::TwoStageTrainer;

#[tokio::main]
async fn main() -> litsea::Result<()> {
    let trainer = TwoStageTrainer::new(50, 0.99, Path::new("./features"))?;
    let running = AtomicBool::new(true);
    let metrics = trainer.train(&running, Path::new("./model.model"))?;

    println!("Stage 1: {:.2}%, Stage 2: {:.2}%", metrics.stage1.accuracy, metrics.stage2.accuracy);

    Ok(())
}

TwoStageMetrics

#![allow(unused)]
fn main() {
pub struct TwoStageMetrics {
    pub stage1: MulticlassMetrics,
    pub stage2: MulticlassMetrics,
}
}

The in-sample metrics of a TwoStageTrainer::train run. stage1 measures the boundary classifier over its two classes (B/O); stage2 measures the word-level tagger over the UPOS tag classes. Both fields are MulticlassMetrics – the same type PerceptronTrainer::train returns above, exposing accuracy plus macro-averaged precision and recall.

In-memory training

Each trainer has an in-memory counterpart to its path-based constructor and train, so the whole pipeline can run without a filesystem — on wasm32-unknown-unknown, for instance, where the path-based methods are compiled out.

#![allow(unused)]
fn main() {
use litsea::{Extractor, Language, Trainer};
use std::sync::atomic::AtomicBool;

let corpus = "これ は テスト です 。\n";

let mut features = Vec::new();
Extractor::new(Language::Japanese).extract_to_writer(corpus, &mut features)?;
let features = String::from_utf8(features).expect("features are UTF-8");

let mut model = Vec::new();
let metrics = Trainer::from_features(0.01, 10_000, &features)?
    .train_to_writer(&AtomicBool::new(true), &mut model)?;
}

The features arrive as a &str rather than a reader because AdaBoost scans them twice: once to build the feature vocabulary, once to build the instances against it.

Both routes produce the same model, byte for byte, which the crate’s tests assert.

Training the segmentation model directly

For the AdaBoost pipeline the learner’s own API is enough, without going through a features file at all:

#![allow(unused)]
fn main() {
use litsea::{AdaBoost, Language, Segmenter};

let mut learner = AdaBoost::new(0.01, 10_000);
let segmenter = Segmenter::new(Language::Japanese);
segmenter.add_corpus_with_writer(corpus, |attrs, label| learner.add_instance(attrs, label));
learner.train(&AtomicBool::new(true));
learner.save_model_to_writer(&mut model)?;
}

This is equivalent for a fresh learner. It is not equivalent when continuing from a loaded model: Trainer’s two-pass route seeds each instance’s boosting weight from the existing model, while add_instance starts every instance at 1.0.

Reproducibility

A training run is a function of its input: the same features trained twice produce the same model. This holds because AveragedPerceptron::add_instance stores its features sorted — HashSet iteration order varies between sets, and perceptron updates are order-sensitive, so before that two runs in one process could disagree.

AdaBoost

The AdaBoost struct implements binary classification for word boundary detection.

Definition

#![allow(unused)]
fn main() {
pub struct AdaBoost {
    // private: threshold: f64, num_iterations: usize
    // (read via threshold() / num_iterations())
    // internal fields: model weights, features, instances, etc.
}
}

Constructor

AdaBoost::new

#![allow(unused)]
fn main() {
pub fn new(threshold: f64, num_iterations: usize) -> Self
}

Creates a new AdaBoost instance with the specified hyperparameters.

#![allow(unused)]
fn main() {
use litsea::adaboost::AdaBoost;

let mut learner = AdaBoost::new(0.01, 100);
}

AdaBoost::default

AdaBoost also implements Default, which is equivalent to AdaBoost::new(0.01, 100) – the default hyperparameters used across the library and CLI.

Accessors

threshold

#![allow(unused)]
fn main() {
pub fn threshold(&self) -> f64
}

Returns the early-stopping threshold this learner was created with.

num_iterations

#![allow(unused)]
fn main() {
pub fn num_iterations(&self) -> usize
}

Returns the maximum number of boosting iterations.

Model Loading

load_model_from_path

#![allow(unused)]
fn main() {
pub fn load_model_from_path(&mut self, path: &Path) -> litsea::Result<()>
}

Loads model weights from a local file, synchronously. Malformed files (empty, missing bias line, duplicate bias lines or features, non-finite weights) are rejected with LitseaError::InvalidData. This is the preferred method for local files – no async runtime is needed.

#![allow(unused)]
fn main() {
use std::path::Path;

learner.load_model_from_path(Path::new("./models/japanese.model"))?;
}

load_model_from_reader

#![allow(unused)]
fn main() {
pub fn load_model_from_reader<R: BufRead>(&mut self, reader: R) -> litsea::Result<()>
}

Loads model weights from any BufRead source, such as an in-memory buffer or an already-open file.

load_model

#![allow(unused)]
fn main() {
pub async fn load_model(&mut self, uri: &str) -> litsea::Result<()>
}

Loads model weights from a URI. Supports:

• Local file path: ./models/japanese.model
• File URI: file:///path/to/model
• HTTP: http://example.com/model (requires the remote_model feature)
• HTTPS: https://example.com/model (requires the remote_model feature)

#![allow(unused)]
fn main() {
learner.load_model("https://example.com/model").await?;
}

save_model

#![allow(unused)]
fn main() {
pub fn save_model(&self, filename: &Path) -> litsea::Result<()>
}

Saves model weights to a file. Returns an error if the model is empty.

save_model_to_writer

#![allow(unused)]
fn main() {
pub fn save_model_to_writer<W: Write>(&self, writer: &mut W) -> litsea::Result<()>
}

Writes the model to an arbitrary writer in the same text format as save_model; this is the format-producing core save_model delegates to. It is public so the model can be embedded as a section of a larger file without going through a file path – the two-stage model format uses it to embed the stage-1 AdaBoost model directly. The writer is not flushed. Returns an error if the model is empty.

Training Methods

initialize_features

#![allow(unused)]
fn main() {
pub fn initialize_features(&mut self, filename: &Path) -> litsea::Result<()>
}

Reads a features file and builds the feature index. Must be called before initialize_instances.

initialize_instances

#![allow(unused)]
fn main() {
pub fn initialize_instances(&mut self, filename: &Path) -> litsea::Result<()>
}

Reads the same features file and initializes labeled instances with their weights.

train

#![allow(unused)]
fn main() {
pub fn train(&mut self, running: &AtomicBool)
}

Runs the AdaBoost training loop. Set running to false to stop early.

add_instance

#![allow(unused)]
fn main() {
pub fn add_instance(&mut self, attributes: HashSet<String>, label: i8)
}

Adds a single training instance with its feature set and label.

Prediction

predict

#![allow(unused)]
fn main() {
pub fn predict(&self, attributes: &HashSet<String>) -> i8
}

Predicts the label for a given feature set. Returns +1 (boundary) or -1 (non-boundary).

#![allow(unused)]
fn main() {
use std::collections::HashSet;

let mut attrs = HashSet::new();
attrs.insert("UW4:は".to_string());
attrs.insert("UC4:I".to_string());
// ... more features

let label = learner.predict(&attrs);
// label == 1 (boundary) or -1 (non-boundary)
}

bias

#![allow(unused)]
fn main() {
pub fn bias(&self) -> f64
}

Returns the bias term: -sum(all model weights) / 2.0. The value is cached and kept in sync by every weight-mutating path, so this is O(1).

Evaluation

metrics

#![allow(unused)]
fn main() {
pub fn metrics(&self) -> BinaryMetrics
}

Calculates evaluation metrics on the training data.

BinaryMetrics

Defined in litsea::metrics (also re-exported as litsea::BinaryMetrics):

#![allow(unused)]
fn main() {
pub struct BinaryMetrics {
    pub accuracy: f64,          // Accuracy in percentage
    pub precision: f64,         // Precision in percentage
    pub recall: f64,            // Recall in percentage
    pub num_instances: usize,
    pub true_positives: usize,
    pub false_positives: usize,
    pub false_negatives: usize,
    pub true_negatives: usize,
}
}

Averaged Perceptron

The AveragedPerceptron struct implements multiclass classification over opaque string labels. It is the training-side learner behind both stages of the two-stage POS architecture and the bundled segmentation models’ collapse recipe (litsea train --perceptron).

Definition

#![allow(unused)]
fn main() {
pub struct AveragedPerceptron {
    // internal fields: slots (feature -> per-class weights + averaging state), step, classes, instances
}
}

Constructor

AveragedPerceptron::new

#![allow(unused)]
fn main() {
pub fn new() -> Self
}

Creates a new empty Averaged Perceptron instance.

#![allow(unused)]
fn main() {
use litsea::perceptron::AveragedPerceptron;

let mut learner = AveragedPerceptron::new();
}

Adding Instances

add_instance

#![allow(unused)]
fn main() {
pub fn add_instance(&mut self, features: HashSet<String>, label: String)
}

Adds a training instance with a feature set and a label. Unknown classes are automatically registered.

#![allow(unused)]
fn main() {
use std::collections::HashSet;
use litsea::perceptron::AveragedPerceptron;

let mut learner = AveragedPerceptron::new();
let mut feats = HashSet::new();
feats.insert("UW4:猫".to_string());
feats.insert("UC4:H".to_string());
learner.add_instance(feats, "B-NOUN".to_string());
}

Training

train

#![allow(unused)]
fn main() {
pub fn train(&mut self, num_epochs: usize, running: &AtomicBool)
}

Runs the Averaged Perceptron training loop for the given number of epochs. Set running to false to stop early. Weights are automatically averaged at the end of training.

#![allow(unused)]
fn main() {
use std::sync::atomic::AtomicBool;

let running = AtomicBool::new(true);
learner.train(10, &running);
}

Prediction

predict

#![allow(unused)]
fn main() {
pub fn predict(&self, features: &HashSet<String>) -> String
}

Predicts the class label for a given feature set. Computes a score for each class and returns the class name with the highest score. Returns an empty string if no classes are registered.

#![allow(unused)]
fn main() {
use std::collections::HashSet;

let mut attrs = HashSet::new();
attrs.insert("UW4:は".to_string());
attrs.insert("UC4:I".to_string());
// ... more features

let label = learner.predict(&attrs);
// label == "B-ADP", "O", etc.
}

Accessors

classes

#![allow(unused)]
fn main() {
pub fn classes(&self) -> &[String]
}

Returns the registered class names in their sorted storage order – the order used for weight-vector indexing and predict’s argmax tie-breaking (first strictly-greater class wins). Empty if no classes are registered. Used by the two-stage collapse procedure (see Pre-trained Models) and by the packed two-stage runtime.

Model I/O

save_model

#![allow(unused)]
fn main() {
pub fn save_model(&self, path: &Path) -> litsea::Result<()>
}

Saves model weights to a file. Returns an error if the model is empty.

save_model_to_writer

#![allow(unused)]
fn main() {
pub fn save_model_to_writer<W: Write>(&self, writer: &mut W) -> litsea::Result<()>
}

Writes the model to an arbitrary writer in the same text format as save_model; this is the format-producing core save_model delegates to. It is public so the model can be embedded as a section of a larger file without going through a file path – the two-stage model format uses it to embed the stage-2 word tagger directly. The writer is not flushed. Returns an error if no classes are registered (an empty model).

load_model_from_path

#![allow(unused)]
fn main() {
pub fn load_model_from_path(&mut self, path: &Path) -> litsea::Result<()>
}

Loads model weights from a local file, synchronously. This is the preferred method for local files – no async runtime is needed.

#![allow(unused)]
fn main() {
use std::path::Path;

learner.load_model_from_path(Path::new("./perceptron.model"))?;
}

load_model_from_reader

#![allow(unused)]
fn main() {
pub fn load_model_from_reader<R: BufRead>(&mut self, reader: R) -> litsea::Result<()>
}

Loads model weights from any BufRead source, such as an in-memory buffer or an already-open file.

load_model

#![allow(unused)]
fn main() {
pub async fn load_model(&mut self, uri: &str) -> litsea::Result<()>
}

Loads model weights from a URI. Supports the following URI schemes:

• Local file path: ./perceptron.model
• File URI: file:///path/to/model
• HTTP: http://example.com/model (requires the remote_model feature)
• HTTPS: https://example.com/model (requires the remote_model feature)

#![allow(unused)]
fn main() {
learner.load_model("https://example.com/models/perceptron.model").await?;
}

Evaluation

metrics

#![allow(unused)]
fn main() {
pub fn metrics(&self) -> MulticlassMetrics
}

Calculates evaluation metrics on the training data.

MulticlassMetrics

Defined in litsea::metrics (also re-exported as litsea::MulticlassMetrics):

#![allow(unused)]
fn main() {
pub struct MulticlassMetrics {
    pub accuracy: f64,                            // Overall accuracy in percentage
    pub macro_precision: f64,                     // Macro-averaged precision in percentage
    pub macro_recall: f64,                        // Macro-averaged recall in percentage
    pub num_instances: usize,                     // Number of instances
    pub correct_per_class: HashMap<String, usize>,   // Correct count per class
    pub predicted_per_class: HashMap<String, usize>,  // Predicted count per class
    pub gold_per_class: HashMap<String, usize>,       // Gold label count per class
}
}

UPOS

The upos module defines the Universal POS (UPOS) tagset and segment label types used for POS tagging.

Upos

Definition

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Upos {
    ADJ,    // Adjective
    ADP,    // Adposition
    ADV,    // Adverb
    AUX,    // Auxiliary
    CCONJ,  // Coordinating conjunction
    DET,    // Determiner
    INTJ,   // Interjection
    NOUN,   // Noun
    NUM,    // Numeral
    PART,   // Particle
    PRON,   // Pronoun
    PROPN,  // Proper noun
    PUNCT,  // Punctuation
    SCONJ,  // Subordinating conjunction
    SYM,    // Symbol
    VERB,   // Verb
    X,      // Other
}
}

Litsea supports all 17 UPOS tags from the Universal Dependencies project:

Constant

Upos::ALL

#![allow(unused)]
fn main() {
pub const ALL: [Upos; 17]
}

Returns an array of all 17 UPOS tags.

Trait Implementations

• Display: Converts to a string such as "NOUN", "VERB", etc.
• FromStr: Parses a string into Upos. Returns a ParseUposError for invalid strings.

#![allow(unused)]
fn main() {
use litsea::upos::Upos;

let pos: Upos = "NOUN".parse().unwrap();
assert_eq!(pos.to_string(), "NOUN");
}

ParseUposError

ParseUposError (re-exported at the crate root as litsea::ParseUposError) is returned when a string is not a valid UPOS tag. Its input() accessor returns the string that failed to parse, and the message reads Unknown UPOS tag: '<input>'.

SegmentLabel

Definition

The SegmentLabel type combines word boundary detection with POS tagging. Each character position is assigned one of 18 labels:

• B(Upos) (17 labels): Word boundary with the given UPOS tag (e.g., B-NOUN, B-VERB)
• O (1 label): Non-boundary (continuation of the current word)

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum SegmentLabel {
    B(Upos),  // Start of a word (boundary). Carries POS information.
    O,        // Continuation of a word (non-boundary).
}
}

#![allow(unused)]
fn main() {
use litsea::upos::SegmentLabel;

// Segment labels for "今日は" (kyou wa)
// 今 → B-NOUN  (start of "今日", tagged as NOUN)
// 日 → O       (continuation of "今日")
// は → B-ADP   (start of "は", tagged as ADP)
}

Methods

all_labels

#![allow(unused)]
fn main() {
pub fn all_labels() -> Vec<SegmentLabel>
}

Returns a vector of all 18 SegmentLabel values (not strings): the 17 B(Upos) labels followed by O.

is_boundary

#![allow(unused)]
fn main() {
pub fn is_boundary(&self) -> bool
}

Returns whether this is a boundary label (B-*).

pos

#![allow(unused)]
fn main() {
pub fn pos(&self) -> Option<Upos>
}

Returns the UPOS tag. Returns None for the non-boundary label (O).

Trait Implementations

• Display: Converts to a string such as "B-NOUN", "O", etc.
• FromStr: Parses a string into SegmentLabel. Returns a ParseSegmentLabelError for invalid strings.

#![allow(unused)]
fn main() {
use litsea::upos::{SegmentLabel, Upos};

let label: SegmentLabel = "B-NOUN".parse().unwrap();
assert!(label.is_boundary());
assert_eq!(label.pos(), Some(Upos::NOUN));

let label_o: SegmentLabel = "O".parse().unwrap();
assert!(!label_o.is_boundary());
assert_eq!(label_o.pos(), None);
}

ParseSegmentLabelError

ParseSegmentLabelError (re-exported at the crate root as litsea::ParseSegmentLabelError) is returned when a string is not a valid segment label. It has two variants:

• InvalidFormat – the string is neither O nor of the form B-<UPOS> (message: Invalid segment label: '<input>'. Expected 'O' or 'B-<UPOS>')
• InvalidPos – the B- prefix was present but the POS part failed to parse (wraps the underlying ParseUposError)

Language

The Language enum defines language-specific behavior, including character type classification.

Language Enum

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
#[non_exhaustive]
pub enum Language {
    #[default]
    Japanese,
    Chinese,
    Korean,
    English,
}
}

The enum is marked #[non_exhaustive] because new languages are expected to be added without a breaking change; external match expressions over Language therefore need a wildcard arm (_ => ...).

Traits

• Default – Returns Language::Japanese
• Display – Returns lowercase name ("japanese", "chinese", "korean", "english")
• FromStr – Parses from full name or ISO 639-1 code (case-insensitive)

Parsing

#![allow(unused)]
fn main() {
use litsea::language::Language;

// Full names
let ja: Language = "japanese".parse().unwrap();
let zh: Language = "chinese".parse().unwrap();
let ko: Language = "korean".parse().unwrap();
let en: Language = "english".parse().unwrap();

// ISO 639-1 codes
let ja: Language = "ja".parse().unwrap();
let zh: Language = "zh".parse().unwrap();
let ko: Language = "ko".parse().unwrap();
let en: Language = "en".parse().unwrap();

// Case-insensitive
let ko: Language = "KOREAN".parse().unwrap();

// Invalid
assert!("french".parse::<Language>().is_err());
}

char_type

#![allow(unused)]
fn main() {
pub fn char_type(&self, c: char) -> &'static str
}

Classifies a character into its language-specific type code. Returns "O" (Other) if the character does not belong to any class.

Classification is a direct match on character ranges – allocation-free, O(1), and with no regex involved.

#![allow(unused)]
fn main() {
use litsea::language::Language;

let lang = Language::Japanese;
assert_eq!(lang.char_type('あ'), "I");
assert_eq!(lang.char_type('漢'), "H");
assert_eq!(lang.char_type('@'), "O");
}

Internally, char_type is a table lookup over the numeric type id returned by a private per-language function (japanese_char_type_id, chinese_char_type_id, korean_char_type_id, english_char_type_id), so string codes and numeric ids cannot drift apart. The classes common to all languages – "P" (punctuation), "A" (Latin), and "N" (digits) – are handled by a shared helper that is checked after the language-specific classes (English widens "P" to cover ASCII punctuation too; see English).

ParseLanguageError

Parsing a Language from a string fails with ParseLanguageError, which is re-exported at the crate root (litsea::ParseLanguageError):

#![allow(unused)]
fn main() {
use litsea::language::{Language, ParseLanguageError};

let err: ParseLanguageError = "french".parse::<Language>().unwrap_err();
assert_eq!(err.input(), "french");
}

• input() – Returns the string that failed to parse
• The error message enumerates the supported languages: Unsupported language: 'french'. Supported: japanese (ja), chinese (zh), korean (ko), english (en)

Evaluation

Held-out quality evaluation of segmentation and POS tagging (litsea::evaluation). This is the library API behind the litsea evaluate subcommand. evaluate_pos evaluates a segmenter built with with_two_stage_learner (see Two-Stage Tagging) through segment_with_pos.

Metrics Types

#![allow(unused)]
fn main() {
pub struct SegmentationMetrics {
    pub word_precision: f64,     // %
    pub word_recall: f64,        // %
    pub word_f1: f64,            // %
    pub boundary_precision: f64, // %
    pub boundary_recall: f64,    // %
    pub boundary_f1: f64,        // %
    pub sentences: usize,
    pub gold_words: usize,
    pub predicted_words: usize,
}

pub struct PosMetrics {
    pub segmentation: SegmentationMetrics,
    pub tagged_precision: f64, // %: span and tag both match
    pub tagged_recall: f64,    // %
    pub tagged_f1: f64,        // %
}
}

Both are re-exported at the crate root. Tokens are matched by exact character-offset spans over the concatenation of the gold tokens; pure-whitespace tokens are excluded from scoring (the Korean/English space-preserving protocol; a no-op for languages written without spaces).

Functions

evaluate_segmentation

#![allow(unused)]
fn main() {
pub fn evaluate_segmentation<I, S>(segmenter: &Segmenter, gold: I) -> SegmentationMetrics
where
    I: IntoIterator<Item = Vec<S>>,
    S: Into<String>,
}

Segments the concatenation of each gold sentence’s tokens with [Segmenter::segment] and scores the result. Empty sentences are skipped.

evaluate_pos

#![allow(unused)]
fn main() {
pub fn evaluate_pos<I, S>(segmenter: &Segmenter, gold: I) -> litsea::Result<PosMetrics>
where
    I: IntoIterator<Item = Vec<(S, Upos)>>,
    S: Into<String>,
}

Like evaluate_segmentation but drives [Segmenter::segment_with_pos] and additionally scores tagged words. Returns LitseaError::PosLearnerNotSet if the segmenter has neither a POS learner nor a two-stage learner set.

Gold-line parsers

#![allow(unused)]
fn main() {
pub fn parse_gold_line(line: &str, tsv: bool) -> Vec<String>
pub fn parse_gold_pos_line(line: &str, tsv: bool) -> Vec<(String, Upos)>
}

Both split on spaces (or tabs with tsv = true, where a token may be a literal space – see English for why that matters for space-delimited languages); parse_gold_pos_line additionally splits each token at its last / (the training pipeline’s rule), defaulting to Upos::X for missing or unparsable tags. With tsv = true, a literal space token has no /POS suffix and always gets Upos::X, but this is harmless – whitespace tokens are excluded from tagged-word scoring by content, not by their assigned tag. This is the gold format for space-delimited languages’ two-stage POS models, which since issue #198 are also trained on this same space-preserving corpus.

Example

#![allow(unused)]
fn main() {
use litsea::adaboost::AdaBoost;
use litsea::evaluation::{evaluate_segmentation, parse_gold_line};
use litsea::language::Language;
use litsea::segmenter::Segmenter;

let mut learner = AdaBoost::new(0.01, 100);
learner.load_model_from_path(std::path::Path::new("./models/japanese.model"))?;
let segmenter = Segmenter::with_learner(Language::Japanese, learner);

let gold = std::fs::read_to_string("./resources/eval/japanese_gsd_test.txt")?;
let sentences = gold.lines().map(|l| parse_gold_line(l, false));
let metrics = evaluate_segmentation(&segmenter, sentences);
println!("word F1: {:.2}%", metrics.word_f1);
Ok::<(), Box<dyn std::error::Error>>(())
}

Two-Stage Model

The two_stage module defines the litsea-two-stage v1 model container (TwoStageLearner), the stage-2 feature-set selector (TwoStageFeatureSet), and the model-kind detector (ModelKind). See Two-Stage Tagging for the architecture and measured quality/speed figures, and TwoStageTrainer for training a model from scratch.

TwoStageLearner

#![allow(unused)]
fn main() {
pub struct TwoStageLearner {
    // private: stage1: AdaBoost,
    // private: stage2: AveragedPerceptron,
    // private: lexicon: HashMap<String, Vec<(Upos, u32)>>,
    // private: dominance: f64,
}
}

Owns the three parts of a two-stage model: a stage-1 boundary classifier (scalar weights, AdaBoost format), a candidate-tag lexicon, and a stage-2 word-level tagger (AveragedPerceptron). Follows the same construction and (de)serialization conventions as AdaBoost and AveragedPerceptron.

Constructors

#![allow(unused)]
fn main() {
pub fn new() -> Self
pub fn from_parts(
    stage1: AdaBoost,
    stage2: AveragedPerceptron,
    lexicon: impl IntoIterator<Item = (String, Vec<(Upos, u32)>)>,
    dominance: f64,
) -> Result<Self>
}

new creates an empty learner (fill it with a load_model* call before use). from_parts builds a learner from its pieces, validating the combination: dominance must be in (0.5, 1.0], every stage-2 class name must be a valid Upos tag, and every lexicon entry must have a non-empty surface (no tab/newline), a non-empty tag list with positive counts, and no duplicate tag or surface. Lexicon entries are normalized to the canonical order (count descending, ties by tag name ascending) regardless of input order.

Model I/O

#![allow(unused)]
fn main() {
pub fn save_model(&self, path: &Path) -> Result<()>
pub fn save_model_to_writer<W: Write>(&self, writer: &mut W) -> Result<()>
pub async fn load_model(&mut self, uri: &str) -> Result<()>
pub fn load_model_from_path(&mut self, path: &Path) -> Result<()>
pub fn load_model_from_reader<R: BufRead>(&mut self, reader: R) -> Result<()>
}

Same conventions as AdaBoost/AveragedPerceptron: save_model/load_model work with file paths or (for load_model) file:///http(s):// URIs (the latter requires the remote_model feature); the *_to_writer/*_from_reader variants work with any writer/reader. Saving an empty learner (no lexicon entries, or either embedded learner untrained) returns LitseaError::InvalidInput. Loading validates the full file structure — see Model File Format for the on-disk layout — and rejects malformed content with LitseaError::InvalidData; the learner is left unmodified on a load error.

#![allow(unused)]
fn main() {
use std::path::Path;

use litsea::two_stage::TwoStageLearner;

let mut learner = TwoStageLearner::new();
learner.load_model_from_path(Path::new("./models/japanese_pos.model"))?;
}

Accessors

#![allow(unused)]
fn main() {
pub fn stage1(&self) -> &AdaBoost
pub fn stage2(&self) -> &AveragedPerceptron
pub fn dominance(&self) -> f64
pub fn lexicon_len(&self) -> usize
pub fn lexicon_entry(&self, surface: &str) -> Option<&[(Upos, u32)]>
}

dominance is the classifier-skip threshold: at inference, a known surface whose most frequent tag covers at least this fraction of its training occurrences is tagged without invoking the stage-2 classifier at all. lexicon_entry returns the candidate tags observed for a surface during training, most-frequent-first, or None if the surface was never seen.

To actually run inference, install the learner on a Segmenter via Segmenter::with_two_stage_learner rather than calling into TwoStageLearner directly — the segmenter compiles it into packed scoring tables for fast lookup.

TwoStageFeatureSet

#![allow(unused)]
fn main() {
#[non_exhaustive]
pub enum TwoStageFeatureSet {
    Full,
    Balanced,
    #[default]
    Fast,
}
}

Selects which of the 23 word-level stage-2 templates (see Feature Extraction) get written by Extractor::extract_two_stage. Fast (the default) is the minimal measured set — surface, word length, first/last char type, adjacent context char + type, 2-char prefix/suffix. Balanced adds first/last char identity and the word type-code string. Full includes every template. Segmentation quality is identical across all three sets (it is decided entirely by stage 1); only tagging quality and throughput vary. The relative ordering of the three sets (not their exact figures, which were measured on an early prototype at a different epoch count than the bundled models) is documented on this type’s own rustdoc; see Pre-trained Models for the bundled models’ current, measured numbers.

Implements FromStr (case-insensitive: "full", "balanced", "fast") and Display (lowercase). Marked #[non_exhaustive] — external match expressions need a wildcard arm.

ModelKind

#![allow(unused)]
fn main() {
pub enum ModelKind {
    AdaBoost,
    AveragedPerceptron,
    TwoStage,
}
}

ModelKind::detect(content: &str) -> ModelKind inspects a model file’s first line to identify its format — a dispatch heuristic, not full validation. AdaBoost is the plain segmentation format (also the format of a collapsed two-stage stage 1); AveragedPerceptron is a standalone perceptron file (the output of train --perceptron and the payload format of the [stage2] section — this was the removed joint POS model format, and is not loadable as a POS model); TwoStage is the litsea-two-stage container.

Wrong-kind files get precise errors from TwoStageLearner’s loaders: pointing them at a standalone Averaged Perceptron file fails with “joint POS models are no longer supported — retrain with litsea train --pos”, and any other non-two-stage content fails with a missing-magic-line error.

CLI Reference Overview

The litsea CLI provides commands for word segmentation, model training, and text processing.

The CLI binary is built with the library’s remote_model feature enabled, so http(s):// model URIs work out of the box – unlike the litsea library itself, whose default features do not include remote loading.

Usage

litsea <COMMAND> [OPTIONS] [ARGS]

Commands

Global Options

Typical Workflow

AdaBoost Workflow (Word Segmentation Only)

flowchart LR
    A["1. scripts/download_udtreebank.sh"] --> B["2. scripts/corpus_udtreebank.sh"]
    B --> C["3. litsea extract"]
    C --> D["4. litsea train"]
    D --> E["5. litsea segment"]

1. Download a UD Treebank: conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp)
2. Convert to corpus format: bash scripts/corpus_udtreebank.sh "$conllu_file" corpus.txt
3. Extract features: litsea extract -l japanese corpus.txt features.txt
4. Train a model: litsea train -t 0.0001 -i 20000 features.txt model.model
5. Segment text: echo "text" | litsea segment -l japanese model.model

Two-Stage Workflow (Word Segmentation with POS Tagging)

flowchart LR
    A["1. scripts/download_udtreebank.sh"] --> B["2. scripts/corpus_udtreebank.sh -p"]
    B --> C["3. litsea extract --pos"]
    C --> D["4. litsea train --pos"]
    D --> E["5. litsea segment --pos"]

1. Download a UD Treebank: conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp)
2. Convert to POS corpus format: bash scripts/corpus_udtreebank.sh -p "$conllu_file" pos_corpus.txt
3. Extract two-stage features: litsea extract --pos -l japanese pos_corpus.txt features_prefix
4. Train a two-stage model: litsea train --pos --num-epochs 50 features_prefix model.model
5. Segment with POS tags: echo "text" | litsea segment --pos -l japanese model.model

See Two-Stage Tagging for the architecture, and Training Models or train for the full flag reference.

extract

Extract features from a corpus file for model training.

Usage

litsea extract [OPTIONS] <CORPUS_FILE> <FEATURES_FILE>

Arguments

Options

Corpus Format

The input corpus must have words separated by spaces, one sentence per line:

Litsea は TinySegmenter を 参考 に 開発 さ れ た 。
Rust で 実装 さ れ た コンパクト な 単語 分割 ソフトウェア です 。

TSV Corpus Format (--format tsv)

With --format tsv, tokens are separated by tab characters and a token may be a literal space " ". This preserves the original spacing of the sentence in the training text, which is essential for languages like Korean and English where spaces mark most word boundaries (see Korean and English). Generate such a corpus from a UD Treebank with corpus_udtreebank.sh -s:

litsea extract -l korean --format tsv ./ko_corpus.tsv ./ko_features.txt
litsea extract -l english --format tsv --tag-free ./en_corpus.tsv ./en_features.txt

Output Format

The features file contains one line per character position. For the corpus line これ は テスト です 。, the first two lines are:

-1	BC1:OI	BC2:II	BC3:II	BP1:UU	BP2:UU	BQ1:UOI	BQ2:UII	BQ3:UOI	BQ4:UII	BW1:B1こ	BW2:これ	...
1	BC1:II	BC2:II	BC3:IK	BP1:UU	BP2:UO	BQ1:UII	BQ2:UII	BQ3:OII	BQ4:OII	BW1:これ	BW2:れは	...

• 1 = word boundary
• -1 = non-boundary
• Features are written tab-separated in alphabetically sorted order, so each line starts with the BC1: feature rather than following the template definition order

Examples

# Japanese
litsea extract -l japanese ./corpus.txt ./features.txt

# Chinese
litsea extract -l zh ./corpus_zh.txt ./features_zh.txt

# Korean
litsea extract -l ko ./corpus_ko.txt ./features_ko.txt

# English
litsea extract -l en ./corpus_en.txt ./features_en.txt

Output to stderr on success:

Feature extraction completed successfully.

Two-Stage Feature Extraction

When the --pos flag is specified, extract expects a POS corpus instead of a plain word-separated corpus. Each line contains words annotated with UPOS tags in the format word/POS:

POS Corpus Format

これ/PRON は/ADP テスト/NOUN です/AUX 。/PUNCT
今日/NOUN は/ADP いい/ADJ 天気/NOUN です/AUX ね/PART 。/PUNCT

extract --pos writes three files derived from FEATURES_FILE as a prefix, for the two-stage segmentation + POS tagging architecture:

Pass the same prefix to litsea train --pos:

litsea extract --pos -l japanese ./pos_corpus.txt ./pos_features
# writes ./pos_features.stage1, .stage2, .lexicon

Space-Preserving POS Corpus (--pos --format tsv)

For a space-delimited language, combine --pos with --format tsv (issue #198). The corpus is then a tab-separated list of word/POS tokens in which a token may be a literal space carrying no /POS suffix — the format corpus_udtreebank.sh -p -s emits:

I/PRON	 	do/AUX	n't/PART	 	know/VERB	./PUNCT

bash scripts/corpus_udtreebank.sh -p -s "$conllu_file" ./pos_corpus.tsv
litsea extract --pos --format tsv -l english --stage2-features full ./pos_corpus.tsv ./pos_features

This is how the bundled korean_pos.model and english_pos.model are trained. Training on the unspaced corpus instead costs Korean ~5.9pt and English ~20.8pt of held-out Word F1, because stage 1 never sees the spaces that mark most word boundaries and stage 2’s context features see different neighbours than they will at inference.

Whitespace tokens get no stage-2 row — they are ~43% of tokens in a spaced corpus and would train one degenerate X class — but they do get a lexicon entry, which makes them single-candidate and therefore tagged deterministically through the model’s fixed-tag path, skipping the classifier entirely.

Japanese and Chinese should use plain --pos: their text has no spaces, so there is no spacing to preserve.

train

Train a word segmentation model using AdaBoost.

Usage

litsea train [OPTIONS] <FEATURES_FILE> <MODEL_FILE>

Arguments

Options

Output

Training metrics are printed to stderr:

Metrics are computed on the training data; with enough iterations the model can fit the training corpus almost perfectly, so evaluate on held-out text for a realistic quality estimate.

Result Metrics:
  Accuracy: 100.00% ( 1075868 / 1075869 )
  Precision: 100.00% ( 161283 / 161284 )
  Recall: 100.00% ( 161283 / 161283 )
  Confusion Matrix:
    True Positives: 161283
    False Positives: 1
    False Negatives: 0
    True Negatives: 914585

Ctrl+C Handling

Training supports graceful interruption:

• First Ctrl+C: Stops training and saves the model at its current state
• Second Ctrl+C: Exits immediately without saving

This allows you to stop long-running training sessions without losing progress.

Examples

Basic training:

litsea train -t 0.0001 -i 20000 ./features.txt ./models/my_model.model

This is a generic plain-AdaBoost example. The bundled japanese.model, chinese.model, korean.model, and english.model use a different procedure – see Training Procedure.

Training with higher precision (lower threshold, more iterations):

litsea train -t 0.001 -i 5000 ./features.txt ./model.model

Retraining from an existing model:

litsea train -t 0.0001 -i 20000 -m ./models/my_model.model \
    ./new_features.txt ./models/my_model_v2.model

Hyperparameter Tuning

Generic Perceptron Training

When the --perceptron flag is specified, train uses the Averaged Perceptron algorithm instead of AdaBoost. Labels are opaque strings, so this mode trains any multiclass classifier from a label\tfeature\t... features file. Its main use is training the 2-class (B/O) boundary perceptron of the bundled segmentation models’ collapse recipe (see Training Procedure).

Usage

litsea train --perceptron [OPTIONS] <FEATURES_FILE> <MODEL_FILE>

Perceptron Training Options

Examples

# Train a 2-class boundary perceptron (collapse recipe, step 3)
litsea train --perceptron --num-epochs 50 ./features.txt ./perceptron.model

Output

Perceptron training metrics are printed to stderr (macro-averaged precision and recall):

Result Metrics (Perceptron):
  Accuracy: 98.23% ( 277213 )
  Macro Precision: 96.82%
  Macro Recall: 93.30%

Ctrl+C Handling

Same as AdaBoost training, perceptron training supports graceful interruption. The first Ctrl+C stops training and saves the model at its current state.

Perceptron Hyperparameters

Two-Stage Model Training

With --pos, train builds a two-stage model: a binary boundary classifier (stage 1) plus a word-level POS tagger (stage 2), assembled with the candidate-tag lexicon into a single litsea-two-stage v1 file. Both stages train as Averaged Perceptrons for --num-epochs epochs; stage 1 is then collapsed to scalar weights in the existing AdaBoost format (a lossless transformation — see the module docs of litsea::trainer for the derivation) so the runtime scores it exactly as it scores a plain segment() model.

Usage

litsea train --pos [OPTIONS] <FEATURES_PREFIX> <MODEL_FILE>

FEATURES_PREFIX is the same prefix passed to extract --pos.

Example

litsea extract --pos -l japanese ./pos_corpus.txt ./pos_features
litsea train --pos --num-epochs 50 ./pos_features ./models/japanese_pos.model

Output

Result Metrics (Two-Stage):
  Stage 1 (boundary) Accuracy: 99.86% ( 277213 )
  Stage 1 Macro Precision: 99.85%
  Stage 1 Macro Recall: 99.86%
  Stage 2 (tagging) Accuracy: 99.09% ( 168333 )
  Stage 2 Macro Precision: 98.96%
  Stage 2 Macro Recall: 98.77%

As with the other modes, these are in-sample metrics; evaluate on held-out text with litsea evaluate --pos for a realistic quality estimate.

segment

Segment text into words using a trained model.

Usage

echo "text" | litsea segment [OPTIONS] <MODEL_URI>

Arguments

Options

Input / Output

• Input: Reads from stdin, one sentence per line. Empty lines are skipped.
• Output: Writes to stdout, space-separated tokens, one line per input line.
• Pipelines: A downstream consumer closing the pipe early (e.g. litsea segment model | head -1) terminates the command successfully (exit code 0), so segment composes cleanly in shell pipelines.

Examples

Japanese:

echo "LitseaはTinySegmenterを参考に開発された。" \
  | litsea segment -l japanese ./models/RWCP.model

Litsea は TinySegmenter を 参考 に 開発 さ れ た 。

Chinese:

echo "中文分词测试。" | litsea segment -l chinese ./models/chinese.model

Korean:

echo "한국어 단어 분할 테스트입니다." \
  | litsea segment -l korean ./models/korean.model

English:

echo "I don't know." | litsea segment -l english ./models/english.model

Processing a file:

cat input.txt | litsea segment -l japanese ./models/japanese.model > output.txt

Loading a model from a URL:

echo "テスト文です。" \
  | litsea segment -l japanese https://example.com/models/japanese.model

POS-Tagged Segmentation (--pos)

When the --pos flag is specified, segmentation and POS tagging are performed together with a two-stage model produced by train --pos. Pointing --pos at any other model kind fails with a precise error (a standalone Averaged Perceptron file — the removed joint POS model format — is rejected with a hint to retrain with train --pos).

Usage

echo "text" | litsea segment --pos [OPTIONS] <MODEL_URI>

Output Format

Each token is output in word/POS format. POS tags conform to the UPOS tag set.

echo "今日はいい天気ですね。" \
  | litsea segment --pos -l japanese ./models/japanese_pos.model

今日/NOUN は/ADP いい/ADJ 天気/NOUN です/AUX ね/PART 。/PUNCT

Processing a File

cat input.txt | litsea segment --pos -l japanese ./models/japanese_pos.model > output.txt

Parallel Batch Segmentation (--threads)

Sentences are independent, so batch throughput scales with cores without any engine change: lines are read in chunks, each chunk is split across the workers (each holding its own reusable segmentation buffer), and the outputs are written strictly in input order.

litsea segment --threads 8 -l japanese ./models/japanese.model < corpus.txt > segmented.txt

--threads 1 (the default) uses exactly the previous sequential loop. Note cargo bench -- external_corpus remains a single-threaded engine measurement — CLI-level thread scaling and engine throughput are different numbers and should not be compared directly (see Benchmarking).

Notes

• The --language flag must match the language the model was trained for
• The CLI loads models through the async URI API and supports HTTP/HTTPS with TLS (rustls); the library also offers synchronous local loading (load_model_from_path)
• The model URI is not restricted to file paths – any valid URL is accepted
• When using --pos, the model must be a two-stage model trained with train --pos

evaluate

Evaluate a trained model against a held-out gold corpus and print quality metrics. Unlike the in-sample metrics printed by train, this measures quality on text the model has never seen.

A gold corpus is a text file containing the correct answers: one sentence per line, already segmented into the correct tokens by human annotation (the same file formats used for training corpora — space- or tab-separated tokens, or word/POS with --pos). “Gold” refers to the gold standard the model’s output is judged against. For a meaningful held-out evaluation it must contain sentences that were not used to train the model — the bundled files in resources/eval/ are the UD GSD test splits, while the bundled models are trained on the train splits.

Usage

litsea evaluate [OPTIONS] <MODEL_URI> <GOLD_FILE>

Arguments

Options

Metrics

Two token sequences are compared for every sentence:

• Gold tokens – the reference segmentation from the gold corpus: the human-annotated correct answer (here, the tokenization of the UD GSD treebank test split). The evaluated sentence text is reconstructed by concatenating them.
• Predicted tokens – what the model produces when the reconstructed sentence text is fed to segment (or segment --pos), exactly as a user would at inference time.

Predicted and gold tokens are matched by exact character-offset spans over the reconstructed sentence. Pure-whitespace tokens are excluded from scoring, so the Korean/English space-preserving protocol does not inflate the numbers.

A word counts as correct only when both of its boundaries are correct, so word metrics are always at least as strict as boundary metrics — a single misplaced boundary invalidates the two words on either side of it. Sentences is the number of evaluated (non-empty) gold sentences.

Examples

Reproduce the documented held-out figures with the bundled gold data (resources/eval/, converted from the UD GSD test splits):

litsea evaluate -l japanese models/japanese.model resources/eval/japanese_gsd_test.txt
litsea evaluate -l korean --format tsv models/korean.model resources/eval/korean_gsd_test.tsv
litsea evaluate -l chinese models/chinese.model resources/eval/chinese_gsd_test.txt
litsea evaluate -l english --format tsv models/english.model resources/eval/english_ewt_test.tsv
litsea evaluate --pos -l japanese models/japanese_pos.model resources/eval/japanese_gsd_test_pos.txt

The Korean and English two-stage models are trained on the space-preserving corpus (issue #198), so evaluate them with --pos --format tsv against the *_pos_spaced.tsv gold – that protocol matches both their training and the spaced text segment --pos receives in practice:

litsea evaluate --pos --format tsv -l korean models/korean_pos.model resources/eval/korean_gsd_test_pos_spaced.tsv
litsea evaluate --pos --format tsv -l english models/english_pos.model resources/eval/english_ewt_test_pos_spaced.tsv

The *_pos.txt gold for those two languages measures the unspaced protocol they used before #198; it is kept only for reproducing the older published numbers (see Pre-trained Models).

Output:

Evaluation Metrics:
  Sentences: 543
  Word Precision: 96.73%
  Word Recall: 96.66%
  Word F1: 96.70%
  Boundary Precision: 98.63%
  Boundary Recall: 98.56%
  Boundary F1: 98.59%

Language Bindings

Litsea is a Rust library, but it is also usable from other languages. The bindings live in this repository as workspace members, so they are versioned and released together with litsea itself.

Crates

Design principles

These apply to every binding.

Models are never embedded

A binding package ships code only. The caller supplies a model as raw bytes, a filesystem path, a file:// path, or an http(s):// URL. The bundled models range from 84 KB to 8 MB each, so embedding all four languages would push a wheel or npm package past 20 MB; keeping models external also means a model can be updated without republishing the binding.

See Pre-trained Models for where to get a model.

The model kind is detected, not declared

The CLI needs --pos to know whether it is loading a segmentation model or a two-stage POS model. Bindings do not: they read the model bytes once and dispatch on the detected kind, so has_pos is a property of the loaded model rather than something the caller has to get right.

Cancellation is explicit

Training can run for a long time, and litsea’s trainers stop early when their running flag is cleared. The CLI drives that flag from a Ctrl-C handler, but a library must not install one: signal handling is process-global and the host language usually owns it already. Bindings therefore expose a cancellation token that the caller triggers.

Shared logic lives in one crate

Everything that is not FFI-specific — language-name parsing, model loading and kind detection, token offsets, trainer orchestration, error categories — lives in litsea-binding-core. Each binding only maps that surface onto its host language’s types and exception model.

litsea-binding-core

litsea-binding-core holds the FFI-independent logic shared by every Litsea language binding. It depends only on litsea (plus Tokio on native targets, for the blocking wrappers) and never on PyO3, napi, ext-php-rs, magnus, or wasm-bindgen, so it can be unit-tested without any host-language toolchain.

Installation

[dependencies]
litsea-binding-core = "0.13.0"

Module Map

graph LR
    A["litsea_binding_core::segmenter"] --- B["CoreSegmenter"]
    C["litsea_binding_core::model"] --- D["build_segmenter, read_model_file, read_model_uri"]
    E["litsea_binding_core::token"] --- F["TokenView"]
    G["litsea_binding_core::language"] --- H["parse_language, SUPPORTED_LANGUAGES"]
    I["litsea_binding_core::trainer"] --- J["CoreExtractor, CoreTrainer, CorePerceptronTrainer, CoreTwoStageTrainer"]
    K["litsea_binding_core::cancel"] --- L["CancelToken"]
    M["litsea_binding_core::error"] --- N["CoreError, ErrorKind, CoreResult"]
    O["litsea_binding_core::runtime"] --- P["block_on"]

Segmentation

#![allow(unused)]
fn main() {
use litsea::Language;
use litsea_binding_core::CoreSegmenter;

let segmenter = CoreSegmenter::from_path(Language::Japanese, "models/japanese.model".as_ref())?;

assert_eq!(
    segmenter.segment("これはテストです。"),
    vec!["これ", "は", "テスト", "です", "。"]
);
}

For space-delimited languages the whitespace is returned as its own token, so the tokens still reconstruct the input exactly — korean.model splits "안녕하세요 반갑습니다" into ["안녕하세요", " ", "반갑습니다"].

CoreSegmenter holds an Arc<Segmenter> plus a Mutex<SegmentBuffer>. Segmenter is Send + Sync and a segmenter built from a loaded model has its packed tables already compiled, so concurrent segment calls take only an internal read lock; the mutex protects the scratch buffer alone. One instance can therefore be shared across threads and reused indefinitely, which is what the bindings do.

Byte offsets are exact: tokens tile the input without gaps or overlaps, so &text[token.byte_start..token.byte_end] == token.surface holds for every token, including for space-preserving languages such as Korean and English.

Note that segment_with_pos_batch cannot amortize allocations the way segment_batch does — litsea has no buffer-reusing variant of segment_with_pos.

Model loading

All of them go through build_segmenter, which decides what to build from the model file itself:

Because the bytes are read once and then dispatched, a remote model is downloaded a single time.

Errors

CoreError carries an ErrorKind plus a message. The kinds are stable strings intended to be surfaced to the host language.

The set does not change when the remote_model feature is toggled, so a binding’s exception hierarchy stays fixed.

Training

Available on native targets only; feature extraction and training are file-based.

#![allow(unused)]
fn main() {
use litsea::Language;
use litsea_binding_core::{CancelToken, CoreExtractor, CoreTrainer, CorpusFormat};

CoreExtractor::new(Language::Japanese).extract(
    "corpus.txt".as_ref(),
    "features.txt".as_ref(),
    CorpusFormat::PlainText,
    false, // tag_free
)?;

let metrics = CoreTrainer::new(0.01, 10_000, "features.txt".as_ref())?
    .train(&CancelToken::new(), "japanese.model".as_ref())?;
println!("accuracy: {:.2}%", metrics.accuracy);
}

CoreTwoStageTrainer mirrors the CLI’s train --pos flow. It can only be used once, because litsea’s TwoStageTrainer::train consumes the trainer (stage 1 is collapsed into an AdaBoost model and cannot be retrained in place); a second call returns an InvalidArgument error, and is_available() reports the state.

Cancellation semantics

Cancelling is cooperative and is not an error:

• the trainer stops at its next check point,
• the partially trained model is still written to the destination path,
• and its metrics are returned normally.

Checks happen once per boosting iteration for AdaBoost training, and once per epoch and per instance for perceptron training, so perceptron training reacts far faster. CancelToken clones share one flag, so a token handed to a background thread can stop training that another thread is driving.

Platform support

On wasm32-unknown-unknown, trainer, runtime, read_model_file, and CoreSegmenter::from_path are compiled out — wasm32 has no filesystem and no blocking runtime. WASM callers fetch the model bytes in JavaScript and use CoreSegmenter::from_bytes.

Features

Python

litsea-python exposes Litsea to Python 3.10+ through PyO3 and maturin. It is published to PyPI as litsea.

Installation

pip install litsea

Wheels are built against the stable ABI (abi3-py310), so one wheel per platform covers every supported Python version.

Getting a model

The package contains no models. Download one from the models/ directory and pass its path — see Pre-trained Models.

There is no flag to say what kind of model you have: the file identifies itself, and has_pos reports what the loaded model can do.

Segmentation

from litsea import Language, Segmenter

seg = Segmenter.open(Language.JAPANESE, "models/japanese.model")

seg.segment("これはテストです。")
# ['これ', 'は', 'テスト', 'です', '。']

A language name works anywhere a Language does — Segmenter.open("ja", ...) and Segmenter.open("japanese", ...) are equivalent.

For space-delimited languages the whitespace is returned as its own token, so the tokens always reconstruct the input:

Segmenter.open("ko", "models/korean.model").segment("안녕하세요 반갑습니다")
# ['안녕하세요', ' ', '반갑습니다']

POS tagging

seg = Segmenter.open(Language.JAPANESE, "models/japanese_pos.model")

for token in seg.segment_with_pos("これはテストです。"):
    print(token.surface, token.pos.name, token.start, token.end)
# これ PRON 0 6
# は ADP 6 9
# テスト NOUN 9 18
# です AUX 18 24
# 。 PUNCT 24 27

start and end are byte offsets into the input, so text.encode()[token.start:token.end].decode() returns the surface. Calling segment_with_pos on a segmentation-only model raises PosUnavailableError.

API

Language and Upos are PyO3 classes, not enum.Enum subclasses: their members are class attributes, so iterate them with Language.all() and Upos.all() rather than for x in Language.

Training

from litsea import Extractor, Language, Trainer

Extractor(Language.JAPANESE).extract("corpus.txt", "features.txt")
metrics = Trainer(0.01, 10_000, "features.txt").train("japanese.model")
print(f"accuracy: {metrics.accuracy:.2f}%")

A TwoStageTrainer can only run once — training collapses stage 1 into an AdaBoost model, which consumes the trainer. available reports whether it can still be used, and a second train() raises InvalidArgumentError.

Cancelling

Training releases the GIL, so another thread can stop it:

import threading
from litsea import CancelToken, Trainer

cancel = CancelToken()
threading.Timer(60.0, cancel.cancel).start()
metrics = Trainer(0.01, 100_000, "features.txt").train("japanese.model", cancel=cancel)

Cancelling is not an error: training stops at its next check point, still writes the partially trained model, and returns its metrics. The binding never installs a signal handler, so Ctrl-C handling remains the application’s.

Errors

Every exception derives from LitseaError.

Threading and the GIL

A Segmenter is immutable and safe to share between threads. segment_batch, segment_with_pos_batch, extract, and every train release the GIL.

Single-sentence segment and segment_with_pos keep it. Releasing the GIL requires owning the input string (PyO3’s Ungil bound forbids touching Python-owned memory with the GIL released), and that copy costs more than segmenting one sentence. Use the batch methods for bulk work.

Development

make setup-venv            # create the venv and install the dev tools
make test-litsea-python    # cargo test + maturin develop + pytest
make lint-litsea-python    # clippy + ruff
make build-litsea-python   # build a release wheel into litsea-python/dist

The parity tests build the litsea CLI and compare the binding’s output against it, so the reference implementation — not a hardcoded expectation — decides what is correct.

Node.js

litsea-nodejs exposes Litsea to Node.js 20+ through napi-rs. It is published to npm as litsea, with prebuilt native binaries for Linux, macOS, and Windows on x64 and arm64.

For browsers, use litsea-wasm instead.

Installation

npm install litsea

Getting a model

The package contains no models. Download one from the models/ directory and pass its path — see Pre-trained Models. The model file identifies its own kind, so hasPos reports what the loaded model can do and no flag is needed.

Segmentation

import { Segmenter } from 'litsea'

const seg = Segmenter.open('japanese', 'models/japanese.model')

seg.segment('これはテストです。')
// [ 'これ', 'は', 'テスト', 'です', '。' ]

The language name and its ISO 639-1 code are interchangeable ('ja', 'japanese').

For space-delimited languages the whitespace is returned as its own token, so the tokens always reconstruct the input:

Segmenter.open('ko', 'models/korean.model').segment('안녕하세요 반갑습니다')
// [ '안녕하세요', ' ', '반갑습니다' ]

POS tagging

const seg = Segmenter.open('japanese', 'models/japanese_pos.model')

seg.segmentWithPos('これはテストです。')
// [ { surface: 'これ', start: 0, end: 6, pos: 'PRON' },
//   { surface: 'は', start: 6, end: 9, pos: 'ADP' },
//   { surface: 'テスト', start: 9, end: 18, pos: 'NOUN' },
//   { surface: 'です', start: 18, end: 24, pos: 'AUX' },
//   { surface: '。', start: 24, end: 27, pos: 'PUNCT' } ]

start and end are byte offsets. JavaScript string indices are UTF-16 code units, so slice with a Buffer:

Buffer.from(text).subarray(token.start, token.end).toString()   // === token.surface

pos is undefined on tokens from segmentTokens, which does no tagging.

API

Type definitions are generated by napi-rs and shipped as index.d.ts.

Asynchronous by design

Downloading a model, extracting features, and training all return promises and run on libuv’s threadpool, so the event loop keeps turning. That is what makes cancellation useful:

import { CancelToken, Trainer } from 'litsea'

const cancel = new CancelToken()
setTimeout(() => cancel.cancel(), 60_000)

const metrics = await new Trainer(0.01, 100_000, 'features.txt').train('japanese.model', cancel)

Cancelling is not an error: training stops at its next check point, still writes the partially trained model, and resolves with its metrics. The binding never installs a signal handler.

Segmentation itself is synchronous: it is fast enough that a promise would cost more than the work.

A TwoStageTrainer can only be used once — training collapses stage 1 into an AdaBoost model, which consumes it. available reports the state, and a second train() rejects.

Errors

Every error carries a code, matching the error kinds the other bindings expose. Rejected promises carry the same codes as thrown errors.

Because napi::Status is a closed enum, the code reaches JavaScript two ways: synchronous calls use napi’s string-status error, and asynchronous ones rebuild the JavaScript Error object in Task::reject so the property survives the rejection.

Development

make test-litsea-nodejs    # cargo test + napi build + node --test
make lint-litsea-nodejs    # clippy
make build-litsea-nodejs   # release build

index.js and index.d.ts are generated by napi build and committed; CI rebuilds them and fails if the committed copies are stale. The parity tests build the litsea CLI and compare the binding’s output against it.

PHP

litsea-php exposes Litsea to PHP 8.1+ through ext-php-rs. It is distributed on Packagist as litsea/litsea.

Installation

A PHP extension is a shared object built against a specific PHP ABI, so unlike PyPI and npm there is no prebuilt package: you build it and enable it.

cargo build --release -p litsea-php
php -d extension=/path/to/target/release/liblitsea_php.so your-script.php

Add it to php.ini (extension=/path/to/liblitsea_php.so) to load it everywhere. The build needs a Rust toolchain and libclang.

Getting a model

The extension contains no models. Download one from the models/ directory and pass its path — see Pre-trained Models. The model identifies its own kind, so hasPos() reports what was loaded and no flag is needed.

Segmentation

use Litsea\Segmenter;

$seg = Segmenter::open('japanese', 'models/japanese.model');

$seg->segment('これはテストです。');
// ['これ', 'は', 'テスト', 'です', '。']

The language name and its ISO 639-1 code are interchangeable ('ja', 'japanese').

For space-delimited languages the whitespace is returned as its own token, so the tokens always reconstruct the input:

Segmenter::open('ko', 'models/korean.model')->segment('안녕하세요 반갑습니다');
// ['안녕하세요', ' ', '반갑습니다']

POS tagging

$seg = Segmenter::open('japanese', 'models/japanese_pos.model');

foreach ($seg->segmentWithPos('これはテストです。') as $token) {
    printf("%s\t%s\t[%d:%d]\n", $token->surface, $token->pos, $token->start, $token->end);
}
// これ    PRON    [0:6]
// は      ADP     [6:9]
// テスト  NOUN    [9:18]
// です    AUX     [18:24]
// 。      PUNCT   [24:27]

start and end are byte offsets, and PHP strings are byte strings, so substr($text, $token->start, $token->end - $token->start) returns the surface directly — no encoding-aware slicing needed, unlike JavaScript.

API

ext-php-rs renames methods and properties to camelCase, so the PHP surface reads as segmentWithPos(), hasPos(), and $metrics->numInstances.

Cancellation is pre-call only

This is the one place where PHP differs from the other bindings, and it is a property of the host rather than a gap here.

The Python binding releases the GIL and the Node.js binding runs training on a worker thread, so both can stop a run that is already going. A PHP request is single-threaded, and pcntl signal handlers cannot interrupt a blocking native call, so no PHP code runs while train() executes. A CancelToken therefore only takes effect if it was cancelled before the call:

$cancel = new Litsea\CancelToken();
$cancel->cancel();

$metrics = (new Litsea\Trainer(0.01, 100000, 'features.txt'))->train('japanese.model', $cancel);

Cancelling is not an error: training stops at its next check point, still writes the partially trained model, and returns its metrics.

Because everything blocks, run training from the CLI SAPI rather than a web request.

Errors

Every exception derives from Litsea\LitseaException, so one catch handles them all — the same hierarchy the Python binding exposes.

Development

make test-litsea-php    # cargo test + build the extension + PHPUnit
make lint-litsea-php    # clippy
make build-litsea-php   # release build

The parity tests build the litsea CLI and compare the binding’s output against it.

Ruby

litsea-ruby exposes Litsea to Ruby 3.1+ through magnus and rb-sys. It is published to RubyGems as litsea.

Installation

gem install litsea

The gem is source-only and compiles the extension on install, so a Rust toolchain is required.

Getting a model

The gem contains no models. Download one from the models/ directory and pass its path — see Pre-trained Models. The model identifies its own kind, so has_pos? reports what was loaded and no flag is needed.

Segmentation

require "litsea"

seg = Litsea::Segmenter.open(:japanese, "models/japanese.model")

seg.segment("これはテストです。")
# => ["これ", "は", "テスト", "です", "。"]

The language accepts a Symbol or a String, and the ISO 639-1 code works too (:ja, "japanese").

For space-delimited languages the whitespace is returned as its own token, so the tokens always reconstruct the input:

Litsea::Segmenter.open(:korean, "models/korean.model").segment("안녕하세요 반갑습니다")
# => ["안녕하세요", " ", "반갑습니다"]

POS tagging

seg = Litsea::Segmenter.open(:japanese, "models/japanese_pos.model")

seg.segment_with_pos("これはテストです。").each do |token|
  puts "#{token.surface}\t#{token.pos}\t[#{token.start}..#{token.end}]"
end
# これ    PRON    [0..6]
# は      ADP     [6..9]
# テスト  NOUN    [9..18]
# です    AUX     [18..24]
# 。      PUNCT   [24..27]

start and end are byte offsets. Ruby’s String#[] counts characters, so slice with byteslice:

text.byteslice(token.start, token.end - token.start)   # == token.surface

API

Releasing the GVL

Long-running work — loading a model, extracting features, training — runs with the Global VM Lock released, so other Ruby threads keep going. That is what makes cancellation useful:

cancel = Litsea::CancelToken.new
Thread.new { sleep 60; cancel.cancel }

metrics = Litsea::Trainer.new(0.01, 100_000, "features.txt").train("japanese.model", cancel: cancel)

Cancelling is not an error: training stops at its next check point, still writes the partially trained model, and returns its metrics. The binding never installs a signal handler.

Neither magnus nor rb-sys wraps rb_thread_call_without_gvl — magnus lists it among the C functions it does not bind, and it is declared in a header outside rb-sys’s generated bindings — so the binding declares it itself in src/gvl.rs, behind an extern "C" trampoline that catches panics so none can unwind across the C frame. Two tests hold that claim honest: one asserts another Ruby thread keeps ticking during training, and one asserts a cancel from another thread lands inside the training window. Removing the GVL release turns both red.

Segmentation of a single sentence keeps the GVL: it is short enough that releasing it would cost more than the work.

A TwoStageTrainer can only be used once — training collapses stage 1 into an AdaBoost model, which consumes it. available? reports the state, and a second train raises.

Errors

Every error derives from Litsea::Error, so one rescue handles them all — the same hierarchy the Python and PHP bindings expose.

Development

make test-litsea-ruby    # cargo test + rake compile + rake test
make lint-litsea-ruby    # clippy + rubocop
make build-litsea-ruby   # release build

bundle must be usable with the active Ruby; a version manager’s shim can exist while the selected interpreter has no bundler, so the Makefile checks and says so. The parity tests build the litsea CLI and compare the binding’s output against it.

WebAssembly

litsea-wasm runs Litsea in browsers, Deno, and bundlers through wasm-bindgen. It is published to npm as litsea-wasm.

For Node.js, use the native binding litsea instead: it is faster and can train models.

Installation

npm install litsea-wasm

The module is 178 KB (82 KB gzipped), measured on a release wasm-pack build. Models are downloaded separately.

Usage

import init, { Segmenter } from 'litsea-wasm'

await init()

const bytes = new Uint8Array(await (await fetch('/models/japanese.model')).arrayBuffer())
const seg = Segmenter.fromBytes('japanese', bytes)

seg.segment('これはテストです。')
// [ 'これ', 'は', 'テスト', 'です', '。' ]

seg.free()

The language name and its ISO 639-1 code are interchangeable. The model file identifies its own kind, so hasPos reports what was loaded.

POS tagging

seg.segmentWithPos('これはテストです。')
// [ Token { surface: 'これ', pos: 'PRON', start: 0, end: 6 }, ... ]

start and end are byte offsets into the UTF-8 encoding, and JavaScript string indices are UTF-16 code units, so slice with TextEncoder / TextDecoder:

const bytes = new TextEncoder().encode(text)
new TextDecoder().decode(bytes.subarray(token.start, token.end))   // === token.surface

What the host removes

This is the most constrained of the five bindings, and each gap was measured rather than assumed.

Why there is no training

litsea gained filesystem-free extract/train APIs in #218, and they compile for wasm32-unknown-unknown, so this binding could expose training. It does not, for two reasons (#221):

• A browser is not where training belongs. A tab would hold the corpus, the features extracted from it (much larger than the corpus), and the model at once. Deciding the API shape would have required measuring that first, and the measurement could well have concluded it is impractical at any useful corpus size.
• The reference implementation does not either. lindera-python, lindera-nodejs, lindera-php, and lindera-ruby all ship a trainer; lindera-wasm ships none. Litsea’s bindings match that shape.

Train with the CLI or one of the native bindings, and load the resulting model here.

Memory

Segmenter holds the compiled model, which is several megabytes for a POS model, and WebAssembly objects are not garbage collected. Call free() when one is no longer needed.

Caching models

Models are 84 KB – 8 MB and cross the network once per visitor — the one cost this binding has that the native ones do not. The package ships an optional helper:

import { fetchModel, clearModelCache } from 'litsea-wasm/js/cache.js'

const bytes = await fetchModel('/models/japanese.model')

It stores fetched models in Cache Storage keyed by URL, and falls back to a plain fetch when Cache Storage is unavailable (an insecure context), so callers do not branch. It is plain JavaScript outside the wasm module, so a page that does not use it pays nothing.

Errors

Every error carries a code, matching the Node.js binding so the two JavaScript bindings agree.

Development

make test-litsea-wasm    # cargo test + headless browser tests
make lint-litsea-wasm    # clippy on wasm32
make build-litsea-wasm   # wasm-pack build --target web

The browser tests cannot spawn a process, so tests/generate_fixtures.sh runs the litsea CLI first and writes its output next to the test; the test asserts equality against it. The reference implementation still decides what is correct, as in every other binding.

Override the browser with make test-litsea-wasm WASM_BROWSER=chrome. If every test passes and the run then fails with PermissionDenied, the geckodriver on PATH is snap-confined — the tests themselves ran.

Training Guide

This guide walks you through training custom word segmentation and POS tagging models with Litsea.

Both workflows use Universal Dependencies (UD) Treebanks as the data source.

Word Segmentation (AdaBoost)

1. Prepare a corpus from a UD Treebank: conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp) && bash scripts/corpus_udtreebank.sh "$conllu_file" corpus.txt
2. Extract features from the corpus
3. Train a model using AdaBoost

POS Tagging (Two-Stage)

1. Prepare a POS corpus from a UD Treebank: conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp) && bash scripts/corpus_udtreebank.sh -p "$conllu_file" pos_corpus.txt
2. Extract two-stage features: litsea extract --pos -l japanese pos_corpus.txt features
3. Train a two-stage POS model: litsea train --pos --num-epochs 50 features model.model

Per-Language Differences

The pipeline (prepare → extract → train) and the scripts are shared by all four languages. Only two things are language-specific:

1. The -l flag on extract selects the language’s character-type classification (Japanese 8 types, Chinese 9, Korean 10, English 7; Korean and English use no WC features — see the language support overview). Models are therefore language-specific.
2. Korean and English use the space-preserving TSV corpus format. Both are written with spaces between words, and those spaces are the strongest boundary signal, so their corpora keep them as tokens (corpus_udtreebank.sh -s + litsea extract --format tsv). Japanese and Chinese are written without spaces, so they use the plain space-separated format.

# Japanese / Chinese: space-separated corpus
bash scripts/corpus_udtreebank.sh "$conllu_file" corpus.txt
litsea extract -l japanese corpus.txt features.txt

# Korean / English: space-preserving TSV corpus
bash scripts/corpus_udtreebank.sh -s "$conllu_file" corpus.tsv
litsea extract -l korean --format tsv corpus.tsv features.txt

The train step’s command shape is the same for all four languages, but the actual hyperparameters differ. -t 0.0001 -i 20000 (see Training Models) is a good starting point when training a plain AdaBoost model from scratch with litsea train, but it is not what the bundled japanese/chinese/korean/english models use – those go through a different procedure with per-language epoch counts and pruning. See Training Procedure for the actual recipe.

Additional Topics

• Evaluating Models – assess model quality
• Retraining Models – fine-tune existing models

Preparing a Corpus

A good training corpus is essential for model accuracy. This guide explains how to prepare one using Universal Dependencies (UD) Treebanks.

Data Source: UD Treebanks

Litsea uses UD Treebanks as the data source for both word segmentation and POS tagging. UD Treebanks provide high-quality, manually annotated data in CoNLL-U format for many languages.

Available Treebanks

Step 1: Download a UD Treebank

Use scripts/download_udtreebank.sh to download a UD Treebank. It prints the path to the training CoNLL-U file to stdout:

conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp)

Supported languages: ja (Japanese, default), ko (Korean), zh (Chinese), en (English). Use -o to specify the output directory (default: current directory).

Corpus for Word Segmentation

For word segmentation (AdaBoost), the corpus must be a plain text file with:

• One sentence per line
• Words separated by spaces

太郎 は 走っ た 。
Litsea は コンパクト な 単語 分割 ソフトウェア です 。

Convert CoNLL-U to Word Segmentation Corpus

Use scripts/corpus_udtreebank.sh to convert a CoNLL-U file to corpus format:

conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp)
bash scripts/corpus_udtreebank.sh "$conllu_file" corpus.txt

This converts the CoNLL-U data into space-separated words (one sentence per line).

Space-Preserving TSV Corpus (Korean, English)

The space-separated format above discards the original spacing of the sentence: the words are later concatenated without spaces for training. For Korean and English this loses the strongest boundary signal (Korean’s inter-eojeol space, English’s inter-word space), so use the -s flag instead, which emits a tab-separated corpus in which every original space (reconstructed from the treebank’s SpaceAfter annotations) is kept as its own token:

conllu_file=$(bash scripts/download_udtreebank.sh -l ko -o /tmp)
bash scripts/corpus_udtreebank.sh -s "$conllu_file" ko_corpus.tsv

conllu_file=$(bash scripts/download_udtreebank.sh -l en -o /tmp)
bash scripts/corpus_udtreebank.sh -s "$conllu_file" en_corpus.tsv

For English specifically, -s also handles multiword tokens (contractions like don't, represented in CoNLL-U as a range line covering two word lines): the range’s member words are joined with no space token between them, and the range’s own SpaceAfter annotation is applied after the last member word. See English for the details.

Extract features from a TSV corpus with litsea extract --format tsv. The -s combines with -p to emit a space-preserving TSV of word/POS tokens — the two-stage POS training corpus for space-delimited languages (issue #198), consumed by litsea extract --pos --format tsv.

Corpus for POS Tagging

For POS tagging (Averaged Perceptron), each word must be annotated with its POS tag.

POS Corpus Format

Each line represents one sentence, with words annotated as word/POS pairs separated by spaces:

これ/PRON は/ADP テスト/NOUN です/AUX 。/PUNCT
Litsea/PROPN は/ADP 単語/NOUN 分割/NOUN ソフトウェア/NOUN です/AUX 。/PUNCT

The POS tags follow the Universal POS (UPOS) tagset with 17 categories: ADJ, ADP, ADV, AUX, CCONJ, DET, INTJ, NOUN, NUM, PART, PRON, PROPN, PUNCT, SCONJ, SYM, VERB, X.

Convert CoNLL-U to POS Corpus

Use scripts/corpus_udtreebank.sh with the -p flag to produce a POS corpus:

conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp)
bash scripts/corpus_udtreebank.sh -p "$conllu_file" pos_corpus.txt

Multi-word tokens and empty nodes are automatically handled during conversion.

Automated Corpus Preparation

Litsea includes helper scripts in the scripts/ directory that automate the UD Treebank download and conversion:

• scripts/download_udtreebank.sh – Downloads a UD Treebank and prints the path to the training CoNLL-U file
• scripts/corpus_udtreebank.sh – Converts a CoNLL-U file to Litsea corpus format

# Download UD Treebank and get CoNLL-U file path
conllu_file=$(bash scripts/download_udtreebank.sh -l ja -o /tmp)

# Generate word segmentation corpus
bash scripts/corpus_udtreebank.sh "$conllu_file" corpus.txt

# Generate POS corpus
bash scripts/corpus_udtreebank.sh -p "$conllu_file" pos_corpus.txt

Supported languages for download_udtreebank.sh: ja (Japanese, default), ko (Korean), zh (Chinese), en (English).

Corpus from Wikipedia Dump

For larger-scale training, you can build a corpus from a full Wikipedia dump using scripts/corpus_wikidump.sh. This extracts plain text with wicket, filters for actual sentences, and tokenizes with lindera.

Usage

# Japanese (default)
bash scripts/corpus_wikidump.sh jawiki-latest-pages-articles.xml.bz2 corpus_ja.txt

# Korean
bash scripts/corpus_wikidump.sh -l ko kowiki-latest-pages-articles.xml.bz2 corpus_ko.txt

# Chinese
bash scripts/corpus_wikidump.sh -l zh zhwiki-latest-pages-articles.xml.bz2 corpus_zh.txt

Options

Sentence Filtering

The script applies two filters to keep only well-formed sentences:

1. Sentence-ending punctuation – Lines must end with 。, ., !, or ?. This excludes section headers (e.g., “参考文献”), list items, and metadata.
2. Minimum length – Lines must be at least 20 characters. This excludes short fragments and isolated labels.

Tokenizer Dictionaries

Corpus Size Guidelines

The recommended corpus size depends on your use case:

The default max_lines=100000 in corpus_wikidump.sh targets the practical-to-high-quality range.

Corpus Quality Tips

• Diversity – Include text from various domains (news, literature, web, etc.)
• Size – See Corpus Size Guidelines above for recommended sizes
• Consistency – Ensure consistent tokenization throughout the corpus
• Deduplication – Remove duplicate sentences to avoid bias
• Cleaning – Remove HTML tags, special formatting, and non-text content

Extracting Features

After preparing a corpus, the next step is to extract features for model training.

Command

litsea extract -l <LANGUAGE> <CORPUS_FILE> <FEATURES_FILE>

Example

litsea extract -l japanese ./corpus.txt ./features.txt

Output:

Feature extraction completed successfully.

What Happens Internally

flowchart TD
    A["Read corpus line by line"] --> B["Split line into words"]
    B --> C["Build chars, types, and tags arrays"]
    C --> D["For each character position"]
    D --> E["Extract 38-42 features"]
    E --> F["Write label + features to file"]

1. The Extractor reads each line from the corpus
2. For each sentence, it creates a Segmenter context with character arrays, type arrays, and tag arrays
3. For each character position (except the first), it extracts features and writes them with the correct label. The two-stage stage-1 pipeline also emits the first position, so that the first word’s boundary decision is part of the training data

Feature File Format

Each line represents one character position. For the corpus line これ は テスト です 。, the first two lines are:

-1	BC1:OI	BC2:II	BC3:II	BP1:UU	BP2:UU	BQ1:UOI	BQ2:UII	BQ3:UOI	BQ4:UII	...
1	BC1:II	BC2:II	BC3:IK	BP1:UU	BP2:UO	BQ1:UII	BQ2:UII	BQ3:OII	BQ4:OII	...

• First column: label (1 = boundary, -1 = non-boundary)
• Remaining columns: features, written tab-separated in alphabetically sorted order (so each line starts with the BC1: feature)

Space-Preserving (TSV) Corpus Format

For a corpus that preserves the original spacing of the sentence – used to train the Korean and English models, since inter-word spaces are their strongest boundary signal (see Korean and English) – pass --format tsv instead of extracting from the default space-separated format:

litsea extract --format tsv -l korean ./ko_corpus.tsv ./ko_features.txt
litsea extract --format tsv --tag-free -l english ./en_corpus.tsv ./en_features.txt

The input is a tab-separated corpus (one sentence per line, tokens separated by tabs) in which a token may be a literal space character (" "). The output feature file format is identical to the default extract; only the corpus parsing differs. --format tsv also combines with --pos – see Two-Stage Feature Extraction below.

Two-Stage Feature Extraction

For two-stage POS tagging (issue #147), use --pos:

litsea extract --pos [--stage2-features full|balanced|fast] <CORPUS_FILE> <FEATURES_PREFIX>

Example

litsea extract --pos -l japanese ./pos_corpus.txt ./pos_features

--pos reads a POS-tagged corpus (word/POS word/POS ...) and, in a single pass over the corpus, writes three files from <FEATURES_PREFIX> instead of one:

litsea train --pos reads all three files back from the same prefix. Combine with --format tsv (issue #198) when the corpus is the space-preserving word/POS TSV that corpus_udtreebank.sh -p -s emits — the protocol the bundled Korean and English two-stage models are trained on.

Choosing --stage2-features

--stage2-features selects which stage-2 word-level templates (see Word-Level Feature Templates) get written to <FEATURES_PREFIX>.stage2, trading tagging quality for throughput:

See Choosing a stage-2 feature set for the measured quality/throughput comparison behind this default.

litsea extract --pos --stage2-features balanced -l chinese ./pos_corpus.txt ./pos_features

File Size Expectations

The features file will be significantly larger than the corpus because each character position generates 38-42 feature strings. For a 1 MB corpus, expect a features file of roughly 50-100 MB.

Training Models

Once features are extracted, train a model using AdaBoost.

Command

litsea train [OPTIONS] <FEATURES_FILE> <MODEL_FILE>

Basic Example

litsea train -t 0.0001 -i 20000 ./features.txt ./models/my_model.model

This is a generic example of plain AdaBoost training. The bundled japanese.model, chinese.model, korean.model, and english.model are not produced this way – see Training Procedure for the procedure actually used for those files.

Training Process

flowchart TD
    A["Initialize features<br/>(read feature names)"] --> B["Initialize instances<br/>(read labels + features)"]
    B --> C["AdaBoost training loop"]
    C --> D{"Converged or<br/>max iterations?"}
    D -->|No| C
    D -->|Yes| E["Save model"]
    E --> F["Output metrics"]

1. Initialize features – Reads the features file to build the feature index
2. Initialize instances – Reads again to load labeled instances and initial weights
3. Training loop – Iteratively selects the best feature, updates model weights, and reweights instances
4. Save model – Writes non-zero feature weights to the model file
5. Output metrics – Prints accuracy, precision, recall, and confusion matrix

Hyperparameters

Note: these are generic starting points for training a plain AdaBoost model from scratch. The bundled japanese.model, chinese.model, korean.model, and english.model are produced by a different procedure – a 2-class Averaged Perceptron collapsed to AdaBoost weights, with per-language epoch counts and pruning – see Training Procedure in Pre-trained Models for how those files are actually made.

Interpreting Output

Metrics are computed on the training data; with enough iterations the model can fit the training corpus almost perfectly, so evaluate on held-out text for a realistic quality estimate. The numbers below are a representative example of train’s output format, not the bundled japanese.model’s actual training log.

Result Metrics:
  Accuracy: 100.00% ( 1075868 / 1075869 )
  Precision: 100.00% ( 161283 / 161284 )
  Recall: 100.00% ( 161283 / 161283 )
  Confusion Matrix:
    True Positives: 161283
    False Positives: 1
    False Negatives: 0
    True Negatives: 914585

• Accuracy – Percentage of correct predictions (both boundaries and non-boundaries)
• Precision – Of predicted boundaries, what fraction is correct
• Recall – Of actual boundaries, what fraction was found
• True Positives – Correctly predicted boundaries
• False Positives – Predicted boundary where there is none
• False Negatives – Missed actual boundaries
• True Negatives – Correctly predicted non-boundaries

Graceful Interruption

Press Ctrl+C once during training to stop and save the model at its current state. Press Ctrl+C twice to exit immediately without saving.

Generic Perceptron Training

For the bundled segmentation models’ collapse recipe (see Training Procedure), use the --perceptron flag. It trains a multiclass Averaged Perceptron over opaque string labels from a label\tfeature\t... features file.

Perceptron Training Command

litsea train --perceptron --num-epochs 50 <FEATURES_FILE> <MODEL_FILE>

Perceptron Training Output

Result Metrics (Perceptron):
  Accuracy: 98.23% ( 277213 )
  Macro Precision: 96.82%
  Macro Recall: 93.30%

• Accuracy – Percentage of correct predictions across all classes
• Macro Precision – Average precision across all classes
• Macro Recall – Average recall across all classes

Press Ctrl+C once during perceptron training to stop and save the model at its current state. Press Ctrl+C twice to exit immediately without saving.

Two-Stage Model Training

For POS tagging, use the --pos flag. It trains a two-stage model (issue #147): a binary boundary classifier (stage 1) plus a word-level tagger (stage 2), assembled with a candidate-tag lexicon into a single litsea-two-stage v1 file. See Two-Stage Tagging for the architecture and the measured quality/speed figures.

Two-Stage Training Command

litsea extract --pos <CORPUS_FILE> <FEATURES_PREFIX>
litsea train --pos --num-epochs 50 <FEATURES_PREFIX> <MODEL_FILE>

extract --pos reads a word/POS corpus and writes three files from FEATURES_PREFIX; train --pos reads them back from the same prefix.

Two-Stage Training Example

litsea extract --pos -l japanese ./pos_corpus.txt ./pos_features
litsea train --pos --num-epochs 50 ./pos_features ./models/japanese_pos.model

Two-Stage Hyperparameters

Two-Stage Training Output

Result Metrics (Two-Stage):
  Stage 1 (boundary) Accuracy: 99.86% ( 277213 )
  Stage 1 Macro Precision: 99.85%
  Stage 1 Macro Recall: 99.86%
  Stage 2 (tagging) Accuracy: 99.09% ( 168333 )
  Stage 2 Macro Precision: 98.96%
  Stage 2 Macro Recall: 98.77%

As with the other modes, these are in-sample metrics; evaluate on held-out text with litsea evaluate --pos for a realistic quality estimate.

Two-Stage Graceful Interruption

Press Ctrl+C once during two-stage training to stop and save the model at its current state. Press Ctrl+C twice to exit immediately without saving.

Evaluating Models

Understanding model quality is essential for producing good segmentation results.

Metrics

The train command outputs three key metrics after training. These are in-sample metrics: they are measured on the training data itself, so they overestimate how the model performs on unseen text. For a realistic picture, always evaluate on a held-out corpus that was not used for training (see the benchmarks below).

Accuracy

Accuracy = (TP + TN) / Total Instances

The percentage of all character positions that were correctly classified (both boundaries and non-boundaries). This is the broadest measure of model quality.

Precision

Precision = TP / (TP + FP)

Of the boundaries the model predicted, what fraction was correct. High precision means few false boundaries (over-segmentation).

Recall

Recall = TP / (TP + FN)

Of the actual boundaries, what fraction did the model find. High recall means few missed boundaries (under-segmentation).

Confusion Matrix

Pre-trained Model Benchmarks

The bundled japanese.model, chinese.model, korean.model, and english.model are trained with a binary-perceptron-collapse procedure, not plain AdaBoost -t/-i training – see Training Procedure for the exact recipe. All are evaluated on the held-out test split of their training treebank. Word F1 scores exact word matches; Boundary F1 scores individual boundary decisions.

Korean and English are trained and evaluated on text that preserves the original spaces (space-preserving TSV corpus; space tokens are excluded from the F1 computation). Since spaces mark most word boundaries in these two languages, this makes the task easier than for Japanese and Chinese, which are written without spaces — the scores are not directly comparable across languages. Korean’s near-deterministic 99.91% and English’s lower 98.31% both come from this same space-preserving protocol; the remaining gap is residual ambiguity English keeps even with spaces present (contractions, hyphenated compounds, abbreviations).

Reproducing the Benchmarks

Every figure in the table above is reproducible with one command using the bundled gold data (resources/eval/, converted from the UD GSD test splits — held-out for the bundled models, which are trained on the train splits):

litsea evaluate -l japanese models/japanese.model resources/eval/japanese_gsd_test.txt
litsea evaluate -l korean --format tsv models/korean.model resources/eval/korean_gsd_test.tsv
litsea evaluate -l chinese models/chinese.model resources/eval/chinese_gsd_test.txt
litsea evaluate -l english --format tsv models/english.model resources/eval/english_ewt_test.tsv

See evaluate for the command reference. POS models are evaluated with --pos; their held-out figures are listed in Pre-trained Models.

Which gold file to use depends on how the model was trained, and the two groups differ:

# Japanese / Chinese: real text has no spaces, so the space-separated
# `word/POS` gold is also the real-world protocol.
litsea evaluate --pos -l japanese models/japanese_pos.model resources/eval/japanese_gsd_test_pos.txt

# Korean / English: trained on the space-preserving corpus (issue #198),
# so evaluate against the space-preserving POS gold with --format tsv.
litsea evaluate --pos --format tsv -l korean models/korean_pos.model resources/eval/korean_gsd_test_pos_spaced.tsv
litsea evaluate --pos --format tsv -l english models/english_pos.model resources/eval/english_ewt_test_pos_spaced.tsv

The *_test_pos.txt files (no space tokens) are still present for Korean and English. They measure the unspaced protocol those models were trained on before #198, so they are useful only for reproducing the older published numbers – for current quality, use the *_pos_spaced.tsv gold above, which matches both how the models train and what segment --pos receives in practice.

Improving Model Quality

If accuracy is unsatisfactory, consider:

1. More training data – A larger and more diverse corpus
2. Lower threshold – Try -t 0.0001 to allow more boosting iterations
3. More iterations – Try -i 20000 or higher. AdaBoost selects one weak learner (feature) per iteration, so the number of iterations caps how many features the model can use; the CLI default (-i 100) produces very small models with much lower held-out accuracy
4. Better corpus quality – Ensure consistent tokenization and clean text
5. Retraining – Start from an existing model and train with additional data (see Retraining Models)

The threshold/iteration tuning above applies to plain AdaBoost training (litsea train without --perceptron/--pos). The bundled models’ own +5-13pt held-out quality gains over plain AdaBoost did not come from tuning -t/-i – they came from training a 2-class Averaged Perceptron and collapsing it losslessly to AdaBoost weights instead. If you are chasing bundled-model-level quality rather than incremental gains, see Training Procedure for that recipe.

Retraining Models

You can improve an existing model by resuming training with new data.

Command

litsea train -t 0.0001 -i 20000 -m <EXISTING_MODEL> <NEW_FEATURES_FILE> <OUTPUT_MODEL>

Example

# Extract features from new corpus
litsea extract -l japanese ./new_corpus.txt ./new_features.txt

# Retrain from existing model
litsea train -t 0.0001 -i 20000 \
    -m ./models/my_model.model \
    ./new_features.txt \
    ./models/my_model_v2.model

How It Works

flowchart LR
    A["Existing model<br/>(weights)"] --> C["Trainer"]
    B["New features"] --> C
    C --> D["Retrained model<br/>(updated weights)"]

1. The trainer initializes features and instances from the new features file
2. It loads the existing model weights via -m
3. Training continues with the loaded weights as a starting point
4. The new model inherits all learned patterns and refines them with new data

Use Cases

• Domain adaptation – Fine-tune a general model on domain-specific text (e.g., medical, legal)
• Incremental improvement – Add more training data without retraining from scratch
• Error correction – Train on examples where the current model makes mistakes

Notes

• The output model can be the same path as the input model (overwrites)
• The -m flag accepts file paths, file://, http://, and https:// URIs
• Retraining starts from the existing weights, so fewer iterations may be needed
• -m is only available for plain AdaBoost training. train --pos does not support -m/--load-model-uri – incremental training of a two-stage model is not supported, so if you need to update one you must retrain it from scratch with train --pos
• The bundled japanese.model, chinese.model, korean.model, and english.model are not produced with this plain -m recipe – they go through the perceptron-collapse procedure described in Training Procedure. Running further incremental AdaBoost training on top of one of them with -m would mix the two approaches; to update one of the bundled models, retrain it from scratch with that procedure instead

Model File Format

Litsea models are stored as simple plain-text files.

Format Specification

<feature_name>\t<weight>
<feature_name>\t<weight>
...
<bias>

• Each line (except the last) contains a feature name and its weight, separated by a tab character
• Zero-weight features are omitted to keep the file compact
• The last line contains the bias term as a single number

Example

BC1:IK	0.3456
BC2:KI	-0.1234
UW4:は	0.5678
UC4:I	0.2345
...
-0.0891

Bias Reconstruction

When loading a model, the bias is reconstructed using:

bias_bucket_weight = -bias_value * 2 - sum(feature_weights_before_the_bias_line)

Files written by save_model always place the bias line last, so this equals the sum of all feature weights. Legacy models (e.g. RWCP.model) place the bias line mid-file; their trailing weight lines are accepted and the bias bucket is computed from the weights preceding the bias line, matching the historical loader.

Validation

The loader rejects malformed files with an explicit error instead of loading them silently:

• an empty file
• a file without a bias line (the typical symptom of a truncated download or interrupted copy)
• more than one bias line
• duplicate feature lines
• non-finite weights or bias values (NaN, inf, -inf), which would otherwise poison every score comparison

The Averaged Perceptron model loader likewise validates its class-count header and rejects non-finite weights.

During prediction:

bias = -sum(all_model_weights) / 2.0    (cached; read once per sentence)
score = bias + sum(model[feature] for feature in input_attributes)

The on-disk format is string-keyed and unchanged, but the segmenter does not score against the strings directly: at load time each feature line is parsed and compiled into a packed u64 integer key for the hot loop (see Prediction Pipeline). Features that the segmenter’s language could never generate (for example type codes of another language) are ignored by that compilation – exactly as they could never match an input attribute before – while the bias is always computed over every weight in the file.

Two-Stage Model Format (litsea-two-stage v1)

A two-stage model bundles a stage-1 boundary classifier, a candidate-tag lexicon, and a stage-2 word-level tagger into a single plain-text file with a magic first line and marker-delimited sections in a fixed order:

litsea-two-stage v1
[params]
dominance\t0.99
[stage1]
<AdaBoost model format: "feature\tweight" lines + one bias line>
[lexicon]
<surface>\t<TAG>:<count>[,<TAG>:<count>...]
[stage2]
<Averaged Perceptron model format: class count, class names, weights>

• The [stage1] and [stage2] sections embed the existing formats described above verbatim and are parsed by the existing loaders.
• Each [lexicon] line maps a word surface to the UPOS tags observed for it in the training corpus with their occurrence counts, most frequent first (ties broken by tag name). Surfaces may contain any character except tab and newline and are not trimmed, so whitespace tokens stay representable.
• The [params] section is optional. Its only key, dominance, is the classifier-skip threshold in (0.5, 1.0]: a known surface whose most frequent tag covers at least this fraction of its training occurrences is tagged without invoking the stage-2 classifier. It defaults to 0.99 when the section is absent.
• Stage-2 class names must be valid UPOS tags; together with every weight and lexicon line containing a tab, this guarantees no content line can collide with a section marker.

The format is purely additive: the magic line is neither a valid AdaBoost weight/bias line nor a perceptron class count, so the existing loaders reject two-stage files with an explicit error, and existing model files keep loading unchanged. A future format revision will use a different magic line (e.g. litsea-two-stage v2); the v1 loader rejects it as an unsupported version. The loader validates section order, the lexicon rules above, and the parameter range, and reports errors with the section name (e.g. [stage2] section: ...).

File Size

Model file sizes vary considerably by model type and language:

RWCP.model and JEITA_Genpaku_ChaSen_IPAdic.model are genuinely tiny (kilobyte-scale) and are the easiest to embed directly in applications or serve over HTTP with minimal overhead. The retrained japanese.model, chinese.model, korean.model, and english.model (see Pre-trained Models) trade some of that compactness for substantial quality gains: they are now ~86 KB-2.0 MB rather than kilobyte-scale, though still small compared to the multi-megabyte two-stage (*_pos.model) models, which are larger because they carry per-class and per-stage weights.

Compatibility

• Model files are encoding-agnostic (feature names are stored as-is)
• The format is deterministic for the usual training workflow: save_model writes features in the learner’s feature order, which is sorted (via BTreeMap) for learners initialized from a features file or loaded from disk. A learner populated only via add_instance() writes features in insertion order instead
• Models are forward-compatible – new features in the input that are not in the model are simply ignored during prediction

Remote Model Loading

Litsea supports loading models from HTTP/HTTPS URLs in addition to local files.

Supported URI Schemes

CLI Usage

echo "テスト" | litsea segment -l japanese https://example.com/japanese.model

Library Usage

#![allow(unused)]
fn main() {
let mut learner = AdaBoost::new(0.01, 100);

// Local file
learner.load_model_from_path(Path::new("./models/japanese.model"))?; // local, synchronous

// HTTP URL
learner.load_model("https://example.com/models/japanese.model").await?;
}

Enabling the Feature

Since 0.6.0 the remote_model feature is opt-in (the library default is local loading only, keeping the dependency tree compact). The CLI enables it, so litsea segment https://... keeps working out of the box; library users need:

litsea = { version = "0.13.0", features = ["remote_model"] }

Implementation Details

• HTTP client: reqwest with rustls (no OpenSSL dependency)
• Custom User-Agent: Litsea/<version>
• The load_model method is async because HTTP loading requires an async runtime
• For the CLI, tokio provides the async runtime

Limits and Failure Handling

• Connect timeout: 10 seconds; overall request timeout: 60 seconds – a stalled server can no longer block model loading indefinitely
• Maximum model size: 256 MiB. A larger advertised Content-Length is rejected before the body is read, and an oversized body is rejected after
• Incomplete downloads: when the server sends a Content-Length, a shorter received body is reported as an incomplete download
• Non-2xx responses are reported as download errors with the HTTP status
• The model parser additionally rejects truncated files (a model without its trailing bias line fails to load; see Model File Format)

WASM Considerations

The wasm32 target is CI-checked only without the remote_model feature (cargo check -p litsea --target wasm32-unknown-unknown --no-default-features):

• HTTP/HTTPS loading is not currently supported on wasm32 – the HTTP client configuration uses reqwest’s ClientBuilder::connect_timeout and timeout, which reqwest’s WASM client does not provide, so the remote_model feature does not build for this target
• Local file paths and the file:// scheme are not supported either – file system access is unavailable, so read_file_bytes returns an Unsupported error

Models must therefore be supplied by other means on wasm32, e.g. by calling load_model_from_reader with model bytes provided by the host environment.

Benchmarking

Litsea includes a Criterion benchmark suite for measuring performance.

Running Benchmarks

cargo bench --bench bench

Or via the Makefile:

make bench

Benchmark Suite

The benchmarks are defined in litsea/benches/bench.rs:

Models are loaded synchronously with load_model_from_path — no async runtime is involved in the benchmarks.

Corpus Throughput (external_corpus)

The external_corpus group reproduces the seven litsea benches of the external tokenizer-speed-bench harness in-repo (the english/english-two-stage cases below have no counterpart there yet), so throughput regressions can be caught with cargo bench alone:

cargo bench --bench bench -- external_corpus

The *-two-stage benches were added alongside the two-stage architecture (#147/#169); they are not part of the original seven tokenizer-speed-bench-mirroring benches above.

One iteration segments every line of the corpus (unfiltered, like the external harness), and the group sets Throughput::Elements to the corpus’s newline-free character count, so Criterion’s elem/s figures read directly as chars/sec.

The corpora live in resources/, byte-identical to the external harness:

Numbers are comparable to, but not identical with, the published tokenizer-speed-bench figures, for two methodological reasons: Criterion uses in-process warmup and sampling instead of 101 process-interleaved single passes, and cargo bench inherits litsea’s tuned release profile (thin LTO, single codegen unit) while the external bench crates build with the default release profile.

API Comparison (segment_into)

The segment_into group pairs the owned-output segment() API against the buffer-reusing segment_into() API (issue #184) on the same four segmentation corpora as external_corpus (same per-line workload, chars/sec via Throughput::Elements):