API Reference

Index

The primary entry point. Wraps the Laurus search engine.

class Index:
    def __init__(
        self,
        path: str | None = None,
        schema: Schema | None = None,
        wal_sync_policy: WalSyncPolicy | None = None,
        commit_policy: CommitPolicy | None = None,
    ) -> None: ...

Constructor

Creating vs. reopening a file-backed index (path given): if <path>/schema.toml does not yet exist, this call creates a new index and persists schema (or an empty schema, if omitted) to it. If <path>/schema.toml already exists, this call reopens the index — schema must be omitted, or a ValueError is raised (it would be ambiguous which schema should win). A ValueError is also raised if path contains an index in the layout used before this convention was introduced (segment files directly under path, no schema.toml).

Methods

search query argument

The query parameter accepts any of the following:

• A DSL string (e.g. "title:hello", "embedding:\"memory safety\"")
• A lexical query object (TermQuery, PhraseQuery, BooleanQuery, …)
• A vector query object (VectorQuery, VectorTextQuery)
• A SearchRequest for full control

The same value kinds are accepted as the elements of search_batch’s queries list — DSL strings, query objects, and SearchRequest instances may be mixed within a single batch.

Highlighting

search/search_batch’s highlight parameter (Issue #1134) requests highlighted fragments per field on each hit’s SearchResult.highlights. It accepts:

• A list of field names: highlight=["body"]
• A dict with a required "fields" key plus any of the optional HighlightConfig knobs: max_fragments, fragment_size, tag, css_class, require_field_match, max_analyzed_chars, return_entire_field_if_no_highlight — e.g. highlight={"fields": ["body"], "tag": "em", "max_fragments": 2}

results = index.search("body:rust", highlight=["body"])
print(results[0].highlights)  # {"body": ["<mark>Rust</mark> is a systems programming language."]}

Highlighting follows the query passed to search/search_batch (or SearchRequest.query/lexical_query — see below), and only stored: true text fields can be highlighted; a field that is absent, not stored, or not text is silently skipped. Omitting highlight leaves every result’s highlights empty. The same highlight= keyword is also accepted by SearchRequest.

WAL sync policy / durability

For a persistent index, every write is appended to a write-ahead log (WAL). By default the WAL is fsync-ed on every record, so each write is fully durable as soon as the call returns. The constructor accepts an optional wal_sync_policy to trade some durability for higher write throughput, and flush_wal() forces a durable barrier on demand.

class WalSyncPolicy:
    @staticmethod
    def per_record() -> WalSyncPolicy: ...
    @staticmethod
    def group(
        max_records: int | None = None,
        max_bytes: int | None = None,
        max_interval_ms: int | None = None,
    ) -> WalSyncPolicy: ...

group() parameters (all keyword-friendly, None keeps the default):

With group commit the WAL is flushed when either max_records or max_bytes is reached, and always at commit(). A crash can lose up to the last unsynced batch — the same trade-off as SQLite’s synchronous = NORMAL. Call flush_wal() to force everything written so far to disk without a full commit().

import laurus

# Opt into group commit with a 1-second periodic flush timer.
policy = laurus.WalSyncPolicy.group(max_records=4096, max_interval_ms=1000)
index = laurus.Index(path="./myindex", wal_sync_policy=policy)

for i in range(10_000):
    index.put_document(f"doc{i}", {"title": f"Document {i}"})

# Force a durable barrier without committing yet.
index.flush_wal()

index.commit()  # also flushes the WAL

Omit wal_sync_policy (or pass WalSyncPolicy.per_record()) to keep the default, fully durable behaviour.

Commit policy / auto-commit

A commit materializes buffered writes into the stores and makes all pending changes searchable. By default an Index never auto-commits, so the caller drives every commit() explicitly. The constructor accepts an optional commit_policy to instead let the engine auto-commit after a fixed number of applied documents, or at least once every fixed time interval.

class CommitPolicy:
    @staticmethod
    def manual() -> CommitPolicy: ...
    @staticmethod
    def every_docs(n: int) -> CommitPolicy: ...
    @staticmethod
    def interval_ms(ms: int) -> CommitPolicy: ...

every_docs(n) counts applied documents across both singular (put_document, add_document) and batch (put_documents, add_documents) ingest, and auto-commits after every n documents — including within a single batch. every_docs(0) is valid and disables auto-commit, making it equivalent to manual().

interval_ms(ms) is the time-based counterpart of every_docs(n): a background timer commits at least every ms milliseconds, so a trailing partial batch is committed even while ingestion is idle. This factory is native-only — on wasm there are no background threads, so the engine treats it as a no-op: the value still constructs, but no timed commit happens under WebAssembly.

commit_policy is orthogonal to wal_sync_policy: wal_sync_policy governs WAL fsync durability, whereas commit_policy governs when the stores materialize pending changes into searchable state. Setting one does not affect the other.

import laurus

# Auto-commit after every 100 applied documents.
index = laurus.Index(
    path="./myindex",
    commit_policy=laurus.CommitPolicy.every_docs(100),
)

for i in range(1_000):
    index.put_document(f"doc{i}", {"title": f"Document {i}"})
# The engine has already committed 10 times (once per 100 documents).

Omit commit_policy (or pass CommitPolicy.manual()) to keep the default, manual-commit behaviour.

Schema

Defines the fields and index types for an Index.

class Schema:
    def __init__(self) -> None: ...

Field methods

Vector quantization & rerank storage (HNSW fields):

• quantizer — "scalar_8bit" (default, 4× compression) or "product_quantization" for higher compression. Product quantization requires subvector_count (must divide dimension).
• rerank_storage — set to "f32" to write a full-precision *.hnsw.f32 sidecar enabling exact Stage-2 rerank; omit to keep the int8-only segment.
• pq_codebook_path — storage-relative file name of a shared PQ codebook (Issue #631), trained once via the laurus train pq-codebook CLI command. Only meaningful with quantizer="product_quantization"; commits then encode against the pre-trained codebook instead of re-training k-means per segment. Omit to keep per-segment training.

Every add_*_field method above raises ValueError when name starts with _ (other than _id), and adds nothing. A schema loaded with from_toml / from_toml_file keeps accepting such a field, so a persisted schema still loads; creating a new Index from it raises ValueError. See Field Naming.

Other methods

Dynamic field policy

Controls what happens when a document is ingested with field names that are not declared in the schema:

• "strict" — Reject the document.
• "dynamic" (default) — Infer a type for each undeclared field and add it to the schema. Warning: integer fields silently truncate incoming float values (3.14 → 3). Use "strict" if you need to reject such type mismatches.
• "ignore" — Silently drop the undeclared fields.

See Schema & Fields for the full behaviour matrix.

Embedder types

See Schema Format Reference → Embedders for the canonical description of each type.

"candle_colbert" is a token-level embedder: it produces one vector per token and serves only a multi-vector field (add_multi_vector_field). It also takes the optional keys "revision" (branch, tag, or commit of the model repository), "query_maxlen", and "doc_maxlen"; see Schema Format Reference → Embedders for their defaults.

schema = laurus.Schema()
schema.add_text_field("body")
schema.add_embedder("colbert", {"type": "candle_colbert", "model": "colbert-ir/colbertv2.0"})
schema.add_multi_vector_field("body_colbert", dimension=128, embedder="colbert")

Analyzer components

Used by add_analyzer(name, tokenizer, *, char_filters=None, token_filters=None) and by the [analyzers.<name>] TOML section. tokenizer is a single dict; char_filters/token_filters are lists of dicts, applied in list order.

See Schema Format Reference → Analyzers for the canonical description of each component.

Tokenizers (tokenizer, exactly one):

Char filters (char_filters, applied to raw text before tokenization):

Token filters (token_filters, applied to the token stream after tokenization):

schema = laurus.Schema()
schema.add_analyzer(
    "ja_ipadic",
    {"type": "lindera", "mode": "normal", "dict": "/var/lib/lindera/ipadic"},
    char_filters=[
        {"type": "unicode_normalization", "form": "nfkc"},
        {"type": "japanese_iteration_mark"},
    ],
    token_filters=[{"type": "lowercase"}],
)
schema.add_text_field("title", analyzer="ja_ipadic")

Distance metrics

Query classes

TermQuery

TermQuery(field: str, term: str)

Matches documents containing the exact term in the given field.

PhraseQuery

PhraseQuery(field: str, terms: list[str])

Matches documents containing the terms in order.

FuzzyQuery

FuzzyQuery(field: str, term: str, *, max_edits: int = 2)

Approximate match allowing up to max_edits edit-distance errors. max_edits is keyword-only.

WildcardQuery

WildcardQuery(field: str, pattern: str)

Pattern match. * matches any sequence of characters, ? matches any single character.

NumericRangeQuery

NumericRangeQuery(field: str, *, min: int | float | None = None, max: int | float | None = None)

Matches numeric values in the range [min, max]. Pass None (or omit) for an open bound. min and max are keyword-only. The numeric type (integer or float) is inferred from the Python type of min/max.

DateTimeRangeQuery

DateTimeRangeQuery(
    field: str, *,
    min: str | datetime.datetime | datetime.date | None = None,
    max: str | datetime.datetime | datetime.date | None = None,
)

Matches DateTime values in the range [min, max] (both bounds inclusive). Pass None (or omit) for an open bound; min and max are keyword-only. A bound is a str literal in any form the query DSL accepts — RFC 3339 ("2024-01-01T09:00:00+09:00", normalized to UTC), naive "YYYY-MM-DDTHH:MM:SS[.fff]" (UTC), or "YYYY-MM-DD" (midnight UTC) — or a datetime.datetime / datetime.date, converted via isoformat() (a naive datetime is UTC). A malformed bound raises ValueError at construction. Usable anywhere a query object is accepted (Index.search, BooleanQuery, SearchRequest), e.g. index.search(laurus.DateTimeRangeQuery("created_at", min="2024-01-01", max="2024-12-31")).

GeoDistanceQuery

GeoDistanceQuery.within_radius(
    field: str, lat: float, lon: float, distance_m: float,
)

Geo-distance (radius) search. Returns documents whose (lat, lon) coordinate is within distance_m metres of the given point.

GeoBoundingBoxQuery

GeoBoundingBoxQuery.within_bounding_box(
    field: str,
    min_lat: float, min_lon: float,
    max_lat: float, max_lon: float,
)

Geo bounding-box search. Returns documents whose (lat, lon) coordinate lies inside the axis-aligned [min_lat, max_lat] × [min_lon, max_lon] rectangle.

Geo3dDistanceQuery

Geo3dDistanceQuery.within_sphere(
    field: str, x: float, y: float, z: float, distance_m: float,
)

Sphere search over a 3D ECEF point field. Returns documents whose (x, y, z) coordinate is within distance_m metres of the centre. See Geo3d concepts for ECEF theory.

Geo3dBoundingBoxQuery

Geo3dBoundingBoxQuery.within_box(
    field: str,
    min_x: float, min_y: float, min_z: float,
    max_x: float, max_y: float, max_z: float,
)

Axis-aligned 3D bounding-box search. Returns documents whose ECEF point lies inside [min_x, max_x] × [min_y, max_y] × [min_z, max_z].

Geo3dNearestQuery

Geo3dNearestQuery.k_nearest(
    field: str,
    x: float, y: float, z: float,
    k: int,
    *,
    initial_radius_m: float | None = None,
    max_radius_m: float | None = None,
)

k-nearest-neighbour search over a 3D ECEF point field. Returns the k documents closest to (x, y, z). The optional initial_radius_m and max_radius_m tune the iterative-expansion search cone.

BooleanQuery

bq = BooleanQuery()
bq.must(query)
bq.should(query)
bq.must_not(query)

Compound boolean query. Construct with no arguments and add clauses one at a time via the must / should / must_not methods. Each method accepts any query object (including a nested BooleanQuery).

must clauses all have to match; must_not clauses must not match. should clauses contribute to scoring; at least one of them must match if there are no must clauses.

SpanQuery

# Single term
SpanQuery.term(field: str, term: str)

# Near: terms appearing within `slop` positions of each other
SpanQuery.near(field: str, terms: list[str], *, slop: int = 0, ordered: bool = True)

# Near with nested SpanQuery clauses
SpanQuery.near_spans(field: str, clauses: list[SpanQuery], *, slop: int = 0, ordered: bool = True)

# Containing: big span contains little span
SpanQuery.containing(field: str, big: SpanQuery, little: SpanQuery)

# Within: include span within exclude span at max distance
SpanQuery.within(field: str, include: SpanQuery, exclude: SpanQuery, distance: int)

Positional / proximity span queries. Construct via the static factory methods. near takes a list of term strings, while near_spans takes a list of SpanQuery objects for nested expressions. slop and ordered are keyword-only.

VectorQuery

VectorQuery(field: str, vector: list[float])

Approximate nearest-neighbor search using a pre-computed embedding vector.

VectorTextQuery

VectorTextQuery(field: str, text: str)

Converts text to an embedding at query time and runs vector search. Requires an embedder configured on the index.

SearchRequest

Full-featured search request for advanced control.

class SearchRequest:
    def __init__(
        self,
        *,
        query=None,
        lexical_query=None,
        vector_query=None,
        filter_query=None,
        fusion=None,
        limit: int = 10,
        offset: int = 0,
        highlight=None,
        rescore: LateInteractionRescore | None = None,
    ) -> None: ...

LateInteractionRescore

Late-interaction (ColBERT MaxSim) rescore of the top search results (Issue #1351). Pass it as rescore= to Index.search or SearchRequest.

class LateInteractionRescore:
    def __init__(
        self,
        field: str,
        query: str | list[list[float]],
        *,
        window_size: int | None = None,
    ) -> None: ...

    @property
    def window_size(self) -> int: ...

The window_size property returns the window the rescore uses (100 when omitted).

The top window_size first-stage results — lexical, vector, or hybrid — are reordered by their MaxSim against the field, highest first, and a rescored result’s score is its MaxSim. Window results without token vectors in the field follow in their first-stage order, and results beyond the window follow last, keeping their first-stage order and score. offset and limit are cut from this rescored ranking. See Vector Search → Late-Interaction Rescore for the ordering, similarity, and cost details.

Errors:

• TypeError at construction when query is neither a str nor a list of lists, or a token vector holds a bool or str.
• ValueError (Invalid argument: rescore: ...) from Index.search for every other invalid value, checked before the search runs: field is not a multi-vector field, the query does not hold 1 to 1,024 vectors of the field’s dimension with finite values, window_size is outside 1..=10,000, or a text query is blank or the field has no token-level embedder.

import laurus

schema = laurus.Schema()
schema.add_text_field("title")
schema.add_multi_vector_field("tokens", dimension=2, distance="dot_product")

index = laurus.Index(schema=schema)
index.put_document("a", {"title": "rust", "tokens": [[0.1, 0.0]]})
index.put_document("b", {"title": "rust language", "tokens": [[0.9, 0.2]]})
index.commit()

rescore = laurus.LateInteractionRescore("tokens", [[1.0, 0.0], [0.0, 1.0]], window_size=50)
results = index.search("title:rust", rescore=rescore)  # "b" (MaxSim 1.1) before "a" (0.1)

# The same rescore inside a SearchRequest (also usable in search_batch)
request = laurus.SearchRequest(query="title:rust", rescore=rescore, limit=5)
results = index.search(request)

With a "candle_colbert" embedder on the field, documents can give the field text and the query can be text too: laurus.LateInteractionRescore("body_colbert", "how do lifetimes work").

SearchResult

Returned by Index.search().

class SearchResult:
    id: str          # External document identifier
    score: float     # Relevance score
    document: dict | None  # Retrieved field values, or None if not stored
    highlights: dict[str, list[str]]  # Highlighted fragments per requested field

highlights maps each field named in highlight to its highlighted fragments (best first); a field that did not highlight is absent from the dict, and highlights is {} when highlight was not requested. See Highlighting.

With rescore, a rescored result’s score is its MaxSim, while results outside the rescore window keep their first-stage score. See LateInteractionRescore.

Fusion algorithms

RRF

RRF(k: float = 60.0)

Reciprocal Rank Fusion. Merges lexical and vector result lists by rank position. k is a smoothing constant; higher values reduce the influence of top-ranked results.

WeightedSum

WeightedSum(lexical_weight: float = 0.5, vector_weight: float = 0.5)

Normalises both score lists independently, then combines them as lexical_weight * lexical_score + vector_weight * vector_score.

Text analysis

SynonymDictionary

class SynonymDictionary:
    def __init__(self) -> None: ...
    def add_synonym_group(self, synonyms: list[str]) -> None: ...

WhitespaceTokenizer

class WhitespaceTokenizer:
    def __init__(self) -> None: ...
    def tokenize(self, text: str) -> list[Token]: ...

SynonymGraphFilter

class SynonymGraphFilter:
    def __init__(
        self,
        dictionary: SynonymDictionary,
        keep_original: bool = True,
        boost: float = 1.0,
    ) -> None: ...
    def apply(self, tokens: list[Token]) -> list[Token]: ...

Token

class Token:
    text: str
    position: int
    start_offset: int
    end_offset: int
    boost: float
    stopped: bool
    position_increment: int
    position_length: int
    token_type: str | None

start_offset and end_offset are UTF-8 byte offsets into the original text. For non-ASCII text they are not str indices; slice the encoded text instead: text.encode()[tok.start_offset:tok.end_offset].decode().

token_type is one of "alphanum", "num", "cjk", "katakana", "hiragana", "hangul", "punctuation", "whitespace", "synonym", "email", "url" and "other", or None. SynonymGraphFilter.apply keeps each token’s offsets and type, and gives each synonym it inserts the type "synonym" and the offsets of the words it replaces.

Field value types

Python values are automatically converted to Laurus DataValue types: