Laurus

A fast, featureful hybrid search library for Rust.

Laurus is a pure-Rust library that combines lexical search (keyword matching via inverted index) and vector search (semantic similarity via embeddings) into a single, unified engine. It is designed to be embedded directly into your Rust application — no external server required.

Key Features

How It Works

graph LR
    subgraph Your Application
        D["Document"]
        Q["Query"]
    end

    subgraph Laurus Engine
        SCH["Schema"]
        AN["Analyzer"]
        EM["Embedder"]
        LI["Lexical Index\n(Inverted Index)"]
        VI["Vector Index\n(HNSW / Flat / IVF)"]
        FU["Fusion\n(RRF / WeightedSum)"]
    end

    D --> SCH
    SCH --> AN --> LI
    SCH --> EM --> VI
    Q --> LI --> FU
    Q --> VI --> FU
    FU --> R["Ranked Results"]

1. Define a Schema — declare your fields and their types (text, integer, vector, etc.)
2. Build an Engine — attach an analyzer for text and an embedder for vectors
3. Index Documents — the engine routes each field to the correct index automatically
4. Search — run lexical, vector, or hybrid queries and get ranked results

Document Map

Quick Example

use std::sync::Arc;
use laurus::{Document, Engine, Schema, SearchRequestBuilder, Result};
use laurus::lexical::{TextOption, TermQuery};
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> Result<()> {
    // 1. Storage
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    // 2. Schema
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_default_field("body")
        .build();

    // 3. Engine
    let engine = Engine::builder(storage, schema).build().await?;

    // 4. Index a document
    let doc = Document::builder()
        .add_text("title", "Hello Laurus")
        .add_text("body", "A fast search library for Rust")
        .build();
    engine.add_document("doc-1", doc).await?;
    engine.commit().await?;

    // 5. Search
    let request = SearchRequestBuilder::new()
        .lexical_search_request(
            laurus::LexicalSearchRequest::new(
                Box::new(TermQuery::new("body", "rust"))
            )
        )
        .limit(10)
        .build();
    let results = engine.search(request).await?;

    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }
    Ok(())
}

License

Laurus is licensed under the MIT License.

Architecture

This page explains how Laurus is structured internally. Understanding the architecture will help you make better decisions about schema design, analyzer selection, and search strategies.

Project Structure

Laurus is organized as a Cargo workspace with five crates:

graph TB
    CLI["laurus-cli\n(Binary Crate)\nCLI + REPL"]
    SRV["laurus-server\n(Library + Binary)\ngRPC Server + HTTP Gateway"]
    MCP["laurus-mcp\n(Library + Binary)\nMCP Server"]
    PY["laurus-python\n(cdylib)\nPython Bindings"]
    LIB["laurus\n(Library Crate)\nCore Search Engine"]

    CLI --> LIB
    CLI --> SRV
    CLI --> MCP
    SRV --> LIB
    MCP --> SRV
    MCP --> LIB
    PY --> LIB

For details on each crate, see:

• Library Overview
• CLI Overview
• Server Overview
• MCP Server Overview
• Python Bindings Overview
• Node.js Bindings Overview
• WASM Bindings Overview
• Ruby Bindings Overview
• PHP Bindings Overview

High-Level Overview

Laurus is organized around a single Engine that coordinates four internal components:

graph TB
    subgraph Engine
        SCH["Schema"]
        LS["LexicalStore\n(Inverted Index)"]
        VS["VectorStore\n(HNSW / Flat / IVF)"]
        DL["DocumentLog\n(WAL + Document Storage)"]
    end

    Storage["Storage (trait)\nMemory / File / File+Mmap"]

    LS --- Storage
    VS --- Storage
    DL --- Storage

All three stores share a single Storage backend, isolated by key prefixes (lexical/, vector/, documents/).

Engine Lifecycle

Building an Engine

The EngineBuilder assembles the engine from its parts:

#![allow(unused)]
fn main() {
let engine = Engine::builder(storage, schema)
    .analyzer(analyzer)      // optional: for text fields
    .embedder(embedder)      // optional: for vector fields
    .build()
    .await?;
}

sequenceDiagram
    participant User
    participant EngineBuilder
    participant Engine

    User->>EngineBuilder: new(storage, schema)
    User->>EngineBuilder: .analyzer(analyzer)
    User->>EngineBuilder: .embedder(embedder)
    User->>EngineBuilder: .build().await
    EngineBuilder->>EngineBuilder: split_schema()
    Note over EngineBuilder: Separate fields into\nLexicalIndexConfig\n+ VectorIndexConfig
    EngineBuilder->>Engine: Create LexicalStore
    EngineBuilder->>Engine: Create VectorStore
    EngineBuilder->>Engine: Create DocumentLog
    EngineBuilder->>Engine: Recover from WAL
    EngineBuilder-->>User: Engine ready

During build(), the engine:

1. Splits the schema — lexical fields go to LexicalIndexConfig, vector fields go to VectorIndexConfig
2. Creates prefixed storage — each component gets an isolated namespace (lexical/, vector/, documents/)
3. Initializes stores — LexicalStore and VectorStore are constructed with their configs
4. Recovers from WAL — replays any uncommitted operations from a previous session

Schema Splitting

The Schema contains both lexical and vector fields. At build time, split_schema() separates them:

graph LR
    S["Schema\ntitle: Text\nbody: Text\ncategory: Text\npage: Integer\ncontent_vec: HNSW"]

    S --> LC["LexicalIndexConfig\ntitle: TextOption\nbody: TextOption\ncategory: TextOption\npage: IntegerOption\n_id: KeywordAnalyzer"]

    S --> VC["VectorIndexConfig\ncontent_vec: HnswOption\n(dim=384, m=16, ef=200)"]

Key details:

• The reserved _id field is always added to the lexical config with KeywordAnalyzer (exact match)
• A PerFieldAnalyzer wraps per-field analyzer settings; if you pass a simple StandardAnalyzer, it becomes the default for all text fields
• A PerFieldEmbedder works the same way for vector fields

Indexing Data Flow

When you call engine.add_document(id, doc):

sequenceDiagram
    participant User
    participant Engine
    participant WAL as DocumentLog (WAL)
    participant Lexical as LexicalStore
    participant Vector as VectorStore

    User->>Engine: add_document("doc-1", doc)
    Engine->>WAL: Append to WAL
    Engine->>Engine: Assign internal ID (u64)

    loop For each field in document
        alt Lexical field (text, integer, etc.)
            Engine->>Lexical: Analyze + index field
        else Vector field
            Engine->>Vector: Embed + index field
        end
    end

    Note over Engine: Document is buffered\nbut NOT yet searchable

    User->>Engine: commit()
    Engine->>Lexical: Flush segments to storage
    Engine->>Vector: Flush segments to storage
    Engine->>WAL: Truncate WAL
    Note over Engine: Documents are\nnow searchable

Key points:

• WAL-first: every write is logged before modifying in-memory structures
• Dual indexing: each field is routed to either the lexical or vector store based on the schema
• Commit required: documents become searchable only after commit()

Search Data Flow

When you call engine.search(request):

sequenceDiagram
    participant User
    participant Engine
    participant Lexical as LexicalStore
    participant Vector as VectorStore
    participant Fusion

    User->>Engine: search(request)

    opt Filter query present
        Engine->>Lexical: Execute filter query
        Lexical-->>Engine: Allowed document IDs
    end

    par Lexical search
        Engine->>Lexical: Execute lexical query
        Lexical-->>Engine: Ranked hits (BM25)
    and Vector search
        Engine->>Vector: Execute vector query
        Vector-->>Engine: Ranked hits (similarity)
    end

    alt Both lexical and vector results
        Engine->>Fusion: Fuse results (RRF or WeightedSum)
        Fusion-->>Engine: Merged ranked list
    end

    Engine->>Engine: Apply offset + limit
    Engine-->>User: Vec of SearchResult

The search pipeline has three stages:

1. Filter (optional) — execute a filter query on the lexical index to get a set of allowed document IDs
2. Search — run lexical and/or vector queries in parallel
3. Fusion — if both query types are present, merge results using RRF (default, k=60) or WeightedSum

Storage Architecture

All components share a single Storage trait implementation, but use key prefixes to isolate their data:

graph TB
    Engine --> PS1["PrefixedStorage\nprefix: 'lexical/'"]
    Engine --> PS2["PrefixedStorage\nprefix: 'vector/'"]
    Engine --> PS3["PrefixedStorage\nprefix: 'documents/'"]

    PS1 --> S["Storage Backend\n(Memory / File / File+Mmap)"]
    PS2 --> S
    PS3 --> S

Per-Field Dispatch

When a PerFieldAnalyzer is provided, the engine dispatches analysis to field-specific analyzers. The same pattern applies to PerFieldEmbedder.

graph LR
    PFA["PerFieldAnalyzer"]
    PFA -->|"title"| KA["KeywordAnalyzer"]
    PFA -->|"body"| SA["StandardAnalyzer"]
    PFA -->|"description"| JA["JapaneseAnalyzer"]
    PFA -->|"_id"| KA2["KeywordAnalyzer\n(always)"]
    PFA -->|other fields| DEF["Default Analyzer\n(StandardAnalyzer)"]

This allows different fields to use different analysis strategies within the same engine.

Summary

Next Steps

• Understand field types and schema design: Schema & Fields
• Learn about text analysis: Text Analysis
• Learn about embeddings: Embeddings

Getting Started

Welcome to Laurus! This section will help you install the library and run your first search.

What You Will Build

By the end of this guide, you will have a working search engine that can:

• Index text documents
• Perform keyword (lexical) search
• Perform semantic (vector) search
• Combine both with hybrid search

Prerequisites

• Rust 1.85 or later (edition 2024)
• Cargo (included with Rust)
• Tokio runtime (Laurus uses async APIs)

Steps

1. Installation — Add Laurus to your project and choose feature flags
2. Quick Start — Build a complete search engine in 5 steps

Workflow Overview

Building a search application with Laurus follows a consistent pattern:

graph LR
    A["1. Create\nStorage"] --> B["2. Define\nSchema"]
    B --> C["3. Build\nEngine"]
    C --> D["4. Index\nDocuments"]
    D --> E["5. Search"]

Installation

Add Laurus to Your Project

Add laurus and tokio (async runtime) to your Cargo.toml:

[dependencies]
laurus = "0.13"
tokio = { version = "1", features = ["full"] }

Feature Flags

Laurus ships with a minimal default feature set. Enable additional features as needed:

Examples

Lexical search only (no embeddings needed):

[dependencies]
laurus = "0.13"

Vector search with local model (no API key required):

[dependencies]
laurus = { version = "0.13", features = ["embeddings-candle"] }

Vector search with OpenAI:

[dependencies]
laurus = { version = "0.13", features = ["embeddings-openai"] }

Everything:

[dependencies]
laurus = { version = "0.13", features = ["embeddings-all"] }

Verify Installation

Create a minimal program to verify that Laurus compiles:

use laurus::Result;

#[tokio::main]
async fn main() -> Result<()> {
    println!("Laurus version: {}", laurus::VERSION);
    Ok(())
}

cargo run

If you see the version printed, you are ready to proceed to the Quick Start.

Quick Start

This tutorial walks you through building a complete search engine in 5 steps. By the end, you will be able to index documents and search them by keyword.

Step 1 — Create Storage

Storage determines where Laurus persists index data. For development and testing, use MemoryStorage:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::storage::memory::MemoryStorage;
use laurus::Storage;

let storage: Arc<dyn Storage> = Arc::new(
    MemoryStorage::new(Default::default())
);
}

Tip: For production, consider FileStorage (with optional use_mmap for memory-mapped I/O). See Storage for details.

Step 2 — Define a Schema

A Schema declares the fields in your documents and how each field should be indexed:

#![allow(unused)]
fn main() {
use laurus::Schema;
use laurus::lexical::TextOption;

let schema = Schema::builder()
    .add_text_field("title", TextOption::default())
    .add_text_field("body", TextOption::default())
    .add_default_field("body")  // used when no field is specified in a query
    .build();
}

Each field has a type. Common types include:

See Schema & Fields for the full list.

Step 3 — Build an Engine

The Engine ties storage, schema, and runtime components together:

#![allow(unused)]
fn main() {
use laurus::Engine;

let engine = Engine::builder(storage, schema)
    .build()
    .await?;
}

When you only use text fields, the default StandardAnalyzer is used automatically. To customize analysis or add vector embeddings, see Architecture.

Step 4 — Index Documents

Create documents with the DocumentBuilder and add them to the engine:

#![allow(unused)]
fn main() {
use laurus::Document;

// Each document needs a unique external ID (string)
let doc = Document::builder()
    .add_text("title", "Introduction to Rust")
    .add_text("body", "Rust is a systems programming language focused on safety and performance.")
    .build();
engine.add_document("doc-1", doc).await?;

let doc = Document::builder()
    .add_text("title", "Python for Data Science")
    .add_text("body", "Python is widely used in machine learning and data analysis.")
    .build();
engine.add_document("doc-2", doc).await?;

let doc = Document::builder()
    .add_text("title", "Web Development with JavaScript")
    .add_text("body", "JavaScript powers interactive web applications and server-side code with Node.js.")
    .build();
engine.add_document("doc-3", doc).await?;

// Commit to make documents searchable
engine.commit().await?;
}

Important: Documents are not searchable until commit() is called.

Step 5 — Search

Use SearchRequestBuilder with a query to search the index:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

// Search for "rust" in the "body" field
let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(
            Box::new(TermQuery::new("body", "rust"))
        )
    )
    .limit(10)
    .build();

let results = engine.search(request).await?;

for result in &results {
    println!("ID: {}, Score: {:.4}", result.id, result.score);
    if let Some(doc) = &result.document {
        if let Some(title) = doc.get("title") {
            println!("  Title: {:?}", title);
        }
    }
}
}

Complete Example

Here is the full program that you can copy, paste, and run:

use std::sync::Arc;
use laurus::{
    Document, Engine,
    Result, Schema, SearchRequestBuilder,
};
use laurus::lexical::{TextOption, TermQuery};
use laurus::lexical::search::searcher::LexicalSearchQuery;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> Result<()> {
    // 1. Storage
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    // 2. Schema
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_default_field("body")
        .build();

    // 3. Engine
    let engine = Engine::builder(storage, schema).build().await?;

    // 4. Index documents
    for (id, title, body) in [
        ("doc-1", "Introduction to Rust", "Rust is a systems programming language focused on safety."),
        ("doc-2", "Python for Data Science", "Python is widely used in machine learning."),
        ("doc-3", "Web Development", "JavaScript powers interactive web applications."),
    ] {
        let doc = Document::builder()
            .add_text("title", title)
            .add_text("body", body)
            .build();
        engine.add_document(id, doc).await?;
    }
    engine.commit().await?;

    // 5. Search
    let request = SearchRequestBuilder::new()
        .lexical_query(
            LexicalSearchQuery::Obj(
                Box::new(TermQuery::new("body", "rust"))
            )
        )
        .limit(10)
        .build();

    let results = engine.search(request).await?;
    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }

    Ok(())
}

Next Steps

• Learn how the Engine works internally: Architecture
• Understand Schema and field types: Schema & Fields
• Add vector search: Vector Search
• Combine lexical + vector: Hybrid Search

Examples

The laurus/examples/ directory contains runnable examples demonstrating different features of the library.

Running Examples

# Run an example without feature flags
cargo run --example <name>

# Run an example with a feature flag
cargo run --example <name> --features <flag>

Available Examples

quickstart

A minimal example showing the basic workflow: create storage, define a schema, build an engine, index documents, and search.

cargo run --example quickstart

Demonstrates: In-memory storage, TextOption, TermQuery, LexicalSearchRequest.

lexical_search

Comprehensive example of all lexical query types, using both the Builder API and the QueryParser DSL.

cargo run --example lexical_search

Demonstrates: TermQuery, PhraseQuery, FuzzyQuery, WildcardQuery, NumericRangeQuery, GeoDistanceQuery / GeoBoundingBoxQuery, BooleanQuery, SpanQuery.

vector_search

Vector search with a mock embedder, including filtered vector search and DSL syntax.

cargo run --example vector_search

Demonstrates: PerFieldEmbedder, VectorSearchRequestBuilder, filtered search, DSL syntax (field:"query").

hybrid_search

Combining lexical and vector search with different fusion algorithms.

cargo run --example hybrid_search

Demonstrates: Lexical-only, vector-only, and hybrid search. Both RRF and WeightedSum fusion algorithms. Builder API and DSL.

geo3d_search

3D Earth-Centered Earth-Fixed (ECEF) geographic search. Indexes six landmarks (Tokyo Tower, Tokyo Skytree, Mt. Fuji summit, Statue of Liberty, Sydney Opera House, an ISS sample point) computed via wgs84_to_ecef.

cargo run --example geo3d_search

Demonstrates: Geo3dDistanceQuery (sphere), Geo3dBoundingBoxQuery (3D AABB), Geo3dNearestQuery (k-NN), and the wgs84_to_ecef conversion utility. The bounding-box query intentionally excludes the ISS sample to highlight that altitude is a first-class third axis.

search_with_candle

Vector search using real BERT embeddings via Hugging Face Candle. The model is downloaded automatically on first run (~80 MB).

cargo run --example search_with_candle --features embeddings-candle

Requires: embeddings-candle feature flag.

Demonstrates: CandleBertEmbedder with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions).

search_with_openai

Vector search using the OpenAI Embeddings API.

export OPENAI_API_KEY=your-api-key
cargo run --example search_with_openai --features embeddings-openai

Requires: embeddings-openai feature flag, OPENAI_API_KEY environment variable.

Demonstrates: OpenAIEmbedder with text-embedding-3-small (1536 dimensions).

multimodal_search

Multimodal (text + image) search using a CLIP model.

cargo run --example multimodal_search --features embeddings-multimodal

Requires: embeddings-multimodal feature flag.

Demonstrates: CandleClipEmbedder, indexing images from the filesystem, text-to-image and image-to-image queries.

synonym_graph_filter

Demonstrates the SynonymGraphFilter for token expansion during analysis.

cargo run --example synonym_graph_filter

Demonstrates: Synonym dictionary creation, synonym-based token expansion, boost application, token position and position_length attributes.

Helper Module: common.rs

The common.rs file provides shared utilities used by the examples:

• memory_storage() – Create an in-memory storage instance
• per_field_analyzer() – Create a PerFieldAnalyzer with KeywordAnalyzer for specific fields
• MockEmbedder – A mock Embedder implementation for testing vector search without a real model

Schema & Fields

The Schema defines the structure of your documents — what fields exist and how each field is indexed. It is the single source of truth for the Engine.

For the TOML file format used by the CLI, see Schema Format Reference.

Schema

A Schema is a collection of named fields. Each field is either a lexical field (for keyword search) or a vector field (for similarity search).

#![allow(unused)]
fn main() {
use laurus::Schema;
use laurus::lexical::TextOption;
use laurus::lexical::core::field::IntegerOption;
use laurus::vector::HnswOption;

let schema = Schema::builder()
    .add_text_field("title", TextOption::default())
    .add_text_field("body", TextOption::default())
    .add_integer_field("year", IntegerOption::default())
    .add_hnsw_field("embedding", HnswOption::default())
    .add_default_field("body")
    .build();
}

Default Fields

add_default_field() specifies which field(s) are searched when a query does not explicitly name a field. This is used by the Query DSL parser.

Field Types

graph TB
    FO["FieldOption"]

    FO --> T["Text"]
    FO --> I["Integer"]
    FO --> FL["Float"]
    FO --> B["Boolean"]
    FO --> DT["DateTime"]
    FO --> G["Geo"]
    FO --> G3["Geo3d"]
    FO --> BY["Bytes"]

    FO --> FLAT["Flat"]
    FO --> HNSW["HNSW"]
    FO --> IVF["IVF"]

Lexical Fields

Lexical fields are indexed using an inverted index and support keyword-based queries.

Text Field Options

TextOption controls how text is indexed:

#![allow(unused)]
fn main() {
use laurus::lexical::TextOption;

// Default: indexed + stored + term vectors + doc values (all true)
let opt = TextOption::default();

// Customize
let opt = TextOption::default()
    .indexed(true)
    .stored(true)
    .multi_valued(false)
    .position_increment_gap(100)
    .term_vectors(true)
    .doc_values(true);
}

doc_values is not unique to TextOption — every lexical field option except BytesOption carries it (IntegerOption, FloatOption, BooleanOption, DateTimeOption, GeoOption, Geo3dOption). BytesOption has no such setting: a Bytes value is never written to DocValues regardless, since neither sorting nor faceting can do anything with it. The effective rule is: a DocValues column is written only when stored and doc_values are both true (and the value’s type isn’t Bytes); setting doc_values: false on a field that is never sorted or faceted on shrinks its segment footprint by skipping the second copy.

Vector Fields

Vector fields are indexed using vector indexes for approximate nearest neighbor (ANN) search.

HNSW Field Options (most common)

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::distance::DistanceMetric;

use laurus::vector::core::quantization::QuantizationMethod;

let opt = HnswOption {
    dimension: 384,                                  // vector dimensions
    distance: DistanceMetric::Cosine,                // distance metric
    m: 16,                                           // max connections per layer
    ef_construction: 200,                            // construction search width
    default_ef_search: Some(100),                    // schema-level ef_search default (issue #644)
    base_weight: 1.0,                                // relative priority vs. other vector fields (issue #1084)
    quantizer: QuantizationMethod::Scalar8Bit,       // mandatory; default Scalar8Bit
    embedder: None,                                  // optional named embedder
};
}

default_ef_search: the search-time recall knob

ef_search controls the dynamic candidate list size during query time (distinct from ef_construction, which only affects index build). Higher values explore more graph neighbours and yield higher recall at the cost of latency.

• Schema-level default: set HnswOption.default_ef_search = Some(ef) to raise the per-field default. When None, the searcher falls back to its built-in 50.
• Per-query override: search requests honour SearchRequestBuilder::vector_ef_search. The per-query value takes precedence over the schema default.
• Auto-lifting: regardless of which source provides ef_search, the searcher lifts the effective value to at least top_k (and top_k * rerank_factor when both are set) so the candidate heap is never undersized for the requested top_k.
• Tracked under Issue #644.

See Vector Indexing for detailed parameter guidance.

Multi-Vector Fields

A MultiVector field holds all the token vectors of a document, for example the per-token embeddings of a ColBERT-style late-interaction model (typically 100–300 vectors of 128 dimensions per passage). It has no ANN index: it is never a vector-search target, and its vectors are read only to rescore candidates that another query (lexical, vector, or hybrid) already found.

#![allow(unused)]
fn main() {
use laurus::{MultiVectorOption, Schema};
use laurus::vector::core::distance::DistanceMetric;

let schema = Schema::builder()
    .add_multi_vector_field(
        "body_colbert",
        MultiVectorOption::new(128).distance(DistanceMetric::Cosine), // Cosine (default) or DotProduct
    )
    .build();
}

• Options: dimension (every token vector’s length), distance, storage, pool_factor and embedder. Only Cosine (vectors are L2-normalized when written, so the similarity is their dot product; a zero vector is kept as-is) and DotProduct are accepted for distance; other metrics are rejected when the index is created. storage picks the on-disk element kind of the token vectors — F32 (default, exact), F16, Int8, or Binary (1 bit per element; requires Cosine) — trading some precision for a smaller index; see MultiVector Storage for the size and quality figures. pool_factor (default 1, no pooling) keeps about 1 / pool_factor of each document’s token vectors by averaging similar ones at index time; see Token pooling. embedder names a token-level embedder, such as a candle_colbert one (MultiVectorOption::new(96).embedder("colbert")).
• Values: a DataValue::VectorArray(Vec<Vec<f32>>) holding at least one and at most 8,192 vectors, each of the field’s dimension and with finite values. In JSON it is an array of equal-length numeric arrays ([[0.1, 0.2, ...], [0.3, 0.4, ...]]). When a token-level embedder serves the field (named in embedder, or registered through EngineBuilder::embedder), a text value works too: it is embedded into token vectors when the document is indexed (Issue #1349), and a document is rejected if the field has no such embedder.
• Not stored: the token vectors are kept only in the vector index (about 150 KB per document for 300 × 128 f32), so get_documents and search results never include the field.
• Not queryable: a field-less vector query and a prefix field selector skip it, naming it in a vector query is an error, and so is naming it in a DSL query.
• Schema changes: changing its dimension, distance or embedder, or changing it to or from another vector field type, is a destructive change. Changing storage is a Reindex change instead — the stored values are decoded and re-encoded, so Engine::update_field(.., UpdateFieldOptions { reindex: true, .. }) rebuilds the field’s existing segments under the new on-disk kind (precision a lossy kind already dropped does not come back). Turning pooling on (pool_factor from 1 to more) is a Reindex change too: the stored token vectors are pooled. Changing or turning off a pool_factor already in use is destructive, because pooled token vectors cannot be restored. An option the field’s index would reject, such as Binary with DotProduct or a pool_factor of 0, is rejected before any data is touched.

Document

A Document is a collection of named field values. Use DocumentBuilder to construct documents:

#![allow(unused)]
fn main() {
use laurus::Document;

let doc = Document::builder()
    .add_text("title", "Introduction to Rust")
    .add_text("body", "Rust is a systems programming language.")
    .add_integer("year", 2024)
    .add_float("rating", 4.8)
    .add_boolean("published", true)
    .build();
}

Indexing Documents

The Engine provides two methods for adding documents, each with different semantics:

#![allow(unused)]
fn main() {
// Upsert: replaces any existing document with id "doc1"
engine.put_document("doc1", doc).await?;

// Append: adds another chunk under the same id "doc1"
engine.add_document("doc1", chunk2).await?;

// Always commit after indexing
engine.commit().await?;
}

Retrieving Documents

Use get_documents to retrieve all documents (including chunks) by external ID:

#![allow(unused)]
fn main() {
let docs = engine.get_documents("doc1").await?;
for doc in &docs {
    if let Some(title) = doc.get("title") {
        println!("Title: {:?}", title);
    }
}
}

Deleting Documents

Delete all documents and chunks sharing an external ID:

#![allow(unused)]
fn main() {
engine.delete_documents("doc1").await?;
engine.commit().await?;
}

Document Lifecycle

graph LR
    A["Build Document"] --> B["put/add_document()"]
    B --> C["WAL"]
    C --> D["commit()"]
    D --> E["Searchable"]
    E --> F["get_documents()"]
    E --> G["delete_documents()"]

Important: Documents are not searchable until commit() is called.

DocumentBuilder Methods

DataValue

DataValue is the unified value enum that represents any field value in Laurus:

#![allow(unused)]
fn main() {
pub enum DataValue {
    Null,
    Bool(bool),
    Int64(i64),
    Float64(f64),
    Text(String),
    Bytes(Vec<u8>, Option<String>),  // (data, optional MIME type)
    Vector(Vec<f32>),
    DateTime(DateTime<Utc>),
    Geo(GeoPoint),                   // 2D WGS84 point (latitude, longitude)
    GeoEcef(GeoEcefPoint),           // 3D ECEF Cartesian point (x, y, z) in metres
    Int64Array(Vec<i64>),            // multi-valued integer field
    Float64Array(Vec<f64>),          // multi-valued float field
    GeoArray(Vec<GeoPoint>),         // multi-valued 2D geo field
    GeoEcefArray(Vec<GeoEcefPoint>), // multi-valued 3D ECEF field
    DateTimeArray(Vec<DateTime<Utc>>), // multi-valued datetime field
    BoolArray(Vec<bool>),            // multi-valued boolean field
    TextArray(Vec<String>),          // multi-valued text field
    BytesArray(Vec<(Vec<u8>, Option<String>)>), // multi-valued bytes field, per-element MIME
    VectorArray(Vec<Vec<f32>>),      // token vectors of a MultiVector field
}
}

DataValue implements From<T> for common types, so you can use .into() conversions:

#![allow(unused)]
fn main() {
use laurus::DataValue;

let v: DataValue = "hello".into();       // Text
let v: DataValue = 42i64.into();         // Int64
let v: DataValue = 3.14f64.into();       // Float64
let v: DataValue = true.into();          // Bool
let v: DataValue = vec![0.1f32, 0.2].into(); // Vector
}

Reserved Fields

Any field name starting with an underscore (_) is reserved for the engine. Declaring such a field is rejected when the index is created, and documents that carry user-supplied _-prefixed keys are rejected at ingest time — so the field can never hold a value either way. The language bindings check even earlier: their field-adding methods (add_*_field / add*Field) reject such a name as soon as it is passed. See Field Naming for the exact error and how an index created before this check is handled.

The only _-prefixed name that is accepted is the allow-listed _id system field described below.

_id — external document identifier

Stores the external document ID supplied to put_document / add_document. It is injected automatically and indexed with KeywordAnalyzer (exact match). You do not need to add it to your schema.

Dynamic Schema

Laurus can accept documents even when some of their fields have not been declared in the schema. The behaviour is controlled by the DynamicFieldPolicy attached to the schema:

Set the policy on the builder:

#![allow(unused)]
fn main() {
use laurus::{DynamicFieldPolicy, Schema};

let schema = Schema::builder()
    .dynamic_field_policy(DynamicFieldPolicy::Dynamic)
    .build();
}

Type inference rules (Dynamic policy)

Vector fields (Hnsw, Flat, Ivf, MultiVector) are never inferred: they must be declared in the schema explicitly, since dimension, distance metric, and embedder configuration cannot be recovered from a value alone. An undeclared field carrying token vectors is rejected. Bytes values can be parsed from the {data, mime} object shape above, but an undeclared field carrying a Bytes value is still rejected rather than auto-registered — bytes fields must always be declared explicitly, same as vector fields. Mixing markers from more than one shape (2D lat/lon, 3D x/y/z, or bytes data) in a single object is rejected as ambiguous; use exactly one shape per object.

Multi-valued fields

Integer and Float fields can be declared with multi_valued = true to hold multiple values per document. A range query matches a document if any of its values satisfies the predicate (Lucene-style “any match” semantics with constant scoring — there is no per-match BM25 weighting).

Geo and Geo3d fields accept multi_valued = true as well. Every point of a multi-valued geo field becomes its own entry in the field’s BKD tree (2 dimensions for Geo, 3 for Geo3d), so distance and bounding-box queries — and, for Geo3d, nearest queries — match a document if any of its points satisfies the predicate. A document is reported once, scored by its closest matching point (for the 2D bounding-box query, the point closest to the box centre). Under the Dynamic policy an array that mixes 2D and 3D objects, or numbers and objects, is rejected; each element gets the same range validation as a single point.

DateTime fields accept multi_valued = true as well (Issue #1184). Every instant of a multi-valued datetime field becomes its own 1-dimensional entry in the field’s BKD tree, so DateTimeRangeQuery, a NumericRangeQuery on the field, and DSL date ranges such as seen_at:[2024-06-01 TO 2024-12-31] match a document if any of its instants is in range (Lucene-style “any match” with constant scoring — there is no per-match BM25 weighting). A document is reported once even when several of its instants match, and sub-second instants are honored (the bounds are encoded as fractional seconds). Under the Dynamic policy an array whose elements are all RFC 3339 strings infers a multi-valued DateTime; only RFC 3339 is recognised here (not the naive or date-only forms the query DSL accepts), and an array of strings where any element is not RFC 3339 infers a multi-valued Text instead (Issue #1175) — it is not an error. Note the asymmetry: a single RFC 3339 string still infers Text (unchanged, so existing text fields keep their behaviour), while an array of RFC 3339 strings infers a multi-valued DateTime — declare the field in the schema if you want a single-valued DateTime.

Boolean fields accept multi_valued = true as well (Issue #1180), but the mechanism differs from every other multi-valued type: a Boolean field has no BKD points. Every element of a multi-valued boolean field is indexed as its own "true" / "false" term posting, so TermQuery and DSL term queries such as flags:true match a document if any element equals the queried value (Lucene-style “any match”). A document is reported once for a given term even when several of its elements equal it (postings are aggregated per document and term), and Boolean fields remain not range-queryable. Because these are ordinary term postings, repeated elements affect BM25 scoring the way they do in Lucene: [true, true] yields one posting with term frequency 2 and field length 2, so it scores higher than [true] for flags:true, while [true, false] scores slightly lower than [true] because of length normalization. There is no dedupe and no constant scoring for term queries (constant-scored term queries are tracked in Issue #580). Under the Dynamic policy a JSON array whose elements are all booleans (e.g. [true, false]) infers a multi-valued Boolean; a mixed array such as [true, 1] is rejected with the existing error that array fields must contain only numeric values. Note the asymmetry with scalars: a single "true" string coerces to a Bool on a declared Boolean field, and a string array such as ["true", "false"] sent to a declared multi-valued Boolean field is parsed element-wise under the same rule; on an undeclared field, however, that array infers a multi-valued Text (Issue #1175), not a Boolean — use JSON booleans for arrays you want inferred as booleans. Turning multi_valued on for an existing Boolean field is a metadata-only change; turning it off requires a reindex when the field is stored, and is Destructive when the field is stored: false — unlike the BKD-backed types there is no point tree to rebuild from, only the stored value.

Text fields accept multi_valued = true as well (Issue #1175), together with a second option, position_increment_gap (default 100, matching Lucene and Elasticsearch). Every element of a multi-valued text field is analyzed on its own by the field’s analyzer, and the tokens of all elements are appended to one ascending position sequence: the first token of element n + 1 is placed position_increment_gap positions after the last token of element n. The gap is charged per element, even for an element that analyzes to no tokens. Everything else follows from that numbering:

• A TermQuery (or a DSL term such as tags:rust) matches a document if any element contains the term (Lucene-style “any match”).
• A PhraseQuery cannot span two elements unless its slop is at least position_increment_gap — the threshold is exactly slop == gap. With the default gap, ["hello world", "foo bar"] does not match the phrase "world foo" at slop 0 through 99 and does match at slop 100. Span queries (SpanNearQuery) inherit the same protection.
• A gap of 0 numbers the elements as if they had been concatenated, so phrases then match across element boundaries. The gap is never a “restart at 0”.
• Repeated terms across elements raise the term frequency (and hence the BM25 score) but not the hit count, like the other multi-valued types. The field length is the number of positions, not the number of tokens — a synonym stacked on another token (position_increment = 0) shares its position and is not counted again — so the gap itself does not inflate BM25 length normalization.
• Positions must be stored for phrase queries of two or more terms to work at all (term_vectors: true, the default); without them the search is rejected with a query error, for multi-valued and single-valued fields alike. A quoted value that analyzes to one token, such as tags:"lang/rust" on a keyword field, is a term match and does not need positions.

A multi-valued text field is highlighted element by element — only the elements that match contribute fragments, and a fragment never straddles two elements; see Highlighting. Under the Dynamic policy a JSON array whose elements are all strings infers a multi-valued DateTime when every element parses as RFC 3339 and a multi-valued Text otherwise (e.g. ["rust", "search"]); a single string still infers Text, never a datetime. Turning multi_valued on for an existing Text field is a metadata-only change; turning it off requires a reindex when the field is stored and is Destructive when it is stored: false — like Boolean, there is no BKD tree to rebuild from. Changing position_increment_gap requires a reindex when the field stores positions (term_vectors: true) — Reindex when stored, Destructive otherwise — because the existing postings were numbered under the old gap; when term_vectors is false the gap is unobservable and the change is metadata-only.

Bytes fields accept multi_valued = true as well (Issue #1176), but the mechanism differs from every other multi-valued type, Boolean included: BytesOption has no indexed flag at all — a Bytes value is never lexically indexed, so there are no BKD points and no term postings, and multi_valued has no “any match” query semantics to speak of. It only governs the stored shape and the ingestion arity. DataValue::BytesArray is Vec<(Vec<u8>, Option<String>)>: unlike the bare Vec<T> every other *Array variant uses, each element mirrors the scalar Bytes(Vec<u8>, Option<String>) shape exactly, carrying its own MIME type. A Bytes value — scalar or array — can never be inferred for an undeclared field under the Dynamic policy (see Type inference rules above); a multi-valued bytes field must always be declared explicitly. Once declared, a single Bytes value or a base64 Text string is auto-wrapped into a one-element array (a Text element is decoded as base64, the same rule the scalar field uses), and an empty numeric array ([]) is accepted as an empty byte-value list. Turning multi_valued on for an existing Bytes field is a metadata-only change; turning it off requires a reindex when the field is stored, and is Destructive when the field is stored: false — like Boolean and Text, there is no point tree or postings to rebuild from, only the stored value.

Faceting expands every multi-valued type element by element (Issue #1187): each integer, float, boolean, datetime or text element becomes its own facet value (a text element containing / becomes a hierarchical path), counted once per document even when an element repeats; geo and bytes arrays are not facetable and contribute nothing. See Faceting for the element formats.

Single values sent to a multi-valued field are auto-wrapped into a one-element array; arrays sent to a single-valued field are rejected rather than silently truncating (the error tells you to declare the field with multi_valued = true). An empty array sent to a multi-valued geo, datetime, boolean or text field is accepted and simply has no points, instants or terms, so it matches no spatial, range, term or phrase query; an empty array on a multi-valued bytes field is accepted the same way and is simply stored as an empty list — a Bytes field is never queried, so there is no query for it to not match.

Segments that contain multi-valued geo, datetime, boolean, text or bytes values use new stored-field type tags and cannot be read by builds that predate these features; there is no format version bump, so an older reader fails loudly instead of misreading the data. Stored multi-valued datetimes are kept at microsecond precision (one i64 Unix microsecond per instant, so sub-microsecond digits are truncated), whereas a single-valued DateTime keeps its full precision. Stored multi-valued booleans are written one byte per element (not bit-packed). Stored multi-valued text values are written as an element count followed by each string with its length prefix — the same body as a single-valued text value. Stored multi-valued bytes values are written as an element count followed by, for each element, the same (MIME, then data) length-prefixed layout the scalar Bytes value uses (an empty MIME string means None).

Type conflicts

When a value arrives for a field that is already declared, Laurus attempts to coerce the value to the declared type. The coercion rules are:

Coercion errors interact with the policy:

• Strict: error is returned immediately.
• Dynamic: error is returned — the coercion layer already applied every conversion that is considered safe.
• Ignore: the offending field is dropped; the rest of the document is indexed.

⚠️ Warning: silent information loss is possible.

Several coercions throw away information without reporting an error:

• An Integer field truncates incoming Float values (3.14 → 3, -3.9 → -3). Ingest does not fail.
• A Float field may lose precision for very large integers that do not fit in an f64 mantissa.
• A Text field accepts any scalar by stringifying it, losing the original type.
• Ignore drops incompatible fields quietly.

If the correctness of your data matters more than the convenience of schema-less ingestion, use DynamicFieldPolicy::Strict (or declare every field up-front). The Dynamic policy prioritises keeping the document ingestable over preserving every bit of incoming data.

Query DSL and undeclared fields

Once the schema is settled, the query parser validates that every field:value clause references a declared field. Typos such as titl:hello (for title:hello) produce a clear parse error instead of returning silently-empty results.

Dynamic Field Management

Fields can be added to, removed from, or changed on a running engine at runtime.

Adding a Field

Use Engine::add_field() to add a new field to the schema.

Adding a Lexical Field

let updated_schema = engine.add_field(
    "category",
    FieldOption::Text(TextOption::default()),
).await?;

Adding a Vector Field

let updated_schema = engine.add_field(
    "embedding",
    FieldOption::Flat(FlatOption::default().dimension(384)),
).await?;

Existing documents are unaffected—they simply have no value for the new field.

If a schema-persist hook was configured via Engine::builder().persist_schema_with(hook), add_field/delete_field/ update_field call it themselves before returning, persisting the schema on the caller’s behalf (both laurus-cli and laurus-server configure this hook to write schema.toml). Otherwise, as before, the returned Schema should be persisted (e.g., to schema.toml) by the caller.

Removing a Field

Use Engine::delete_field() to remove a field from the schema.

let updated_schema = engine.delete_field("category").await?;

When a field is deleted:

• The field definition is removed from the schema.
• Existing indexed data for the field remains in the index but becomes inaccessible through queries.
• If the field was listed in default_fields, it is automatically removed.
• Any per-field analyzer or embedder registered for the field is unregistered.

Changing a Field

Use Engine::update_field() to change an existing field’s type/options on a running engine.

let outcome = engine.update_field(
    "title",
    FieldOption::Text(TextOption::default().analyzer("english")),
    UpdateFieldOptions { reindex: true, ..Default::default() },
).await?;

The requested change is classified into one of three kinds, reported as outcome.classification:

• MetadataOnly — no existing data is affected; always applied (e.g. an HNSW field’s default_ef_search).
• Reindex — can be rebuilt from the field’s already-stored values (e.g. a text field’s analyzer or — while term_vectors is on — its position_increment_gap, indexed: false → true, or an HNSW field’s m/ef_construction).
• Destructive — cannot be rebuilt from existing data; applying it discards the field’s data (e.g. a vector field’s dimension/embedder/ distance, a type change on a stored: false field, or turning multi_valued off on a stored: false Boolean, Text or Bytes field).

Reindex and Destructive changes are rejected unless you explicitly pass UpdateFieldOptions { reindex: true, .. } — an opt-in gate against accidentally triggering an expensive rebuild or a data-losing change.

Applying a Destructive change records the field’s name in the schema’s pending_reindex set. This is a visibility mechanism against the known Solr failure mode where the schema and the index silently drift apart — check it via GetSchema / laurus get schema to find fields whose data no longer matches the schema, and re-ingest their documents as needed.

A vector field’s registered embedder follows its new embedder option (Issue #1354): naming one registers it, and removing the name unregisters the previous one, so the field falls back to the default embedder (text sent to it is rejected when that default is PrecomputedEmbedder). A change to a field that names no embedder, before or after, leaves an embedder registered directly through EngineBuilder::embedder in place.

Pass UpdateFieldOptions { dry_run: true, .. } to see how a change would be classified without applying anything.

Schema Design Tips

1. Separate lexical and vector fields — a field is either lexical or vector, never both. For hybrid search, create separate fields (e.g., body for text, body_vec for vector).

2. Use KeywordAnalyzer for exact-match fields — category, status, and tag fields should use KeywordAnalyzer via PerFieldAnalyzer to avoid tokenization.

3. Choose the right vector index — use HNSW for most cases, Flat for small datasets, IVF for very large datasets. See Vector Indexing.

4. Set default fields — if you use the Query DSL, set default fields so users can write hello instead of body:hello.

5. Use the schema generator — run laurus create schema to interactively build a schema TOML file instead of writing it by hand. See CLI Commands.

Text Analysis

Text analysis is the process of converting raw text into searchable tokens. When a document is indexed, the analyzer breaks text fields into individual terms; when a query is executed, the same analyzer processes the query text to ensure consistency.

The Analysis Pipeline

graph LR
    Input["Raw Text\n'The quick brown FOX jumps!'"]
    CF["UnicodeNormalizationCharFilter"]
    T["Tokenizer\nSplit into words"]
    F1["LowercaseFilter"]
    F2["StopFilter"]
    F3["StemFilter"]
    Output["Terms\n'quick', 'brown', 'fox', 'jump'"]

    Input --> CF --> T --> F1 --> F2 --> F3 --> Output

The analysis pipeline consists of:

1. Char Filters — normalize raw text at the character level before tokenization
2. Tokenizer — splits text into raw tokens (words, characters, n-grams)
3. Token Filters — transform, remove, or expand tokens (lowercase, stop words, stemming, synonyms)

The Analyzer Trait

All analyzers implement the Analyzer trait:

#![allow(unused)]
fn main() {
pub trait Analyzer: Send + Sync + Debug {
    fn analyze(&self, text: &str) -> Result<TokenStream>;
    fn name(&self) -> &str;
    fn as_any(&self) -> &dyn Any;
}
}

TokenStream is a Box<dyn Iterator<Item = Token> + Send> — a lazy iterator over tokens.

A Token contains:

Built-in Analyzers

StandardAnalyzer

The default analyzer. Suitable for most Western languages.

Pipeline: RegexTokenizer (Unicode word boundaries) → LowercaseFilter → StopFilter (128 common English stop words)

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::standard::StandardAnalyzer;

let analyzer = StandardAnalyzer::default();
// "The Quick Brown Fox" → ["quick", "brown", "fox"]
// ("The" is removed by stop word filtering)
}

JapaneseAnalyzer

Uses morphological analysis for Japanese text segmentation.

Pipeline: UnicodeNormalizationCharFilter (NFKC) → JapaneseIterationMarkCharFilter → LinderaTokenizer → LowercaseFilter → StopFilter (Japanese stop words)

JapaneseAnalyzer::new takes the same arguments as LinderaTokenizer::new: the segmentation mode, a path to a Lindera dictionary directory, and an optional user dictionary path. laurus does not enable Lindera’s embed-* features by default, so a real filesystem path (typically an IPADIC build) is required at runtime.

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::language::japanese::JapaneseAnalyzer;

// Pass the path where you have unpacked the Lindera dictionary.
let analyzer = JapaneseAnalyzer::new(
    "normal",
    "/var/lib/lindera/ipadic",
    None,
)?;
// "東京都に住んでいる" → ["東京", "都", "住ん", "いる"]
}

When the analyzer is referenced from a Schema, supply the parameters through the structured AnalyzerSpec form (see PerFieldAnalyzer below).

KeywordAnalyzer

Treats the entire input as a single token. No tokenization or normalization.

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::keyword::KeywordAnalyzer;

let analyzer = KeywordAnalyzer::new();
// "Hello World" → ["Hello World"]
}

Use this for fields that should match exactly (categories, tags, status codes).

SimpleAnalyzer

Tokenizes text without any filtering. The original case and all tokens are preserved. Useful when you need complete control over the analysis pipeline or want to test a tokenizer in isolation.

Pipeline: User-specified Tokenizer only (no char filters, no token filters)

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::simple::SimpleAnalyzer;
use laurus::analysis::tokenizer::regex::RegexTokenizer;
use std::sync::Arc;

let tokenizer = Arc::new(RegexTokenizer::new()?);
let analyzer = SimpleAnalyzer::new(tokenizer);
// "Hello World" → ["Hello", "World"]
// (no lowercasing, no stop word removal)
}

Use this for testing tokenizers, or when you want to apply token filters manually in a separate step.

EnglishAnalyzer

An English-specific analyzer. Tokenizes, lowercases, and removes common English stop words.

Pipeline: RegexTokenizer (Unicode word boundaries) → LowercaseFilter → StopFilter (128 common English stop words)

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::language::english::EnglishAnalyzer;

let analyzer = EnglishAnalyzer::new()?;
// "The Quick Brown Fox" → ["quick", "brown", "fox"]
// ("The" is removed by stop word filtering, remaining tokens are lowercased)
}

PipelineAnalyzer

Build a custom pipeline by combining any char filters, a tokenizer, and any sequence of token filters:

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::pipeline::PipelineAnalyzer;
use laurus::analysis::char_filter::unicode_normalize::{
    NormalizationForm, UnicodeNormalizationCharFilter,
};
use laurus::analysis::tokenizer::regex::RegexTokenizer;
use laurus::analysis::token_filter::lowercase::LowercaseFilter;
use laurus::analysis::token_filter::stop::StopFilter;
use laurus::analysis::token_filter::stem::StemFilter;

let analyzer = PipelineAnalyzer::new(Arc::new(RegexTokenizer::new()?))
    .add_char_filter(Arc::new(UnicodeNormalizationCharFilter::new(NormalizationForm::NFKC)))
    .add_filter(Arc::new(LowercaseFilter::new()))
    .add_filter(Arc::new(StopFilter::new()))
    .add_filter(Arc::new(StemFilter::new()));  // Porter stemmer
}

PerFieldAnalyzer

PerFieldAnalyzer lets you assign different analyzers to different fields within the same engine:

graph LR
    PFA["PerFieldAnalyzer"]
    PFA -->|"title"| KW["KeywordAnalyzer"]
    PFA -->|"body"| STD["StandardAnalyzer"]
    PFA -->|"description_ja"| JP["JapaneseAnalyzer"]
    PFA -->|other fields| DEF["Default\n(StandardAnalyzer)"]

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::analysis::analyzer::standard::StandardAnalyzer;
use laurus::analysis::analyzer::keyword::KeywordAnalyzer;
use laurus::analysis::analyzer::per_field::PerFieldAnalyzer;

// Default analyzer for fields not explicitly configured
let per_field = PerFieldAnalyzer::new(
    Arc::new(StandardAnalyzer::default())
);

// Use KeywordAnalyzer for exact-match fields
per_field.add_analyzer("category", Arc::new(KeywordAnalyzer::new()));
per_field.add_analyzer("status", Arc::new(KeywordAnalyzer::new()));

let engine = Engine::builder(storage, schema)
    .analyzer(Arc::new(per_field))
    .build()
    .await?;
}

Note: The _id field is always analyzed with KeywordAnalyzer regardless of configuration.

Configuring per-field analyzers from a Schema

Most callers configure analyzers declaratively on the schema rather than wiring them up by hand. The analyzer setting on a text field accepts two shapes:

// 1. A bare name for a parameter-less built-in or a user-registered analyzer.
{ "analyzer": "standard" }
{ "analyzer": "english" }
{ "analyzer": "my_custom_pipeline" }

// 2. A structured object for a parameterised built-in preset. Today only
//    the Japanese preset uses this form (it requires a Lindera dictionary
//    path).
{
  "analyzer": {
    "language": "japanese",
    "mode": "normal",
    "dict": "/var/lib/lindera/ipadic"
  }
}

The bare string "japanese" is rejected because the preset cannot be constructed without a dictionary. Schemas that previously stored "analyzer": "japanese" must migrate to the structured form above.

For full pipelines that do not fit a preset, register the pipeline under schema.analyzers as an AnalyzerDefinition and reference it by name. The names of the parameter-less built-ins are reserved, so give the pipeline a name of its own (see Referencing an analyzer).

Char Filters

Char filters operate on the raw input text before it reaches the tokenizer. They perform character-level normalization such as Unicode normalization, character mapping, and pattern-based replacement. This ensures that the tokenizer receives clean, normalized text.

All char filters implement the CharFilter trait:

#![allow(unused)]
fn main() {
pub trait CharFilter: Send + Sync {
    fn filter(&self, input: &str) -> (String, Vec<Transformation>);
    fn name(&self) -> &'static str;
}
}

The Transformation records describe how character positions shifted, allowing the engine to map token positions back to the original text.

UnicodeNormalizationCharFilter

Applies Unicode normalization to the input text. NFKC is recommended for search use cases because it normalizes both compatibility characters and composed forms.

#![allow(unused)]
fn main() {
use laurus::analysis::char_filter::unicode_normalize::{
    NormalizationForm, UnicodeNormalizationCharFilter,
};

let filter = UnicodeNormalizationCharFilter::new(NormalizationForm::NFKC);
// "Ｓｏｎｙ" (fullwidth) → "Sony" (halfwidth)
// "㌂" → "アンペア"
}

MappingCharFilter

Replaces character sequences using a dictionary. Matches are found using the Aho-Corasick algorithm (leftmost-longest match).

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use laurus::analysis::char_filter::mapping::MappingCharFilter;

let mut mapping = HashMap::new();
mapping.insert("ph".to_string(), "f".to_string());
mapping.insert("qu".to_string(), "k".to_string());

let filter = MappingCharFilter::new(mapping)?;
// "phone queue" → "fone keue"
}

PatternReplaceCharFilter

Replaces all occurrences of a regex pattern with a fixed string.

#![allow(unused)]
fn main() {
use laurus::analysis::char_filter::pattern_replace::PatternReplaceCharFilter;

// Remove hyphens
let filter = PatternReplaceCharFilter::new(r"-", "")?;
// "123-456-789" → "123456789"

// Normalize numbers
let filter = PatternReplaceCharFilter::new(r"\d+", "NUM")?;
// "Year 2024" → "Year NUM"
}

JapaneseIterationMarkCharFilter

Expands Japanese iteration marks (踊り字) to their base characters. Supports kanji (々), hiragana (ゝ, ゞ), and katakana (ヽ, ヾ) iteration marks.

#![allow(unused)]
fn main() {
use laurus::analysis::char_filter::japanese_iteration_mark::JapaneseIterationMarkCharFilter;

let filter = JapaneseIterationMarkCharFilter::new(
    true,  // normalize kanji iteration marks
    true,  // normalize kana iteration marks
);
// "佐々木" → "佐佐木"
// "いすゞ" → "いすず"
}

Using Char Filters in a Pipeline

Add char filters to a PipelineAnalyzer with add_char_filter(). Multiple char filters are applied in the order they are added, all before the tokenizer runs.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::analysis::analyzer::pipeline::PipelineAnalyzer;
use laurus::analysis::char_filter::unicode_normalize::{
    NormalizationForm, UnicodeNormalizationCharFilter,
};
use laurus::analysis::char_filter::pattern_replace::PatternReplaceCharFilter;
use laurus::analysis::tokenizer::regex::RegexTokenizer;
use laurus::analysis::token_filter::lowercase::LowercaseFilter;

let analyzer = PipelineAnalyzer::new(Arc::new(RegexTokenizer::new()?))
    .add_char_filter(Arc::new(
        UnicodeNormalizationCharFilter::new(NormalizationForm::NFKC),
    ))
    .add_char_filter(Arc::new(
        PatternReplaceCharFilter::new(r"-", "")?,
    ))
    .add_filter(Arc::new(LowercaseFilter::new()));
// "Ｔｏｋｙｏ-2024" → NFKC → "Tokyo-2024" → remove hyphens → "Tokyo2024" → tokenize → lowercase → ["tokyo2024"]
}

Tokenizers

Token Filters

Synonym Expansion

The SynonymGraphFilter expands terms using a synonym dictionary:

#![allow(unused)]
fn main() {
use laurus::analysis::synonym::dictionary::SynonymDictionary;
use laurus::analysis::token_filter::synonym_graph::SynonymGraphFilter;

let mut dict = SynonymDictionary::new(None)?;
dict.add_synonym_group(vec!["ml".into(), "machine learning".into()]);
dict.add_synonym_group(vec!["ai".into(), "artificial intelligence".into()]);

// keep_original=true means original token is preserved alongside synonyms
let filter = SynonymGraphFilter::new(dict, true)
    .with_boost(0.8);  // stored on each synonym token, but not yet read by
                        // scoring (see below)
}

The boost parameter is stored on each synonym Token, but neither the indexer nor the query parser reads it today, so it has no effect on scoring or matching — a synonym match currently scores exactly like an exact match (see Searching with synonyms below). Set it if you plan to consume it from a custom pipeline stage; do not rely on it to weight synonym matches lower.

Matching entries of several words

An entry of several words matches consecutive tokens when all of them are alphanumeric (token type Alphanum or Num), so English words separated by spaces match, or when their offsets touch (each token’s end_offset is the next one’s start_offset), as the CJK words a morphological tokenizer splits do. The type comes from Token::metadata, so a token without one needs touching offsets. With 東京大学 in the dictionary, the text 東京大学 split into 東京 and 大学 matches it, but 東京 大学 does not: a space separates the two tokens.

The token graph

The filter stacks each synonym on the word it expands (position_increment = 0), so the output is a graph: a token is an arc from its position to its position + position_length. The matched words and every synonym run from the same start node to the same end node. The last token of a shorter alternative spans the remaining positions, and the token after the match starts at the end node. For ml tutorial with ml and machine learning as synonyms:

A member of several words has inner nodes of its own, so no arc holds words of two members. For ml with machine learning and statistical machine learning as synonyms:

The index stores no position_length, so it lays the graph out on its longest path: the members’ words share positions there. Above, ml, machine and statistical are stored at 0, learning and machine at 1, and learning at 2. A stacked token is stored at its word’s position. Positions otherwise stay dense: a stop word removed by StopFilter leaves no gap. The index does this itself, so an analyzer needs no FlattenGraphFilter. Adding one would also flatten the graph that quoted values are matched through, because the engine parses queries with the same analyzer.

Searching with synonyms

An engine uses a field’s analyzer both to index and to parse queries, so synonyms are expanded on both sides. An unquoted word matches any of its synonyms. A quoted value is matched through the graph (see Phrase Query): "big" with big and large as synonyms matches either word, and "a big dog" also matches “a large dog”. Each member of a group is its own phrase, so with the members above "ml" matches ml, machine learning or statistical machine learning, and not “statistical learning”. Members of the same length are separate phrases too. The phrases of a quoted value are matched together through the graph, so their number, which multiplies with each synonym in the value, sets no limit.

A position with several stacked alternatives is scored as one blended term (a SynonymQuery), not as a sum of each alternative’s independent score: a document holding both big and large scores the same for "big" as one that repeats big twice, not twice as much. Field length also counts positions, not tokens, so a stacked synonym does not lengthen a document either. See BM25.

Because the index keeps positions but not position_length, index-time expansion of multi-word synonyms behaves as in Lucene:

• A phrase may start or end inside a synonym. "learning is" matches “ml is fun”, which reads “machine learning is fun” with the synonym.
• When a group has several multi-word members, their words share positions in the index, so a document with one member matches a phrase that mixes two members’ words: “ml is fun” matches "statistical learning".

Other points to keep in mind:

• keep_original = false replaces the matched word with the other members of its group, so the word itself is neither indexed nor searched. Use keep_original = true for search.
• Filters that remove tokens, such as StopFilter, RemoveEmptyFilter and LimitFilter, can follow SynonymGraphFilter. A removed word leaves no position inside the graph either: with statue of liberty and lady liberty as synonyms and of as a stop word, "statue of liberty" matches both “statue of liberty” and “lady liberty”. A StopFilter before SynonymGraphFilter removes of before the synonym filter sees it, so that member never matches.
• A removed word with an alternative at its position, such as a stacked one-word synonym, leaves its position to it: with the and a as synonyms, removing the leaves a there, and no path skips the word. LimitFilter cuts every path at the same token.
• Documents indexed before Issue #1252 numbered stacked tokens one after another, and documents indexed before Issue #1257 counted every stacked token toward the field length (inflating it). Put them again after upgrading if their fields use SynonymGraphFilter: merging segments copies both the stored positions and the field length unchanged.

Embeddings

Embeddings convert text (or images) into dense numeric vectors that capture semantic meaning. Two texts with similar meanings produce vectors that are close together in vector space, enabling similarity-based search.

The Embedder Trait

All embedders implement the Embedder trait:

#![allow(unused)]
fn main() {
#[async_trait]
pub trait Embedder: Send + Sync + Debug {
    async fn embed(&self, input: &EmbedInput<'_>) -> Result<Vector>;
    async fn embed_batch(&self, inputs: &[EmbedInput<'_>]) -> Result<Vec<Vector>>;
    fn supported_input_types(&self) -> Vec<EmbedInputType>;
    fn name(&self) -> &str;
    fn as_any(&self) -> &dyn Any;
}
}

The embed() method returns a Vector (a struct wrapping Vec<f32>).

EmbedInput supports two modalities:

Token-Level Embedders

Late-interaction models such as ColBERT turn an input into one vector per token instead of one vector in total, and encode queries and documents differently. Such an embedder also implements TokenEmbedder and returns itself from Embedder::as_token_embedder (Issue #1349):

#![allow(unused)]
fn main() {
#[async_trait]
pub trait TokenEmbedder: Send + Sync + Debug {
    async fn embed_tokens(&self, inputs: &[EmbedInput<'_>], role: EmbedRole)
        -> Result<Vec<Vec<Vector>>>;
    fn token_dimension(&self) -> usize;
}

pub enum EmbedRole { Query, Document }
}

A multi-vector field embeds text only through this trait. An embedder without it (any single-vector model) is rejected rather than silently producing a one-token document.

Built-in Embedders

CandleBertEmbedder

Runs a BERT model locally using Hugging Face Candle. No API key required.

Feature flag: embeddings-candle

#![allow(unused)]
fn main() {
use laurus::CandleBertEmbedder;

// Downloads model on first run (~80MB)
let embedder = CandleBertEmbedder::new(
    "sentence-transformers/all-MiniLM-L6-v2"  // model name
)?;
// Output: 384-dimensional vector
}

It encodes like sentence-transformers, and its vectors match sentence-transformers’ within 1e-6 per element for all-MiniLM-L6-v2 and paraphrase-multilingual-MiniLM-L12-v2:

• An input is truncated to the model’s max_seq_length from sentence_bert_config.json (256 tokens for all-MiniLM-L6-v2, 128 for paraphrase-multilingual-MiniLM-L12-v2), special tokens included. A repository without that file falls back to the truncation length in tokenizer.json, then to the model’s max_position_embeddings. The padding configured in tokenizer.json is ignored.
• The vector is the mean of the token vectors, L2-normalized. This is done for every model, also for one whose sentence-transformers pipeline does not normalize (such as paraphrase-multilingual-MiniLM-L12-v2) or pools differently.

CandleBertEmbedder::with_options(model, CandleBertOptions::default().revision("<commit>")) pins the model to a commit.

Migration note (Issue #1340): earlier versions passed the attention mask to the model as token type ids and padded every input to the tokenizer.json length, so their vectors were off (for all-MiniLM-L6-v2, a cosine of only 0.61–0.70 with sentence-transformers’ on short texts). candle_bert vectors are now different, and long inputs keep up to max_seq_length tokens instead of 128 for all-MiniLM-L6-v2. Re-embed indexes built with candle_bert: otherwise new query vectors are compared with old document vectors.

Migration note (Issue #1355): earlier versions cached downloaded models under ~/.cache/huggingface/models--* instead of hf-hub’s own default ~/.cache/huggingface/hub/models--* (also ignoring HF_HUB_CACHE/XDG_CACHE_HOME). Models now download to hf-hub’s default location, shared with the Python huggingface_hub library. Existing downloads under the old path are not reused and can be deleted.

OpenAIEmbedder

Calls the OpenAI Embeddings API. Requires an API key.

Feature flag: embeddings-openai

#![allow(unused)]
fn main() {
use laurus::OpenAIEmbedder;

let embedder = OpenAIEmbedder::new(
    api_key,
    "text-embedding-3-small".to_string()
).await?;
// Output: 1536-dimensional vector
}

CandleClipEmbedder

Runs a CLIP model locally for multimodal (text + image) embeddings.

Feature flag: embeddings-multimodal

#![allow(unused)]
fn main() {
use laurus::CandleClipEmbedder;

let embedder = CandleClipEmbedder::new(
    "openai/clip-vit-base-patch32"
)?;
// Text or images → 512-dimensional vector
}

Migration note (Issue #1355): earlier versions cached downloaded models under ~/.cache/huggingface/models--* instead of hf-hub’s own default ~/.cache/huggingface/hub/models--* (also ignoring HF_HUB_CACHE/XDG_CACHE_HOME). Models now download to hf-hub’s default location, shared with the Python huggingface_hub library. Existing downloads under the old path are not reused and can be deleted.

CandleColbertEmbedder

Runs a BERT-based ColBERT checkpoint locally and produces one vector per token, for late-interaction rescoring over a multi-vector field. It is a token-level embedder only: embed() returns an error.

Feature flag: embeddings-candle

#![allow(unused)]
fn main() {
use laurus::{CandleColbertEmbedder, CandleColbertOptions};

// Uses the checkpoint's own settings (artifact.metadata).
let embedder = CandleColbertEmbedder::new("colbert-ir/colbertv2.0")?;

// Pin the model commit and override the lengths.
let embedder = CandleColbertEmbedder::with_options(
    "answerdotai/answerai-colbert-small-v1",
    CandleColbertOptions::default()
        .revision("934fa8bb4ce2284f4c2baa232d81aca4d076fa5e")
        .doc_maxlen(300),
)?;
}

It encodes like colbert-ai, the reference implementation:

query:    [CLS] [unused0] w1 … wn [SEP] [MASK] … [MASK]   exactly query_maxlen tokens
document: [CLS] [unused1] w1 … wn [SEP]                   at most doc_maxlen tokens

• The query’s [MASK] padding is not attended to, but its vectors are kept: they expand the query.
• Document vectors of punctuation tokens are dropped.
• Every vector goes through the checkpoint’s linear projection and is L2-normalized.

The lengths, the markers and these switches come from the checkpoint’s artifact.metadata (colbert-ai’s defaults, 32 and 220, apply when the file is missing); CandleColbertOptions overrides the lengths. For both checkpoints above, its vectors match colbert-ai’s within 1e-6 per element.

Inference runs on the CPU in a blocking task, in batches of up to 32 inputs. As a guide, on an Apple M4 one query takes about 55 ms with colbertv2.0 and 18 ms with answerai-colbert-small-v1, and a document of about 110 tokens about 135 ms and 45 ms.

Pin revision to a commit: the write-ahead log keeps the text of documents that are not committed yet and embeds it again on recovery, and a changed model would produce different vectors. Only BERT-based checkpoints are supported (ModernBERT-based ones such as lightonai/GTE-ModernColBERT-v1 are not).

PrecomputedEmbedder

Use pre-computed vectors directly without any embedding computation. Useful when vectors are generated externally.

#![allow(unused)]
fn main() {
use laurus::PrecomputedEmbedder;

let embedder = PrecomputedEmbedder::new();  // no parameters needed
}

When using PrecomputedEmbedder, you provide vectors directly in documents instead of text for embedding:

#![allow(unused)]
fn main() {
let doc = Document::builder()
    .add_vector("embedding", vec![0.1, 0.2, 0.3, ...])
    .build();
}

PerFieldEmbedder

PerFieldEmbedder routes embedding requests to field-specific embedders:

graph LR
    PFE["PerFieldEmbedder"]
    PFE -->|"text_vec"| BERT["CandleBertEmbedder\n(384 dim)"]
    PFE -->|"image_vec"| CLIP["CandleClipEmbedder\n(512 dim)"]
    PFE -->|other fields| DEF["Default Embedder"]

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::PerFieldEmbedder;

let bert = Arc::new(CandleBertEmbedder::new("...")?);
let clip = Arc::new(CandleClipEmbedder::new("...")?);


let per_field = PerFieldEmbedder::new(bert.clone());
per_field.add_embedder("text_vec", bert.clone());
per_field.add_embedder("image_vec", clip.clone());

let engine = Engine::builder(storage, schema)
    .embedder(Arc::new(per_field))
    .build()
    .await?;
}

This is especially useful when:

• Different vector fields need different models (e.g., BERT for text, CLIP for images)
• Different fields have different vector dimensions
• You want to mix local and remote embedders

How Embeddings Are Used

At Index Time

When you add a text value to a vector field, the engine automatically embeds it:

#![allow(unused)]
fn main() {
let doc = Document::builder()
    .add_text("text_vec", "Rust is a systems programming language")
    .build();
engine.add_document("doc-1", doc).await?;
// The embedder converts the text to a vector before indexing
}

A multi-vector field whose embedder is a token-level one takes text the same way: the text is embedded into one vector per token, as a document.

At Search Time

When you search with text, the engine embeds the query text as well:

#![allow(unused)]
fn main() {
// Builder API
let request = VectorSearchRequestBuilder::new()
    .add_text("text_vec", "systems programming")
    .build();

// Query DSL
let request = vector_parser.parse(r#"text_vec:"systems programming""#).await?;
}

Both approaches embed the query text using the same embedder that was used at index time, ensuring consistent vector spaces.

A late-interaction rescore takes its query as text too (RescoreOptions::late_interaction_text); the multi-vector field’s token-level embedder embeds it as a query.

Feature Flags Summary

Each embedder requires a specific feature flag to be enabled in Cargo.toml:

The embeddings-all feature enables all embedding features at once. See Feature Flags for details.

Choosing an Embedder

Storage

Laurus uses a pluggable storage layer that abstracts how and where index data is persisted. All components — lexical index, vector index, and document log — share a single storage backend.

The Storage Trait

All backends implement the Storage trait:

#![allow(unused)]
fn main() {
pub trait Storage: Send + Sync + Debug {
    fn loading_mode(&self) -> LoadingMode;
    fn open_input(&self, name: &str) -> Result<Box<dyn StorageInput>>;
    fn create_output(&self, name: &str) -> Result<Box<dyn StorageOutput>>;
    fn file_exists(&self, name: &str) -> bool;
    fn delete_file(&self, name: &str) -> Result<()>;
    fn list_files(&self) -> Result<Vec<String>>;
    fn file_size(&self, name: &str) -> Result<u64>;
    // ... additional methods
}
}

This interface is file-oriented: all data (index segments, metadata, WAL entries, documents) is stored as named files accessed through streaming StorageInput / StorageOutput handles.

Storage Backends

MemoryStorage

All data lives in memory. Fast and simple, but not durable.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::Storage;
use laurus::storage::memory::MemoryStorage;

let storage: Arc<dyn Storage> = Arc::new(
    MemoryStorage::new(Default::default())
);
}

FileStorage

Standard file-system based persistence. Each key maps to a file on disk.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::Storage;
use laurus::storage::file::{FileStorage, FileStorageConfig};

let config = FileStorageConfig::new("/tmp/laurus-data");
let storage: Arc<dyn Storage> = Arc::new(FileStorage::new("/tmp/laurus-data", config)?);
}

FileStorage with Memory Mapping

FileStorage supports memory-mapped file access via the use_mmap configuration flag. When enabled, the OS manages paging between memory and disk; the lexical posting decoder (Issue #504) takes a zero-copy path through StorageInput::as_slice, handing PFOR-bit-packed blocks directly to bitpacking::decompress* instead of allocating an intermediate Vec<u8> and copying through Read.

Default is platform-specific:

• *Unix (Linux / macOS / BSD): true as of Issue #504. Set the LAURUS_NO_MMAP=1 environment variable when constructing the config (via FileStorageConfig::new) to fall back to buffered file I/O for debug sessions or hosts where mmap misbehaves.
• Windows: false as of Issue #508. Windows holds an exclusive lock on memory-mapped files (ERROR_USER_MAPPED_FILE, os error 1224) which prevents the writer from truncating / deleting a segment file while a reader still holds an mmap. The current segment-file lifecycle is incompatible with that lock. Set LAURUS_USE_MMAP=1 to opt in for read-only / read-mostly workloads where commit frequency is low. Full Windows mmap support is tracked in Issue #508.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::Storage;
use laurus::storage::file::{FileStorage, FileStorageConfig};

// mmap is on by default on Unix; on Windows it is off unless
// LAURUS_USE_MMAP=1 is set.
let config = FileStorageConfig::new("/tmp/laurus-data");
let storage: Arc<dyn Storage> = Arc::new(FileStorage::new("/tmp/laurus-data", config)?);

// Explicit opt-out without touching the env var (works on any OS).
let mut buffered_config = FileStorageConfig::new("/tmp/laurus-data");
buffered_config.use_mmap = false;

// Explicit opt-in (works on any OS, including Windows).
let mut mmap_config = FileStorageConfig::new("/tmp/laurus-data");
mmap_config.use_mmap = true;
}

StorageFactory

You can also create storage via configuration:

#![allow(unused)]
fn main() {
use laurus::storage::{StorageConfig, StorageFactory};
use laurus::storage::memory::MemoryStorageConfig;

let storage = StorageFactory::create(
    StorageConfig::Memory(MemoryStorageConfig::default())
)?;
}

PrefixedStorage

The engine uses PrefixedStorage to isolate components within a single storage backend:

graph TB
    E["Engine"]
    E --> P1["PrefixedStorage\nprefix = 'lexical/'"]
    E --> P2["PrefixedStorage\nprefix = 'vector/'"]
    E --> P3["PrefixedStorage\nprefix = 'documents/'"]
    P1 --> S["Storage Backend"]
    P2 --> S
    P3 --> S

When the lexical store writes a key segments/seg-001.dict, it is actually stored as lexical/segments/seg-001.dict in the underlying backend. This ensures no key collisions between components.

You do not need to create PrefixedStorage yourself — the EngineBuilder handles this automatically.

ColumnStorage

In addition to the primary storage backends, Laurus provides a ColumnStorage layer for fast field-level access. This is used internally for operations like faceting, sorting, and aggregation, where accessing individual field values without deserializing entire documents is important.

ColumnValue

ColumnValue represents a single stored column value:

ColumnStorage is managed internally by the Engine – you do not need to interact with it directly.

Choosing a Backend

Recommendations

• Development / Testing: Use MemoryStorage for fast iteration without file cleanup
• Production (general): Use FileStorage for reliable persistence
• Production (large scale): Use FileStorage with use_mmap = true when you have large indexes and want to leverage OS page cache

Next Steps

• Learn how the lexical index works: Lexical Indexing
• Learn how the vector index works: Vector Indexing

Indexing

This section explains how Laurus stores and organizes data internally. Understanding the indexing layer will help you choose the right field types and tune performance.

Topics

Lexical Indexing

How text, numeric, and geographic fields are indexed using an inverted index. Covers:

• The inverted index structure (term dictionary, posting lists)
• BKD trees for numeric range queries
• Segment files and their formats
• BM25 scoring

Vector Indexing

How vector fields are indexed for approximate nearest neighbor search. Covers:

• Index types: Flat, HNSW, IVF
• Parameter tuning (m, ef_construction, n_clusters, n_probe)
• Distance metrics (Cosine, Euclidean, DotProduct)
• Quantization (SQ8, PQ)

Lexical Indexing

Lexical indexing powers keyword-based search. When a document’s text field is indexed, Laurus builds an inverted index — a data structure that maps terms to the documents containing them.

How Lexical Indexing Works

sequenceDiagram
    participant Doc as Document
    participant Analyzer
    participant Writer as IndexWriter
    participant Seg as Segment

    Doc->>Analyzer: "The quick brown fox"
    Analyzer->>Analyzer: Tokenize + Filter
    Analyzer-->>Writer: ["quick", "brown", "fox"]
    Writer->>Writer: Buffer in memory
    Writer->>Seg: Flush to segment when the buffer fills, and on commit()

Step by Step

1. Analyze: The text passes through the configured analyzer (tokenizer + filters), producing a stream of normalized terms
2. Buffer: Each term goes straight into in-memory posting lists, organized by field; the write buffer keeps only what the flush still needs from the document — its stored values, field lengths, and numeric points
3. Flush and commit: When the buffer reaches max_buffered_docs (default 10,000) or max_buffer_memory (default 64 MiB) of estimated memory (the buffered documents plus the posting lists), it is flushed to a new, not yet visible segment; commit() flushes what remains and publishes those segments together

The Inverted Index

An inverted index is essentially a map from terms to document lists:

graph LR
    subgraph "Term Dictionary"
        T1["'brown'"]
        T2["'fox'"]
        T3["'quick'"]
        T4["'rust'"]
    end

    subgraph "Posting Lists"
        P1["doc_1, doc_3"]
        P2["doc_1"]
        P3["doc_1, doc_2"]
        P4["doc_2, doc_3"]
    end

    T1 --> P1
    T2 --> P2
    T3 --> P3
    T4 --> P4

Posting List Contents

Each entry in a posting list contains:

On-Disk Posting Layout

Posting lists are stored in a structure-of-arrays layout with each field written as its own contiguous section. Document IDs and term frequencies are encoded in fixed-size 128-int blocks using bit-packing (Frame-of-Reference plus sorted-delta for doc IDs), with any partial trailing block falling back to varint. This matches the on-disk format used by Tantivy and Lucene 9 and yields fast SIMD-accelerated decoding through the bitpacking crate.

[term, total_frequency, doc_frequency, posting_count N, any_positions, any_weights]
[Skip levels              — v2 onward: num_levels + per-level (len + u32 doc_ids)]
[Section 1: doc_ids       — N/128 bit-packed blocks + varint tail]
[Section 2: frequencies   — N/128 bit-packed blocks + varint tail]
[Section 3: weights       — N raw f32 values (only when any_weights == 1)]
[Section 4: positions     — per-posting flag + varint deltas (only if any)]

Two header flags gate the two optional sections. any_positions has always gated Section 4; any_weights was added in v3 and gates Section 3 the same way. Because every posting Laurus writes carries the default weight of 1.0, Section 3 is absent from every segment the writer produces, and a posting list is 4N - 1 bytes smaller than its v2 equivalent — four bytes recovered per posting, less the one flag byte per list. At a billion postings that is roughly 4 GB that no longer has to be written, read, or cached.

The saving does not make the format lossy: a single weight other than 1.0 sets the flag, restores Section 3, and every value round-trips exactly.

Per-segment doc IDs must fit in u32. Encoding a value beyond u32::MAX fails fast with a clear error to prevent silent corruption of the bit-packed segment.

The decoder exposes a SoA-native fast path (PostingList::decode_soa) that produces parallel Vec<u32> slices for doc_id and frequency directly from disk without the intermediate Vec<Posting> reassembly. The query iterator stores those slices, so next() advances a single u32 cursor over a dense slice instead of striding across a 40-byte Posting struct.

Multi-Level Skip Table

A posting list of N ≥ 8 postings carries a Lucene-90-style multi-level skip table immediately after the header (introduced in the v2 format for skip_to performance on seek-heavy workloads, and unchanged in v3). Each level stores the “last doc id” of a fixed-stride window over doc_ids; level 0 has one entry per SKIP_INTERVAL = 8 postings, level 1 covers 8² postings, and so on until the top level collapses to a single entry.

PostingIterator::skip_to(target) walks the table top-down: at each level a single partition_point narrows the search window by a factor of SKIP_INTERVAL, descends one level, and finishes with a linear scan over the bottom-level window of at most 8 postings. Total work is O(log_8 N + SKIP_INTERVAL) per call — for N = 1 M that is ~25 comparisons, versus the linear O(N / block_size) walk the single-level block_cache paid before.

The table is co-located with the posting list rather than stored in a separate file, matching Lucene 9 / Tantivy. Older segments remain readable: v1 segments (without the on-disk skip table) have it rebuilt from doc_ids at load time, and v2 segments are read with their unconditional weights section intact. Which decoder applies is determined by the sibling .dict file’s version — the only segment-level file carrying a magic and a version number — so the posting payload itself never has to be probed.

A table read from disk must be exactly the one the writer builds from the decoded doc_ids. The decoder compares the two in place and rejects any other table as a corrupt segment: skip_to relies on that shape to keep its strides in range, and a wrong entry would skip matching documents.

Term Dictionary Storage Layers

The dictionary uses a two-layer representation that decouples on-disk compactness from in-memory query speed:

• Disk layer — the .dict file uses a Lucene BlockTreeTermsWriter-style block-tree layout (magic LTDD, schema v3). This produces a compact file: at 100k unique 5-10 byte terms a .dict is ~12.5 bytes / term, roughly 70 % smaller than the prior parallel-array on-disk format.
• In-memory query layer — at build / load time the dictionary populates an AHashMap<term, ordinal> index, an ordinal-indexed sorted Vec<String>, and a single-copy Arc<[TermInfo]>. get / iter / find_prefix / find_range operate exclusively on these structures, so per-query cost matches the prior parallel-array implementation (no per-call FST traversal or in-block linear scan).

The disk-format scratch (FST + BlockSection bytes) is retained only so [BlockTermDictionary::write_to_storage] can re-serialise the dictionary on segment merge without re-encoding from scratch.

[Header                ]  magic "LTDD" + version
[FstSection            ]  fst::Map<u64> keyed by each block's last term,
                          valued by that block's start byte offset
[BlockSection          ]  concatenated 128-term blocks, each containing
                          front-coded term bytes, a bit-packed
                          fixed-size TermInfo block, and a
                          variable-length per-term Block-Max-WAND
                          metadata array
[Footer                ]  total term count + block count
[Checksum footer       ]  CRC-32 of all of the above + magic "LCRC"

Lookup walks the FST once (O(|term|)) to identify the block whose last term is ≥ target, then performs a linear front-coded scan inside the block (≤ 128 entries). Iteration walks the BlockSection sequentially without consulting the FST, decoding each block’s front-coded prefix once and reusing the buffer across terms. Prefix scans combine the two: FST seek to the first matching block, then sequential block walk until the prefix no longer matches.

Compared to a flat per-term FST the block-head FST is one to two orders of magnitude smaller, and the front-coded block bytes plus bit-packed TermInfo typically cut the on-disk size by 50-80 % at production scale.

Numeric and Date Fields

Integer, float, and datetime fields are indexed using a BKD tree — a space-partitioning data structure optimized for range queries:

graph TB
    Root["BKD Root"]
    Root --> L["values < 50"]
    Root --> R["values >= 50"]
    L --> LL["values < 25"]
    L --> LR["25 <= values < 50"]
    R --> RL["50 <= values < 75"]
    R --> RR["values >= 75"]

BKD trees allow efficient evaluation of range queries like price:[10 TO 100] or date:[2024-01-01 TO 2024-12-31].

Geo Fields

Geographic fields come in two flavours, both backed by the same multi-dimensional BKD-Tree primitive:

Geo3d is the right choice when altitude is a first-class dimension — drones, satellites, indoor 3D positioning, or anything else that a 2D Geo field would lose or distort near the poles. See 3D Geographic Search (ECEF) for the coordinate system, WGS84 conversion helpers, and DSL syntax.

Segments

The lexical index is organized into segments. Each segment is an immutable, self-contained mini-index:

graph TB
    LI["Lexical Index"]
    LI --> S1["Segment 0"]
    LI --> S2["Segment 1"]
    LI --> S3["Segment 2"]

    S1 --- F1[".dict (terms)"]
    S1 --- F2[".post (postings)"]
    S1 --- F3[".bkd (numerics)"]
    S1 --- F4[".docs (doc store)"]
    S1 --- F5[".dv (doc values)"]
    S1 --- F6[".meta (metadata)"]
    S1 --- F7[".norms (field-length norms)"]
    S1 --- F8[".ids (doc ids)"]

Segment Lifecycle

1. Create: A new segment is flushed each time the writer’s buffer reaches max_buffered_docs (default 10,000) or max_buffer_memory (default 64 MiB), and once more by commit() for what remains; commit() publishes them all at once, so one commit can add several segments. Both thresholds are set on the index config (LexicalIndexConfig::builder().max_buffered_docs(..) / .max_buffer_memory(..))
2. Search: All segments are searched in parallel and results are merged
3. Merge: After each commit(), once the count exceeds max_segments, an auto-merge merges runs of up to merge_factor segments of similar size (at most 8 times apart), smallest first, keeping the segment count bounded; a manual optimize() force-merges everything into one segment. Merging only similar sizes keeps a large segment from being rewritten every time it absorbs a small one, so each byte is rewritten a logarithmic number of times as the index grows. One commit’s auto-merge keeps merging until it has removed merge_factor - 1 segments, the count is back at max_segments, or no run of similar size is left. The count can therefore stay above max_segments while small segments wait for partners of similar size: as long as each commit publishes fewer than merge_factor segments, it stays within max(max_segments, 1 + log_8(index size / smallest segment)). A large segment with many deleted documents is rewritten, and its deleted documents reclaimed, only when it merges with segments of similar size or on optimize(). max_merged_segment_bytes (default: unbounded) caps the combined estimated size of the segments a single merge will take, so a segment at or over the cap is left unmerged and the count can stabilize above max_segments once the survivors no longer fit together
4. Delete: When a document is deleted, its ID is added to a deletion bitmap rather than physically removed (see Deletions & Compaction)

Resolving Document IDs

Every put_document first looks up the earlier versions of its _id so it can replace them. The committed segments are checked through a Bloom filter over every committed _id key, which the index keeps in memory and shares among its readers:

• An _id the filter has never seen is in no segment, so a put of a new ID costs the same however many segments the index holds.
• A “maybe” (an _id that exists, or about 1% of new IDs at most) probes each segment’s term dictionary to find the exact copies.
• The filter is built on the first lookup after the index is opened, and each lookup after a commit adds only the new segments’ keys. When it fills up, it is rebuilt from the current segments, which also drops the IDs that merges removed.
• It takes 2 bytes per committed document right after a rebuild, down to about 1.3 bytes just before the next one.

Reading After a Commit

A commit’s documents become visible through a new reader, built by the next search or put. Building it does not read again the segments the commit left alone. The index keeps the per-segment readers of the last reader it built, and the next reader takes them over:

• A segment whose metadata and deletions did not change keeps its reader. It keeps everything that reader has loaded: the term dictionary, norms, doc values, stored documents, BKD trees and the posting cache.
• A segment that only gained deletions gets a new reader. The new reader reads the segment’s .delmap and shares everything else with the old one.
• A new segment, such as one the commit flushed or a merge wrote, is opened. Its parts load the first time they are used.

The first search or _id lookup after a commit therefore reads only what the commit changed, however many documents the other segments hold. A reader built before the commit keeps the segment readers it was built with, so it still sees its own snapshot. A merge or a field rebuild releases the readers of the segments it replaced as soon as it publishes, and closing the index releases them all.

What the segments have loaded stays in memory from one commit to the next. Before, it was freed at each commit and loaded again by the next search or update. A search loads every segment’s term dictionary anyway, so the peak does not change. The memory is no longer released between a commit and the next read.

BM25 Scoring

Laurus uses the BM25 algorithm to score lexical search results. BM25 considers:

• Term Frequency (TF): how often the term appears in the document (more = better, with diminishing returns)
• Inverse Document Frequency (IDF): how rare the term is across all documents (rarer = more important)
• Field Length Normalization: shorter fields are boosted relative to longer ones

The formula:

score(q, d) = IDF(q) * (TF(q, d) * (k1 + 1)) / (TF(q, d) + k1 * (1 - b + b * |d| / avgdl))

Where k1 = 1.2 and b = 0.75 are the default tuning parameters.

SIMD Optimization

Vector distance calculations leverage SIMD (Single Instruction, Multiple Data) instructions when available, providing significant speedups for similarity computations in vector search.

Code Example

use std::sync::Arc;
use laurus::{Document, Engine, Schema};
use laurus::lexical::TextOption;
use laurus::lexical::core::field::IntegerOption;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_integer_field("year", IntegerOption::default())
        .build();

    let engine = Engine::builder(storage, schema).build().await?;

    // Index documents
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming")
        .add_text("body", "Rust is a systems programming language.")
        .add_integer("year", 2024)
        .build()
    ).await?;

    // Commit to flush segments to storage
    engine.commit().await?;

    Ok(())
}

Next Steps

• Learn how vector indexes work: Vector Indexing
• Run queries against the lexical index: Lexical Search

Vector Indexing

Vector indexing powers similarity-based search. When a document’s vector field is indexed, Laurus stores the embedding vector in a specialized index structure that enables fast approximate nearest neighbor (ANN) retrieval.

How Vector Indexing Works

sequenceDiagram
    participant Doc as Document
    participant Embedder
    participant Normalize as Normalizer
    participant Index as Vector Index

    Doc->>Embedder: "Rust is a systems language"
    Embedder-->>Normalize: [0.12, -0.45, 0.78, ...]
    Normalize->>Normalize: L2 normalize
    Normalize-->>Index: [0.14, -0.52, 0.90, ...]
    Index->>Index: Insert into index structure

Step by Step

1. Embed: The text (or image) is converted to a vector by the configured embedder
2. Normalize: The vector is L2-normalized only for the Cosine metric (a magnitude-invariant metric). Euclidean, DotProduct, and Manhattan fields keep their original vectors so their distances are preserved
3. Index: The vector is inserted into the configured index structure (Flat, HNSW, or IVF)
4. Commit: On commit(), the index is flushed to persistent storage

Index Types

Laurus supports three vector index types, each with different performance characteristics:

Comparison

Flat Index

The simplest index. Compares the query vector against every stored vector (brute-force).

#![allow(unused)]
fn main() {
use laurus::vector::FlatOption;
use laurus::vector::core::distance::DistanceMetric;

let opt = FlatOption {
    dimension: 384,
    distance: DistanceMetric::Cosine,
    ..Default::default()
};
}

• Pros: 100% recall (exact results), simple, low memory
• Cons: Slow for large datasets (linear scan)
• Use when: You have fewer than ~10,000 vectors, or you need exact results

HNSW Index

Hierarchical Navigable Small World graph. The default and most commonly used index type.

graph TB
    subgraph "Layer 2 (sparse)"
        A2["A"] --- C2["C"]
    end

    subgraph "Layer 1 (medium)"
        A1["A"] --- B1["B"]
        A1 --- C1["C"]
        B1 --- D1["D"]
        C1 --- D1
    end

    subgraph "Layer 0 (dense - all vectors)"
        A0["A"] --- B0["B"]
        A0 --- C0["C"]
        B0 --- D0["D"]
        B0 --- E0["E"]
        C0 --- D0
        C0 --- F0["F"]
        D0 --- E0
        E0 --- F0
    end

    A2 -.->|"entry point"| A1
    A1 -.-> A0
    C2 -.-> C1
    C1 -.-> C0
    B1 -.-> B0
    D1 -.-> D0

The HNSW algorithm searches from the top (sparse) layer down to the bottom (dense) layer, narrowing the search space at each level.

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::distance::DistanceMetric;

let opt = HnswOption {
    dimension: 384,
    distance: DistanceMetric::Cosine,
    m: 16,                  // max connections per node per layer
    ef_construction: 200,   // search width during index building
    ..Default::default()
};
}

HNSW Parameters

Tuning tips:

• Increase m (e.g., 32 or 64) for higher recall at the cost of memory
• Increase ef_construction (e.g., 400) for better index quality at the cost of build time
• At search time, the ef_search parameter (set in the search request) controls the search width

IVF Index

Inverted File Index. Partitions vectors into clusters, then only searches relevant clusters.

graph TB
    Q["Query Vector"]
    Q --> C1["Cluster 1\n(centroid)"]
    Q --> C2["Cluster 2\n(centroid)"]

    C1 --> V1["vec_3"]
    C1 --> V2["vec_7"]
    C1 --> V3["vec_12"]

    C2 --> V4["vec_1"]
    C2 --> V5["vec_9"]
    C2 --> V6["vec_15"]

    style C1 fill:#f9f,stroke:#333
    style C2 fill:#f9f,stroke:#333

#![allow(unused)]
fn main() {
use laurus::vector::IvfOption;
use laurus::vector::core::distance::DistanceMetric;

let opt = IvfOption {
    dimension: 384,
    distance: DistanceMetric::Cosine,
    n_clusters: 100,   // number of clusters
    n_probe: 10,       // clusters to search at query time
    ..Default::default()
};
}

IVF Parameters

Tuning tips:

• Set n_clusters to roughly sqrt(n) where n is the number of vectors
• Set n_probe to 5-20% of n_clusters for a good recall/speed trade-off
• IVF requires a training phase — initial indexing may be slower

Distance Metrics

#![allow(unused)]
fn main() {
use laurus::vector::core::distance::DistanceMetric;

let metric = DistanceMetric::Cosine;      // Default for text
let metric = DistanceMetric::Euclidean;    // For spatial data
let metric = DistanceMetric::Manhattan;    // L1 distance
let metric = DistanceMetric::DotProduct;   // For pre-normalized vectors
let metric = DistanceMetric::Angular;      // Angular distance
}

Note: Vectors are automatically L2-normalized before indexing only for the Cosine metric. Normalizing is magnitude-invariant, so it is safe for Cosine (and additionally tightens the int8 quantization range), but it would change the distances of magnitude-sensitive metrics — so Euclidean, DotProduct, and Manhattan fields are stored without normalization. Lower distance = more similar.

Migration note: non-Cosine fields created before this behavior was corrected have L2-normalized vectors on disk (the original magnitudes are not recoverable). Rebuild such an index to get correct distances.

Quantization

Vectors are stored on disk as 8-bit scalar-quantized integers (Issue #481 Stage 1). Compared to the previous 32-bit float storage this is ~4x smaller with negligible recall loss in practice (Recall@10 remains ≥ 0.95 against the f32 ground truth — see the recall test at laurus/tests/vector_recall_test.rs).

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::quantization::QuantizationMethod;

// `quantizer` defaults to `Scalar8Bit`; the explicit form below is
// equivalent to `HnswOption { dimension: 384, ..Default::default() }`.
let opt = HnswOption {
    dimension: 384,
    quantizer: QuantizationMethod::Scalar8Bit,
    ..Default::default()
};
}

Breaking change (Issue #481 Stage 1): the quantizer field is no longer Option<QuantizationMethod>; quantization is mandatory and defaults to Scalar8Bit. There is no longer an unquantized (f32) on-disk format. Existing pre-Stage-1 vector indexes are intentionally not readable by this version — rebuild the index from source data.

How Scalar8Bit works

• Each segment trains a single global (offset, scale) pair from its f32 vectors at flush time (offset = min, scale = (max - min) / 255).
• Each f32 element is encoded as u8 = clamp(round((v - offset) / scale), 0, 255).
• Per-vector metadata (sum_q: u32, norm_q: f32) is precomputed and persisted alongside the int8 payload so the cosine search hot loop collapses to one int8 SIMD multiply-accumulate plus three scalar corrections — no per-element dequantization at search time.
• Segment files start with the LVS1 magic + a 16-byte header so the reader can detect the format at load time. The header carries a version (a feature ladder: v2 = ordinal-encoded HNSW graph block, v3 = per-segment field-name dictionary) and, from v3 on, the dictionary itself — records then reference field names by a 16-bit id instead of repeating the full name inline.

Two-stage rerank (Issue #481 Stage 2)

Stage 1 stores vectors as int8 only. The graph search runs entirely against int8 distances, which is fast but introduces a small quantization error. Stage 2 adds an optional per-field f32 sidecar so the searcher can rescore the top candidates against the original full-precision vectors:

1. The HNSW int8 graph search returns up to ef_search candidates ranked by quantized cosine distance.
2. The top top_k * rerank_factor candidates are rescored against the f32 vectors loaded from the LRS1 sidecar (*.hnsw.f32).
3. The new ranking is truncated to top_k and returned.

Since #932 the same sidecar mechanism serves Flat and IVF too (the shared RerankPipeline, #650); the description below uses HNSW, the original Stage-2 host. Stage 2 is opt-in per field via HnswOption.rerank_storage:

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::rerank::RerankStorageKind;

let opt = HnswOption {
    rerank_storage: Some(RerankStorageKind::F32),
    ..HnswOption::default()
};
}

Queries pass the rerank factor through VectorIndexQuery::rerank_factor (low-level), SearchRequestBuilder::vector_rerank_factor (engine), or the gRPC / JSON VectorParams.rerank_factor field.

Fields without rerank_storage enabled silently fall back to the Stage 1 int8 ranking even when rerank_factor is set — there is no f32 information to recover from a Stage 1 segment.

LRS1 rerank sidecar

The sidecar is a separate file written next to the LVS1 segment when rerank_storage is enabled:

offset  size  field
------  ----  -------------------------------------------
     0     4  magic         ASCII "LRS1"
     4     2  version       u16 LE  (current = 1)
     6     2  storage_kind  u16 LE  (1 = F32; 0 reserved; 2.. future)
     8     8  reserved      zero-padded
    16     4  dim           u32 LE
    20     4  vector_count  u32 LE
    24     -  payload       vector_count * dim * bytes_per_element
   end     8  footer        magic "LRC1" u32 LE + CRC-32 u32 LE
                             over header + payload

Vectors are written in the same (doc_id, field_name) order as the matching LVS1 segment, so a (sidecar position) → (LVS1 position) mapping is the identity. The HNSW reader loads the sidecar into a RerankStoragePool at init time when the storage loading mode is Eager; Lazy mode skips the sidecar to honor its memory-savings promise (Stage 2 segments opened in Lazy mode silently degrade to Stage 1).

The sidecar is enabled per field by rerank_storage in the schema’s HNSW options, and that setting is honored on every write path: direct commits, the active write segment, and segment merges all re-emit the sidecar for the segment they produce. A merged segment’s sidecar is rebuilt from the source segments’ original f32 sidecars (not from the int8-dequantized segment data), so a merge stays lossless for the rerank vectors instead of compounding one round of quantization error per merge.

New sidecars end with an 8-byte CRC-32 footer over the header and payload that is verified whenever the sidecar is read (both the searcher load and the writer reload), so silent on-disk corruption is rejected instead of skewing rerank scores. Because the header fully determines the content length, the footer is detected by the bytes remaining after the payload — sidecars written before the footer was introduced have zero trailing bytes and still load, with verification skipped.

Bounded allocations from on-disk header counts

Every vector segment header carries element counts (num_vectors, n_clusters, the HNSW graph’s node_count / layer_count / neighbor_count) and per-record byte lengths (the v1/v2 inline field_name_len, PQ codes) that the reader uses to size Vec / HashMap capacities and read buffers. The HNSW CRC footer is verified before the structural parse, but legacy footer-less segments — and the writer reload paths, which run no footer verification — reach those counts unverified. A single flipped byte could otherwise turn a small count into a multi-GiB allocation request that aborts the process through handle_alloc_error (out of memory) before any corruption check runs.

To prevent that, the HNSW, Flat, and IVF loaders bound each such allocation against ground truth — the true file length from StorageInput::size(). Because the bytes a count or buffer describes must physically exist in the file, a header that declares more elements or bytes than the file can hold is rejected as a corrupted segment before the allocation. This generalizes the rerank sidecar’s file-size bound to the main segment readers and covers legacy footer-less segments, so a corrupt or hostile header surfaces a clean “corrupted segment” error rather than an OOM abort.

Recall vs speed trade-off

rerank_factor lets you exchange a small per-query rerank cost (~top_k * rerank_factor exact-distance calls — a few µs at dim 128) for higher Recall@10. The gain depends on the corpus and the graph search budget (ef_search):

• Real clustered embedding data (text-embedding-3, BERT, etc.) reaches Recall@10 ≥ 0.99 at low ef_search; rerank polishes the ranking with negligible latency overhead.
• Synthetic random unit-norm data (the worst case for HNSW recall recovery) needs a higher ef_search for the int8 graph to visit enough true top-10 candidates; rerank then re-orders the visited set but cannot retrieve candidates the graph never reached.

The recall acceptance is split into two CI gates so the rerank kernel and the full HNSW pipeline can fail independently:

• stage2_brute_force_rerank_recall_at_10_meets_kernel_gate asserts Recall@10 ≥ 0.99. Bypasses the HNSW graph entirely (brute-force int8 over the corpus, widen to top_k * rerank_factor, rescore with f32) so any miss is a rerank-kernel regression.
• hnsw_quantized_recall_at_10_with_rerank_meets_stage2_recall_gate asserts Recall@10 ≥ 0.98. Adds the HNSW graph-construction non-determinism that an f32 HNSW baseline would also contribute; the looser gate matches the observed run-to-run variance band on this synthetic adversarial distribution. Real clustered embedding data and a stronger HNSW config (m=32, ef_construction=500) reach ≥ 0.99 on this path too — see the diagnostic sweep below.

The companion stage2_recall_sweep_diagnostic (opt-in via LAURUS_STAGE2_SWEEP=1) sweeps (ef_search, rerank_factor) across three corpus / query distributions and two HNSW configs so production deployments can calibrate the budget for their actual embedding distribution.

Real-data validation (Issue #498)

A third opt-in CI gate validates Stage 2 against a real ANN benchmark dataset (SIFT1M from TEXMEX) so the synthetic-data gates above do not become the only signal:

• hnsw_quantized_recall_at_10_with_rerank_on_sift_meets_stage2_real_data_recall_gate asserts Recall@10 ≥ 0.99 on a 50 000-vector SIFT1M subsample at (m=16, ef_construction=200, ef_search=200, rerank_factor=5).
• The companion bench bench_hnsw_graph_search_rerank_real_data (in laurus/benches/vector_search_bench.rs) measures end-to-end Stage 2 latency on the same fixture. The accompanying example laurus/examples/sift_rerank_probe.rs runs a full (ef_search × rerank_factor × HNSW config) sweep with (Recall, latency) per cell so operators can pick the operating point for their own data shape.

Both are gated on LAURUS_REAL_BENCHMARK=1 AND the presence of the SIFT1M .fvecs files under .cache/sift/sift/. Default CI runs are unchanged. To enable locally:

./scripts/fetch-sift.sh --large   # ~478MB
LAURUS_REAL_BENCHMARK=1 cargo test --release \
    --test vector_recall_test \
    hnsw_quantized_recall_at_10_with_rerank_on_sift_meets_stage2_real_data_recall_gate \
    -- --nocapture
LAURUS_REAL_BENCHMARK=1 cargo bench --bench vector_search_bench \
    -- "HNSW Graph Search Rerank Real"

The Issue #481 wording originally asked for “≥ 3× speedup vs the pre-Stage-1 f32 baseline.” Cross-branch Criterion measurements on SIFT1M-50k (median over 30 samples, same (m, ef_construction, ef_search)) gave 625 µs/query on the pre-Stage-1 f32 HNSW path versus 323 µs/query on the Stage 2 int8 + rerank path — a 1.94× speedup. Issue #498 reduces the real-data gate to ≥ 1.5× accordingly, with the recall side held at the original 0.99. The gap between the original 3× target and the measured 1.94× comes from rerank only re-ordering candidates the int8 graph traversal already visited — a lower ef_search does not pay back through rerank when the candidate set itself becomes too narrow, which a follow-up could address by widening the graph search budget independently of ef_search (the Lucene 99 pattern).

Product Quantization with rerank (Issue #481 Stage 3)

Stage 3 adds an opt-in Product Quantization path for the HNSW index. Each segment trains a per-field codebook of M sub-vectors × K = 256 centroids using Lloyd k-means with k-means++ initialisation, and stores every vector as M bytes (one centroid index per sub-vector). The search hot loop replaces the int8 SIMD kernel with asymmetric distance computation (ADC): per-query the searcher builds an M × K look-up table of squared distances between the query’s sub-vectors and the codebook entries, then scores each candidate as Σ_m lut[m][codes[m]] — M table lookups + M − 1 adds per candidate.

PQ is enabled per field via HnswOption.quantizer:

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::quantization::QuantizationMethod;
use laurus::vector::core::rerank::RerankStorageKind;

let opt = HnswOption {
    dimension: 128,
    quantizer: QuantizationMethod::ProductQuantization { subvector_count: 32 },
    // PQ-only Recall@10 caps out around 0.78-0.92 on SIFT1M, so
    // production deployments should pair PQ with the LRS1 rerank
    // sidecar — same Stage 2 mechanism, just driven by PQ
    // candidate generation instead of int8.
    rerank_storage: Some(RerankStorageKind::F32),
    ..Default::default()
};
}

subvector_count must divide dimension. Common choices for dim = 128: M ∈ {8, 16, 32} (sub_dim 16 / 8 / 4). Larger M trades on-disk compression for higher recall — Issue #481 Stage 3 ships only the 8-bit (K = 256) variant; the on-disk format reserves a 4-bit (K = 16) slot for a future PR.

Minimum training size: PQ trains K = 256 k-means centroids per sub-quantizer, so a segment with fewer than 256 vectors cannot train a meaningful codebook. Such segments are written as Scalar8Bit instead (the LVS1 header is self-describing, so readers dispatch on the stored kind transparently); once the segment grows — or is merged into a larger one — the next write trains PQ as configured. The subvector_count-divides-dimension validation still applies regardless of segment size. Neither the fallback nor the minimum applies when a shared codebook is configured: nothing is trained per segment in that case, so even tiny per-commit segments stay on PQ.

Shared PQ codebook (Issue #631)

By default every segment write trains its own codebook from scratch — a multi-second k-means cost that recurs on every commit and every merge under the segment-per-commit layout (#634/#889). Following the FAISS / Lucene99 precedent, a codebook can instead be trained once on a representative sample and reused by every subsequent segment write, reducing the encode step to a table lookup (measured ~91-92% faster than inline training at 2 048-8 192 vectors).

Enable it by naming a codebook file on the field and training it:

[fields.embedding.Hnsw]
dimension = 128
pq_codebook_path = "embedding.pqcb"

[fields.embedding.Hnsw.quantizer.ProductQuantization]
subvector_count = 32

laurus train pq-codebook --field embedding --input vectors.jsonl --update-schema

(or --from-index in place of --input to sample vectors already committed to the index; or fold training into index creation with laurus create index --train-pq-codebook <jsonl>, which removes the train-before-first-commit ordering hazard entirely — both Issue #920), or programmatically via Engine::train_pq_codebook (engine.train_pq_codebook("embedding", &vectors, None); pair with engine.sample_committed_vectors("embedding", Some(n)) for the from-index flow). Training is CPU-bound and synchronous; thousands of representative vectors are enough — the full corpus is not required.

Semantics:

• The segment format is unchanged. The shared codebook is still embedded inline in each segment header, so shared-codebook and inline-trained segments coexist with no migration, and the codes are byte-identical to what inline training on the same sample would produce.
• The codebook file is resolved once, at index open. Retraining while an engine is open does not hot-swap the codebook mid-session (intentional — a codebook must not change between the commits of one session); reopen the engine to pick up a new file.
• Failure policy — no silent fallback. A configured but not-yet-trained pq_codebook_path is lenient at open (so a schema can be created before training), but a commit that needs to encode hard-errors with the training command to run. A present but corrupt or geometry-mismatched file errors at open. Laurus never silently falls back to per-segment training — an invisible regression to the multi-second training cost would defeat the feature.
• Cosine-metric fields L2-normalize the training sample the same way the writer normalizes indexed vectors, so the codebook basis always matches (the #794 trap).
• FastScan fields are supported too (Issue #920, pq-fastscan feature): a ProductQuantizationFastScan field trains and encodes against a k=16 shared codebook through the exact same commands and file format — the .pqcb header stores k verbatim, so the two variants are distinguished by the stored value, and a k-mismatched codebook (e.g. a k=256 file configured on a FastScan field) fails the commit loudly with both values named.

On-disk format

PQ segments use the same LVS1 header as Scalar8Bit (quant_kind = 2) and carry the codebook inside the per-segment metadata block:

[ Fixed header           16 bytes ]
[ PQ params               8 bytes ]    m / k / sub_dim / padding (u16 × 4)
[ Codebook                m × k × sub_dim × 4 bytes ]
[ Per-vector codes        num_vectors × m bytes ]

For dim = 128, M = 32, K = 256: codebook = 32 × 256 × 4 × 4 = 131 072 bytes (128 KB) per segment plus 32 bytes per vector.

Recall and speed gates (Issue #481 Stage 3)

• Kernel-level test — synthetic 5 000-vector / dim 128 / 100 queries at (m=16, ef_construction=200, ef_search=200, rerank_factor=10, M=32): hnsw_pq_rerank_recall_at_10_meets_stage3_recall_gate asserts Recall@10 ≥ 0.95. Measured 0.9660.
• Real-data test — SIFT1M-50k subsample (opt-in via LAURUS_REAL_BENCHMARK=1, same config): hnsw_pq_rerank_recall_at_10_on_sift_meets_stage3_real_data_recall_gate asserts Recall@10 ≥ 0.95. Measured 0.9965.
• Real-data speed bench — bench_hnsw_graph_search_pq_rerank_real_data (opt-in, Criterion). Cross-branch measurement at the same SIFT1M-50k config: pre-Stage-1 f32 HNSW = 625.21 µs/query (PR #500); Stage 3 PQ + rerank = 299.54 µs/query = 2.09× speedup.

Issue #481 originally asked for ≥ 5× speedup at Recall ≥ 0.95. The PR established that target is not reachable on SIFT1M with the current implementation — both Stage 2 (#500) and this Stage 3 PR have measured speedups in the 1.9-2.1× band because rerank dominates the wall-clock once the candidate set is wide enough to recover recall. The gate was reduced to ≥ 1.5× accordingly. A follow-up could pursue the Lucene 99 pattern (independent graph search budget) and / or a 4-bit PQ variant to close the gap.

PQ → SQ → f32 three-stage rerank (Issue #673)

Stage 3’s rerank widens the PQ ADC candidate set to top_k * rerank_factor and rescores only that leading slice against the exact f32 sidecar — anything the graph traversal found beyond that budget is discarded. Setting ef_search wider than top_k * rerank_factor used to buy nothing beyond that budget on a PQ field; since #673 it activates an extra int8 (SQ) stage that re-ranks the entire ef_search-sized candidate set by a cheap int8 kernel before the narrow exact-stage budget is carved out of it — so the surplus the graph already computed gets a chance to compete on a much better proxy (int8) than the PQ ADC order it arrived in, instead of being thrown away unscored.

The int8 view is derived from the same LRS1 f32 sidecar the exact stage reads (trained once, lazily, on first use, and cached for the segment’s lifetime) — no second on-disk sidecar, no new per-field configuration, and no new query parameter. The chain activates automatically, per query, whenever all of the following hold:

• the field is PQ-quantized (ProductQuantization or ProductQuantizationFastScan);
• a rerank sidecar (rerank_storage: Some(F32)) is loaded;
• rerank_factor is set on the query; and
• effective_ef_search > top_k * rerank_factor — i.e. the graph traversal actually computed more candidates than the exact stage alone would consume. When it doesn’t (the common case where ef_search is left at its default), the SQ stage would narrow nothing and is skipped, so the query takes the identical two-stage (PQ → f32) path it always has.

The score_basis contract (Issue #927) is unaffected either way: the pipeline still ends in the exact f32 stage, so multi-segment fan-out keeps treating those scores as the exact, cross-segment- comparable basis regardless of whether an SQ stage ran ahead of it.

MultiVector Storage

The quantization methods above apply to the ANN index types (Flat, HNSW, IVF). A MultiVector field has no ANN index, so quantizer does not apply to it — instead its own storage option (Issue #1346) picks the on-disk element kind of its token vectors:

#![allow(unused)]
fn main() {
use laurus::{MultiVectorOption, MultiVectorStorage};

let opt = MultiVectorOption::new(128).storage(MultiVectorStorage::Int8);
}

Unlike quantizer’s per-segment global scale, Int8’s scale is per-vector: scale = max(abs(vector)) / 127, computed independently for each token vector, with no corpus- or segment-trained state — which is what lets a segment merge copy rows byte-for-byte instead of re-quantizing on every merge.

Changing storage on an existing field is a Reindex change: Engine::update_field with reindex: true rebuilds the field’s existing segments under the new on-disk kind. It decodes what is stored, so it cannot restore precision a lossy kind already dropped. A schema with no storage key (written before this option existed) defaults to F32 and needs no migration.

Binary

Binary stores one bit per element: 12 bytes per token vector at 96 dimensions, 32 times smaller than F32. Token vectors of a late-interaction model are highly anisotropic — they share one dominant direction — so the plain sign of each element would come out nearly the same for every token. Binary stores the sign of each element’s offset from a center instead, and decodes a row as

x_i     = w_i * (bit_i ? +1 : -1)
decoded = c + x - ((x · c) / (c · c)) * c

where c is the mean token vector and w_i the mean of |v_i - c_i|, both stored as f16. The decoded offset is projected off c: without that, the bits’ spurious component along c acts as a query-independent bias in MaxSim and ranking quality collapses. Queries are not binarized: the float query vectors are scored against the stored bits directly, without decoding the rows (see Cost).

• The field learns c and w once, from the first commit or merge that holds at least 2,048 token vectors (a sample of at most 16,384 of them, spread over the documents). Every later segment is encoded with them, and every Binary segment stores its own copy (4 bytes per dimension), so each segment decodes on its own and merges copy rows verbatim. They are never re-learned, so they do not follow a corpus that drifts afterwards.
• Until then, the field writes its segments as F16. A merge that brings at least 2,048 token vectors together learns the parameters and converts them; converting an existing field to Binary with update_field learns them from the stored vectors. An F16 segment that no merge reaches only costs space.
• Binary requires DistanceMetric::Cosine: it keeps each token vector’s direction, not its magnitude. Under Cosine a decoded vector has a norm of at most 1 (up to f16 rounding), so a score stays at most the number of query vectors.
• It is clearly lossy (see Measured quality), and changing storage away from Binary does not bring the dropped precision back.

Token pooling

pool_factor (Issue #1346) makes a field smaller by keeping fewer token vectors per document. At index time, a document’s token vectors after the first are clustered into max(r / pool_factor, 1) clusters (r is how many there are) and each cluster is replaced by its mean. The stored size and MaxSim’s cost, which grows with the stored vectors, both drop by about the factor, and pooling combines with every storage kind.

#![allow(unused)]
fn main() {
use laurus::{MultiVectorOption, MultiVectorStorage};

let opt = MultiVectorOption::new(128)
    .storage(MultiVectorStorage::Int8)
    .pool_factor(2);
}

• Method. The hierarchical pooling of Clavié et al. (arXiv 2409.14683) as Sentence Transformers and PyLate implement it: the first token vector ([CLS] for a ColBERT model) is kept as is, and the others are grouped by Ward agglomerative clustering on the cosine distance. laurus reproduces SciPy’s clustering; a test checks it against SciPy’s output. In a Cosine field the cluster means are re-normalized, which ranked better on BEIR than the raw means those implementations keep.
• Every write is pooled once. A document is pooled after it is embedded and before the field’s writer is taken, so fresh documents, WAL recovery and text embedded by the field’s embedder are all pooled; queries are not. Each stored document records the factor it was pooled with, so merges never pool it again.
• Turning pooling on for an existing field is a Reindex. Its stored vectors are still the original ones, so Engine::update_field with reindex: true pools them exactly as ingest would have (for F32 storage; other kinds pool their decoded values, and a Binary field keeps the parameters it already learned). The merge pools the stored documents in parallel (one after another on WebAssembly), and ingestion waits until it is done.
• Changing or turning off a factor in use is Destructive. The original token vectors are gone once a document is pooled (multi-vector values are not kept in the document store), so the field’s data is discarded and its documents must be put again.
• Cost. Pooling a document of n token vectors takes O(n²) time and memory: about 3 ms for 300 vectors of 128 dimensions, and about 2 s and 290 MB for the 8,192-vector maximum (Apple M4, release build). Puts made at the same time pool their documents at the same time, outside the field’s writer. Documents pooled at once hold at most 256 MiB together (one document at the maximum, which is then pooled alone), and no more of them run at once than there are cores. A Reindex pools within its own 256 MiB, so documents put through VectorStore directly while VectorStore::rebuild_field runs can bring the total to 512 MiB; Engine::update_field holds puts back until it is done.
• A Binary field learns its parameters from the pooled vectors, and its 2,048-vector threshold counts pooled vectors.

Measured quality

Measured on two BEIR datasets with the BEIR probe: answerdotai/answerai-colbert-small-v1 (96-dimension token vectors, about 236 tokens per document), every storage kind ingesting the same vectors. Exhaustive ranks every document by MaxSim; BM25 → rescore rescores the BM25 top 100. Each cell is nDCG@10, then its paired difference from F32 with a 95% bootstrap confidence interval. The probe commits every 500 documents, so the Binary field learns its parameters in the first commit.

• Every F16 and Int8 interval contains 0, so neither shows a measurable nDCG@10 loss on these datasets. The largest point estimate is Int8’s −0.0021 on SciFact BM25 → rescore.
• Binary loses 1.5 to 5.0 points of nDCG@10. It loses less under BM25 → rescore, where BM25 has already narrowed the candidates, than when MaxSim alone ranks the whole corpus.
• The plain sign of each element (what Vespa’s ColBERT binarization stores, also 12 bytes per token) scored 0.7018 / 0.6978 / 0.3113 / 0.3202 in the same columns. Pooled over both datasets’ 623 queries, Binary is ahead by +0.0117 [−0.0014, +0.0254] exhaustive and +0.0110 [−0.0001, +0.0225] BM25 → rescore. The gain depends on the tokens the parameters are learned from: learning them from the leading 5, 20, 50, 100 or 500 documents instead gave +0.016 to +0.023 exhaustive, and no sample came out behind the plain sign (Issue #1346).
• F16 changes the nDCG@10 of at most 1 of SciFact’s 300 queries and 18 of NFCorpus’s 323. Int8 reorders the top 10 of most queries (237 of 300 on SciFact exhaustive), but the gains and losses cancel out (6 queries up, 7 down).
• F32’s exhaustive nDCG@10 is within 0.005 of the model card’s published numbers (SciFact 0.7477, NFCorpus 0.373), which do not state how they were measured.
• On SciFact the payload is 469 MB (F32), 235 MB (F16), 120 MB (Int8) and 14.7 MB (Binary) for 1.22 M token vectors; segment files add 24 bytes per document and 80 bytes per segment, plus 4 bytes per dimension (384 here) per Binary segment.
• 25 NFCorpus queries have no BM25 hit (single words such as okra that never occur in the corpus; the default analyzer does not stem), so BM25 → rescore scores them 0 under every kind.

With token pooling, measured the same way (the size is that of the segment files, relative to unpooled F32):

• Every interval of pooled F32 and pooled Int8 contains 0. Pooled over both datasets’ 623 queries, the difference from unpooled F32 under exhaustive ranking is −0.0014 [−0.0057, +0.0031] at factor 2, −0.0023 [−0.0071, +0.0026] at factor 3 and −0.0012 [−0.0078, +0.0056] at factor 4, and −0.0028 [−0.0069, +0.0014] for Int8 at factor 2; under BM25 → rescore it is closer to 0. NFCorpus is where the point estimates drop first (about −0.004 at factors 3 and 4).
• Pooling and Int8 multiply: factor 2 with Int8 is 1/7.8 the size of F32 with no measurable loss.
• Binary loses more with pooling: −0.0047 [−0.0120, +0.0024] under exhaustive ranking and −0.0074 [−0.0139, −0.0009] under BM25 → rescore, on top of its own loss.
• Re-normalizing the cluster means matters. With the raw means that Sentence Transformers and PyLate keep, F32 at factors 2, 3 and 4 ranked 0.0033, 0.0045 and 0.0098 lower under exhaustive ranking (pooled over both datasets, every interval above 0; Issue #1346).
• Rescoring time fell about in proportion to the factor: on SciFact exhaustive, 104 ms per query unpooled and 50, 37 and 27 ms at factors 2, 3 and 4 (wall clock on a loaded machine; only the trend is meaningful).

Reproduce with make beir-storage-eval BEIR_DATASET=scifact (or nfcorpus).

Segment Files

All three index types default to the segment-per-commit layout (see below); the monolithic single-file layout remains available via {Flat,Hnsw,Ivf}IndexConfig { segmented: false, .. } for direct index users.

Segment-per-commit layout (default for all three index types)

Since #634 (HNSW) and #889 (Flat and IVF), each commit seals only the newly added vectors as an immutable segment_NNNNNN.{hnsw,flat,ivf} file and registers it in an atomic, checksummed segments.json manifest — the per-commit cost drops from O(index) to O(new documents), with no full-corpus rework for the existing segments:

• Search fans out over the sealed segments (newest generation first) and deduplicates same-document copies with newest-wins masking, so a re-indexed document always resolves to its latest embedding. An adaptive over-fetch — expanding the per-segment budget geometrically until enough live hits survive masking — keeps result quality stable even when a deep band of stale copies awaits a merge.
• Merging: a tiered, generation-contiguous merge policy collapses size-similar segment runs as ingest proceeds (each vector is rewritten O(log N) times over its lifetime), and optimize() force-merges everything into one segment. IVF’s merge does not try to reconcile per-segment cluster ids (they carry no cross-segment meaning): it flattens the deduplicated survivors and retrains centroids from scratch over the merged union, with the cluster count re-derived adaptively for the union’s size. Flat has no f32 rerank sidecar to fall back on, so each merge dequantizes, retrains scalar-quantization parameters, and requantizes — a small amount of quantization drift accumulates across merges, unlike HNSW’s/IVF’s lossless-ish requantization path.
• Per-segment training: HNSW’s graphs and IVF’s centroids are both built independently per segment — IVF adapts its cluster count to each segment’s own size (capped by the schema’s n_clusters) rather than training a fixed count regardless of how many vectors landed in that commit. No cross-segment agreement is needed for either index type, since search fans out per segment and merges top-k results.
• Deletions are logical: an index-level bitmap ({name}.delmap) is consulted by every segment reader and physically reclaimed by merges.
• Migration is zero-copy: an existing monolithic segment file is registered verbatim as the first segment on first open — one manifest write, no data movement.
• Durability: segment files and the manifest are written via temp-file + fsync + atomic rename, and the manifest carries the WAL checkpoint published only after all covered state is durable — see Persistence.

Writer retention across commits (monolithic layout only)

Retention only applies to the monolithic layout (segmented: false); a segmented writer always starts from an empty buffer, so there is nothing to retain. For the monolithic HNSW layout, the store keeps its writer cached across commits (Issue #864): after a commit the writer’s in-memory state is equivalent to the file it just wrote, so the first upsert after a commit extends it in place instead of reloading (and dequantizing) the whole .hnsw from storage — under commit-heavy ingest that reload was O(index size) per cycle. The cache is dropped whenever the index is rewritten behind the writer — auto-compaction firing inside a commit, or an explicit optimize() — because both rebuild the file through a fresh writer and clear the deletion bitmap, and a stale retained writer would write the reclaimed vectors back — and whenever a commit fails partway, where the writer/disk agreement is unknown. As a bonus, a retained writer with no pending changes skips the index rewrite entirely, so a no-change commit() costs no .hnsw write at all. The monolithic Flat and IVF layouts keep the previous drop-on-commit behavior, unaudited for the same invariants — reaching for that code path today requires opting out of the segmented default explicitly.

Code Example

use std::sync::Arc;
use laurus::{Document, Engine, Schema};
use laurus::lexical::TextOption;
use laurus::vector::HnswOption;
use laurus::vector::core::distance::DistanceMetric;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_hnsw_field("embedding", HnswOption {
            dimension: 384,
            distance: DistanceMetric::Cosine,
            m: 16,
            ef_construction: 200,
            ..Default::default()
        })
        .build();

    // With an embedder, text in vector fields is automatically embedded
    let engine = Engine::builder(storage, schema)
        .embedder(my_embedder)
        .build()
        .await?;

    // Add text to the vector field — it will be embedded automatically
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming")
        .add_text("embedding", "Rust is a systems programming language.")
        .build()
    ).await?;

    engine.commit().await?;

    Ok(())
}

Next Steps

• Search the vector index: Vector Search
• Combine with lexical search: Hybrid Search

Search

This section covers how to query your indexed data. Laurus supports three search modes that can be used independently or combined.

Topics

Lexical Search

Keyword-based search using an inverted index. Covers:

• All query types: Term, Phrase, Boolean, Fuzzy, Wildcard, Range, Geo, Span
• BM25 scoring and field boosts
• Using the Query DSL for text-based queries

Vector Search

Semantic similarity search using vector embeddings. Covers:

• VectorSearchRequestBuilder API
• Multi-field vector search and score modes
• Filtered vector search

Hybrid Search

Combining lexical and vector search for best-of-both-worlds results. Covers:

• SearchRequestBuilder API
• Fusion algorithms (RRF, WeightedSum)
• Filtered hybrid search
• Pagination with offset/limit

For spelling correction, see Spelling Correction in the Library section.

Lexical Search

Lexical search finds documents by matching keywords against an inverted index. Laurus provides a rich set of query types that cover exact matching, phrase matching, fuzzy matching, and more.

Basic Usage

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(
            Box::new(TermQuery::new("body", "rust"))
        )
    )
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

Query Types

TermQuery

Matches documents containing an exact term in a specific field.

#![allow(unused)]
fn main() {
use laurus::lexical::TermQuery;

// Find documents where "body" contains the term "rust"
let query = TermQuery::new("body", "rust");
}

Note: Terms are matched after analysis. If the field uses StandardAnalyzer, both the indexed text and the query term are lowercased, so TermQuery::new("body", "rust") will match “Rust” in the original text.

PhraseQuery

Matches documents containing an exact sequence of terms.

#![allow(unused)]
fn main() {
use laurus::lexical::query::phrase::PhraseQuery;

// Find documents containing the exact phrase "machine learning"
let query = PhraseQuery::new("body", vec!["machine".to_string(), "learning".to_string()]);

// Or use the convenience method from a phrase string:
let query = PhraseQuery::from_phrase("body", "machine learning");
}

Phrase queries require term positions to be stored (the default for TextOption). On a multi-valued text field (multi_valued = true) the elements share one position sequence separated by the field’s position_increment_gap (default 100), so a phrase never spans two elements unless its slop reaches that gap — see Schema and Fields.

BooleanQuery

Combines multiple queries with boolean logic.

#![allow(unused)]
fn main() {
use laurus::lexical::query::boolean::{BooleanQuery, BooleanQueryBuilder, Occur};

let query = BooleanQueryBuilder::new()
    .must(Box::new(TermQuery::new("body", "rust")))       // AND
    .must(Box::new(TermQuery::new("body", "programming"))) // AND
    .must_not(Box::new(TermQuery::new("body", "python")))  // NOT
    .build();
}

FuzzyQuery

Matches terms within a specified edit distance (Levenshtein distance).

#![allow(unused)]
fn main() {
use laurus::lexical::query::fuzzy::FuzzyQuery;

// Find documents matching "programing" within edit distance 2
// This will match "programming", "programing", etc.
let query = FuzzyQuery::new("body", "programing");  // default max_edits = 2
}

WildcardQuery

Matches terms using wildcard patterns.

#![allow(unused)]
fn main() {
use laurus::lexical::query::wildcard::WildcardQuery;

// '?' matches exactly one character, '*' matches zero or more
let query = WildcardQuery::new("filename", "*.pdf")?;
let query = WildcardQuery::new("body", "pro*")?;
let query = WildcardQuery::new("body", "col?r")?;  // matches "color" and "colour"
}

PrefixQuery

Matches documents containing terms that start with a specific prefix.

#![allow(unused)]
fn main() {
use laurus::lexical::query::prefix::PrefixQuery;

// Find documents where "body" contains terms starting with "pro"
// This matches "programming", "program", "production", etc.
let query = PrefixQuery::new("body", "pro");
}

RegexpQuery

Matches documents containing terms that match a regular expression pattern.

#![allow(unused)]
fn main() {
use laurus::lexical::query::regexp::RegexpQuery;

// Find documents where "body" contains terms matching the regex
let query = RegexpQuery::new("body", "^pro.*ing$")?;

// Match version-like patterns
let query = RegexpQuery::new("version", r"^v\d+\.\d+")?;
}

Note: RegexpQuery::new() returns Result because the regex pattern is validated at construction time. Invalid patterns will produce an error.

NumericRangeQuery

Matches documents with numeric field values within a range.

#![allow(unused)]
fn main() {
use laurus::lexical::NumericRangeQuery;
use laurus::lexical::core::field::NumericType;

// Find documents where "price" is between 10.0 and 100.0 (inclusive)
let query = NumericRangeQuery::new(
    "price",
    NumericType::Float,
    Some(10.0),   // min
    Some(100.0),  // max
    true,         // include min
    true,         // include max
);

// Open-ended range: price >= 50
let query = NumericRangeQuery::new(
    "price",
    NumericType::Float,
    Some(50.0),
    None,     // no upper bound
    true,
    false,
);
}

Numeric range queries are constant-scored: every matching document receives the same score (the query’s boost, 1.0 by default), following Lucene’s PointRangeQuery semantics. Matching uses the field’s BKD tree when the segment has one; segments without a BKD tree for the field (for example, segments none of whose documents carry the field, or a field configured indexed = false, stored = true) fall back to scanning only the stored documents actually present in that segment.

DateTimeRangeQuery

Matches documents whose DateTime field value falls within a range.

#![allow(unused)]
fn main() {
use chrono::{TimeZone, Utc};
use laurus::lexical::DateTimeRangeQuery;

// Find documents where "created_at" is within 2024 (both bounds inclusive)
let query = DateTimeRangeQuery::between(
    "created_at",
    Utc.with_ymd_and_hms(2024, 1, 1, 0, 0, 0).unwrap(),
    Utc.with_ymd_and_hms(2024, 12, 31, 23, 59, 59).unwrap(),
);

// Open-ended range: created_at > 2024-06-15T12:00:00Z (exclusive)
let query = DateTimeRangeQuery::after(
    "created_at",
    Utc.with_ymd_and_hms(2024, 6, 15, 12, 0, 0).unwrap(),
);

// From textual bounds, using the same literal grammar as the query DSL
let query = DateTimeRangeQuery::from_literals(
    "created_at",
    Some("2024-01-01"),            // min (date only = midnight UTC)
    Some("2024-12-31T23:59:59Z"),  // max
    true,                          // include min
    true,                          // include max
)?;
}

new(field, lower, upper, lower_inclusive, upper_inclusive) takes Option<DateTime<Utc>> bounds; on_or_after, before and on_or_before complete the one-sided helpers, and .with_boost() sets the constant score. The query is evaluated against the field’s one-dimensional BKD point (seconds with a microsecond fraction, so sub-second bounds compare exactly) with the same constant scoring and stored-document fallback as NumericRangeQuery. from_literals accepts RFC 3339, naive YYYY-MM-DDTHH:MM:SS[.fff] (UTC) and YYYY-MM-DD (midnight UTC) — the query the DSL builds for created_at:[2024-01-01 TO 2024-12-31] (see Query DSL). A NumericRangeQuery over a DateTime field with epoch-second bounds keeps working.

GeoDistanceQuery / GeoBoundingBoxQuery

Match documents by 2D geographic location (WGS84 latitude / longitude).

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo::{GeoBoundingBoxQuery, GeoDistanceQuery};

// Find documents within 10 km (= 10 000 m) of Tokyo Station (35.6812, 139.7671)
let query = GeoDistanceQuery::within_radius("location", 35.6812, 139.7671, 10_000.0)?; // distance in metres

// Find documents within a bounding box (min_lat, min_lon, max_lat, max_lon)
let query = GeoBoundingBoxQuery::within_bounding_box(
    "location",
    35.0, 139.0,  // min (lat, lon)
    36.0, 140.0,  // max (lat, lon)
)?;
}

Both queries score by distance (closer documents rank higher: linear decay from the circle’s centre, or from the box’s centre). Matching uses the field’s BKD tree when the segment has one, reading the coordinates directly from the tree — so an index-only field (indexed = true, stored = false) works without stored documents. Segments without a BKD tree for the field (for example, segments none of whose documents carry the field, or a field configured indexed = false, stored = true) fall back to scanning only the stored documents actually present in that segment.

Geo3dDistanceQuery / Geo3dBoundingBoxQuery / Geo3dNearestQuery

Three queries target 3D Geo3d fields backed by ECEF Cartesian coordinates (metres). Use them when altitude matters or when a 2D Geo field would introduce pole singularities. See 3D Geographic Search for the coordinate system, WGS84 conversion helpers, and worked examples.

#![allow(unused)]
fn main() {
use laurus::GeoEcefPoint;
use laurus::lexical::query::geo3d::{
    Geo3dDistanceQuery, Geo3dBoundingBoxQuery, Geo3dNearestQuery,
};

let centre = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);

// Sphere: docs within 5 km of `centre`
let q = Geo3dDistanceQuery::new("position", centre, 5_000.0);

// Axis-aligned 3D bounding box (constructor validates min ≤ max per axis)
let min = GeoEcefPoint::new(-4_000_000.0, 3_300_000.0, 3_650_000.0);
let max = GeoEcefPoint::new(-3_900_000.0, 3_400_000.0, 3_750_000.0);
let q = Geo3dBoundingBoxQuery::new("position", min, max)?;

// k-NN: 10 nearest neighbours, with a custom radius schedule
let q = Geo3dNearestQuery::new("position", centre, 10)
    .with_initial_radius(500.0)
    .with_max_radius(1_000_000.0);
}

Geo3d queries require the field to be indexed (indexed = true, the default): they run entirely on the field’s BKD tree and return no hits from segments that lack one — there is no stored-document fallback for 3D queries.

SpanQuery

Matches terms based on their proximity within a document. Use SpanTermQuery and SpanNearQuery to build proximity queries:

#![allow(unused)]
fn main() {
use laurus::lexical::query::span::{SpanQuery, SpanTermQuery, SpanNearQuery};

// Find documents where "quick" appears near "fox" (within 3 positions)
let query = SpanNearQuery::new(
    "body",
    vec![
        Box::new(SpanTermQuery::new("body", "quick")) as Box<dyn SpanQuery>,
        Box::new(SpanTermQuery::new("body", "fox")) as Box<dyn SpanQuery>,
    ],
    3,    // slop (max distance between terms)
    true, // in_order (terms must appear in order)
);
}

Scoring

Lexical search results are scored using BM25. The score reflects how relevant a document is to the query:

• Higher term frequency in the document increases the score
• Rarer terms across the index increase the score
• Shorter documents are boosted relative to longer ones

Field Boosts

You can boost specific fields to influence relevance using the SearchRequestBuilder:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

let request = SearchRequestBuilder::new()
    .lexical_query(LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust"))))
    .add_field_boost("title", 2.0)  // title matches count double
    .add_field_boost("body", 1.0)
    .build();
}

Lexical Search Options

Lexical search behavior is controlled via LexicalSearchOptions on the SearchRequest, or by using builder methods on SearchRequestBuilder:

Field-sorted searches (sort_by: Field { .. }) always scan every candidate document — there is no early termination for field sorts, unlike the block-max-driven early termination available for score sorts. This guarantees the returned hits are the true top-K by field value rather than an early-terminated approximation, and it means total_hits reflects the true number of matches rather than a scan-truncated count.

When timeout_ms is set, the time budget is enforced cooperatively during the search: the scan loops (including each segment of a multi-segment fanout) check the deadline periodically and abort as soon as the budget is used up, returning a timeout error rather than running the query to completion first. The check is batched (every few thousand scanned documents), so an unset timeout_ms and the common in-budget case pay no measurable overhead.

Builder Methods

SearchRequestBuilder provides convenience methods for lexical options:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::{LexicalSearchQuery, SortField, SortOrder};

let request = SearchRequestBuilder::new()
    .lexical_query(LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust"))))
    .lexical_min_score(0.5)
    .lexical_timeout_ms(5000)
    .lexical_parallel(true)
    .sort_by(SortField::Field { name: "date".to_string(), order: SortOrder::Desc })
    .add_field_boost("title", 2.0)
    .add_field_boost("body", 1.0)
    .limit(20)
    .build();
}

.highlight(fields) and .highlight_config(config) request highlighted fragments per hit — see Highlighting.

Using the Query DSL

Instead of building queries programmatically, you can use the text-based Query DSL:

#![allow(unused)]
fn main() {
use laurus::lexical::QueryParser;
use laurus::analysis::analyzer::standard::StandardAnalyzer;
use std::sync::Arc;

let analyzer = Arc::new(StandardAnalyzer::default());
let parser = QueryParser::new(analyzer).with_default_field("body");

// Simple term
let query = parser.parse("rust")?;

// Boolean
let query = parser.parse("rust AND programming")?;

// Phrase
let query = parser.parse("\"machine learning\"")?;

// Field-specific
let query = parser.parse("title:rust AND body:programming")?;

// Fuzzy
let query = parser.parse("programing~2")?;

// Range
let query = parser.parse("year:[2020 TO 2024]")?;

// DateTime range (date-only bounds are midnight UTC)
let query = parser.parse("created_at:[2024-01-01 TO 2024-12-31]")?;
}

See Query DSL for the complete syntax reference.

Filter Result Cache

Filter clauses — tenancy, category, status flags, and similar — are frequently reused across many requests. Re-evaluating them from scratch every time re-walks the same posting lists. Laurus memoises the set of document ids a filter matches so a repeated filter becomes a single lookup instead of a full posting walk.

• Snapshot-scoped, self-invalidating. The cache lives on the reader, which is rebuilt on every commit() / optimize() / refresh(). Each reader is a point-in-time snapshot, so the cache needs no manual invalidation: after the index changes, the next search starts from a fresh, empty cache and always reflects committed data.
• Score-independent. A filter selects documents without affecting relevance, so the cached value is a plain doc-id set (a Roaring bitmap). It is used for the filter_query of a hybrid / filtered search and feeds both the lexical and vector sides.
• Reused inside boolean queries. An Occur::Filter clause within a BooleanQuery (e.g. must(user_query).filter(tenant_filter)) also draws its matched set from the cache instead of re-walking postings — including the per-segment fan-out path on multi-segment indexes.
• Safe by construction. Only queries with a canonical key are cached. Term, phrase, prefix, wildcard, regexp, fuzzy, range, geo, and geo3d queries are cacheable, as are boolean queries composed entirely of cacheable clauses with at least one positive (Must / Should / Filter) clause. Boolean queries with no positive clause, span queries, and multi-field queries are evaluated fresh (never cached), so results are always correct.

The cache is enabled by default. Tune or disable it via the index config:

#![allow(unused)]
fn main() {
use laurus::lexical::store::config::LexicalIndexConfig;

let config = LexicalIndexConfig::builder()
    .query_filter_cache_capacity(4096) // entries per snapshot; 0 disables the cache
    .build();
}

Parsed Query Cache

Searching with a DSL string (SearchRequest::from_dsl, or a LexicalSearchQuery::Dsl) parses the string with the pest grammar and re-tokenises its terms with the analyzer on every call. Autocomplete and popular-query workloads repeat the same strings, so Laurus memoises dsl string → parsed query: a repeated DSL string is parsed once and then reused (a cheap clone of the parsed query tree).

Like the filter cache, it is snapshot-scoped: the cache lives on the searcher, which is rebuilt on every commit() / optimize() / refresh(). The analyzer and default fields are fixed for that searcher, so the DSL string alone is the key; a schema/analyzer change yields a fresh, empty cache. Enabled by default; tune or disable via the index config:

#![allow(unused)]
fn main() {
use laurus::lexical::store::config::LexicalIndexConfig;

let config = LexicalIndexConfig::builder()
    .parsed_query_cache_capacity(2048) // entries per snapshot; 0 disables the cache
    .build();
}

Posting Cache

Evaluating a term reads its posting list from the segment’s .post file and decodes it (varint doc-ids, deletion filtering, skip table). Without caching, every query for the same term repeats that read + decode — and on cloud/remote storage the read dominates. Each segment reader keeps a small cache of decoded, deletion-filtered posting lists, so a repeated (field, term) lookup within a snapshot reuses the decoded list. The per-term iterator shares the cached list directly (a reference-count bump, not a copy), so evaluating a term never duplicates its posting arrays — this removed the dominant allocation cost on the multi-segment scoring path.

A segment reader’s deletions are fixed for its life, so a cached list is always consistent with them. A commit that leaves a segment unchanged keeps its reader, and the cache with it; a segment whose deletions changed gets a new reader with an empty cache (see Reading After a Commit). The cache is byte-budget bounded (posting lists vary widely in size) — least-recently-used lists are evicted once the budget is exceeded, and a single list larger than the whole budget is not cached. It is enabled by default, and each segment’s cache gets its own max_cache_memory budget (default 128 MiB, which also bounds the reader’s term-info cache), so the worst case grows with the segment count. Control it via the index config:

#![allow(unused)]
fn main() {
use laurus::lexical::store::config::LexicalIndexConfig;

let config = LexicalIndexConfig::builder()
    .enable_posting_cache(false) // disable entirely
    .build();

let config = LexicalIndexConfig::builder()
    .max_cache_memory(256 * 1024 * 1024) // or resize the per-segment budget (bytes)
    .build();
}

Next Steps

• Semantic similarity search: Vector Search
• Combine lexical + vector: Hybrid Search
• Return each chunked document once: Result Grouping
• Full DSL syntax reference: Query DSL

Vector Search

Vector search finds documents by semantic similarity. Instead of matching keywords, it compares the meaning of the query against document embeddings in vector space.

Basic Usage

Builder API

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

let request = SearchRequestBuilder::new()
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "embedding".to_string(),
                payload: DataValue::Text("systems programming language".to_string()),
                weight: 1.0,
            },
        ])
    )
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

The QueryPayload stores raw data (text, bytes, etc.) that will be embedded at search time using the configured embedder.

Query DSL

#![allow(unused)]
fn main() {
use laurus::vector::VectorQueryParser;

let parser = VectorQueryParser::new(embedder.clone())
    .with_default_field("embedding");

let request = parser.parse(r#"embedding:"systems programming""#).await?;
}

VectorSearchQuery

The vector search query is specified as a VectorSearchQuery enum:

QueryPayload

QueryVector

Examples

#![allow(unused)]
fn main() {
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::{QueryPayload, QueryVector};
use laurus::vector::core::vector::Vector;
use laurus::data::DataValue;

// Text query (will be embedded at search time)
let query = VectorSearchQuery::Payloads(vec![
    QueryPayload {
        field: "text_vec".to_string(),
        payload: DataValue::Text("machine learning".to_string()),
        weight: 1.0,
    },
]);

// Pre-computed vector
let query = VectorSearchQuery::Vectors(vec![
    QueryVector {
        vector: Vector::from(vec![0.1, 0.2, 0.3]),
        weight: 1.0,
        fields: Some(vec!["embedding".to_string()]),
    },
]);
}

Multi-Field Vector Search

You can search across multiple vector fields in a single request:

#![allow(unused)]
fn main() {
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

let query = VectorSearchQuery::Payloads(vec![
    QueryPayload {
        field: "text_vec".to_string(),
        payload: DataValue::Text("cute kitten".to_string()),
        weight: 1.0,
    },
    QueryPayload {
        field: "image_vec".to_string(),
        payload: DataValue::Text("fluffy cat".to_string()),
        weight: 1.0,
    },
]);
}

Each clause produces a vector that is searched against its respective field. Results are combined using the configured score mode.

Score Modes

Parallel Multi-Vector Execution

When a request carries multiple query vectors (e.g., several fields, or several query embeddings combined by a score mode), Laurus dispatches the per-query similarity searches in parallel via rayon.

Behaviour:

• The parallelisation lives in VectorIndexSearcher::search_batch’s default implementation. On native builds (default native feature), once queries.len() reaches the searcher’s parallel_threshold (default 4, overridable per searcher), the per-query HNSW / Flat / IVF searches run on rayon’s global thread pool. Below that threshold the serial loop wins because rayon’s dispatch overhead (~1-2 µs) would otherwise dominate a single 50-200 µs query.
• On wasm32 targets the serial path is always used because rayon is unavailable.
• Aggregation (the score-mode merge) and the final sort run serially after the per-query phase. Score ties are broken by ascending doc_id so the results are deterministic regardless of rayon’s work-stealing schedule.

The external API surface (VectorStore::search, gRPC Search, REST POST /v1/search, all language bindings) is unchanged; parallel execution is purely an internal optimisation enabled by upgrading laurus. Speedups scale with the host’s available cores — on a 4-core / 8-thread laptop CPU the parallel path reaches roughly 2× throughput at B = 64 query vectors, limited by physical core count and HyperThreading sharing.

Parallel Brute-Force Scan

Flat and IVF indexes rank candidates with an exhaustive distance scan rather than a graph walk. When one query’s candidate count reaches an internal threshold (2048), that scan is dispatched across rayon’s global thread pool; below it the serial loop wins because rayon’s per-job dispatch (~1-2 µs) would dominate a small scan.

This is orthogonal to the per-query parallelism above: a batch parallelises across queries, and each large query further parallelises its own scan on the same pool, with work-stealing bounding total parallelism to the pool size (no OS-thread oversubscription). The distance kernel has no side effects, so results are collected in arbitrary order and then sorted, keeping output deterministic. On wasm32 (no rayon) the scan is always serial.

Speedup scales with the host’s physical cores and is largest for big Flat indexes or a wide IVF n_probe; scans below the threshold are unaffected.

IVF Cluster Selection

Before the distance scan, an IVF query first chooses which clusters to scan: it scores the query against every centroid and keeps the n_probe nearest. Because the probed clusters are then merged and re-ranked by similarity, the centroids’ relative order does not matter, so the nearest n_probe are taken with an O(K) partial selection (select_nth_unstable_by) over the K centroids rather than a full O(K log K) sort. The saving grows with the cluster count K; at K = 2048 it cut the per-query coarse step by roughly 18% (Issue #668). The centroid scan itself stays serial — each centroid is a single distance computation, so dispatching it to rayon costs more than it saves at realistic cluster counts (K ≈ √N).

Weights

Use the ^ boost syntax in DSL or weight in QueryVector to adjust how much each field contributes:

text_vec:"cute kitten"^1.0 image_vec:"fluffy cat"^0.5

This means text similarity counts twice as much as image similarity.

A vector field’s schema-level base_weight (Issue #1084) works the same way but is a per-field default rather than a per-query override: it is multiplied into the query weight above, so base_weight and ^/weight compose. Because both only affect the vector side’s own scoring, neither can rebalance a hybrid search’s lexical-vs-vector split — FusionAlgorithm::RRF is rank-only and ignores the underlying score entirely, and FusionAlgorithm::WeightedSum min-max normalizes each side before applying lexical_weight/vector_weight, which cancels out a uniform per-field scalar. What both weighting mechanisms DO affect is the relative priority among vector fields when a query targets two or more at once.

This per-field summing only happens when the query names its target fields. A field-less query cannot apply base_weight (it has no single field to look it up against — Issue #1084), so when it reaches a document through more than one same-dimension field it takes that document’s best-scoring field instead of summing across them (Issue #1343; see Field Routing below).

Field Routing

In a multi-field schema, each vector field has its own HNSW graph. By default a query searches every vector field; restricting it to named fields skips the others’ graph work entirely.

Two routing inputs are honored, in priority order:

1. Per-query — QueryVector.fields. The DSL parser sets this from the field a clause names, so image_vec:"fluffy cat" only searches image_vec.
2. Request-level — VectorSearchParams.fields, a list of selectors:
   • Exact("image_vec") — match a field by exact name.
   • Prefix("image_") — match every field whose name starts with the prefix (resolved against the index’s field names).

When neither is set, all fields are searched (the default). A query routed to a field never returns documents that lack a vector in that field.

A document that holds vectors in two or more of the searched fields still comes back only once: the field-less fanout collapses to that document’s best-scoring field (Issue #1343), so limit/top_k counts documents, not (document, field) pairs. This is unlike an explicit multi-field query (fields: [Exact("a"), Exact("b")]), which sums the fields’ scores — see Weights above.

You can apply lexical filters to narrow the vector search results:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

// Vector search with a category filter
let request = SearchRequestBuilder::new()
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "embedding".to_string(),
                payload: DataValue::Text("machine learning".to_string()),
                weight: 1.0,
            },
        ])
    )
    .filter_query(Box::new(TermQuery::new("category", "tutorial")))
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

The filter query runs first on the lexical index to identify allowed document IDs, then the vector search is restricted to those IDs.

Filter-Aware HNSW Traversal

For HNSW fields the allowed-ID set is pushed into the graph search itself, not just applied afterwards. During traversal the frontier still expands through every neighbour — including non-matching ones — so the search can cross non-matching regions of the graph, but only matching documents enter the result set.

This matters for selective filters. A plain post-filter inspects the fixed ef_search window of nearest neighbours and keeps whichever happen to match; when matches are rare they can fall entirely outside that window, so the query returns far fewer hits than exist — sometimes none, even though matching documents are reachable. Filter-aware traversal keeps searching until it has collected enough matches or hits an internal visit cap (a multiple of ef_search) that bounds latency for very selective filters.

The unfiltered path is unchanged: with no filter the traversal behaves exactly as before. Flat and IVF fields honour the allow-set inline — a candidate whose document ID is not in the set is skipped before the distance kernel runs, so a selective filter avoids the wasted distance computations a post-filter would incur. Their scan is exhaustive either way, so recall is unchanged; the store’s post-filter still runs afterwards as a redundant safety net.

When the allow-set is smaller than ef_search, the HNSW field skips the graph walk entirely and scores the allowed documents directly. With so few candidates there is nothing for the graph to find; the direct scan touches exactly the allowed documents — never more than the walk would — and is exact, so a very selective filter returns the true nearest matches rather than an approximation. Larger allow-sets keep using the filter-aware traversal above.

Deletion-Aware HNSW Traversal

Logically deleted documents stay in the HNSW graph until compaction (see Deletions & Compaction), so the walk must skip them the same way it skips filtered-out documents. The graph traversal applies a single admission rule: a node enters the result set only if it matches the filter (when one is present) and is not deleted, while the frontier still expands through deleted nodes to preserve connectivity.

This is what keeps recall correct as deletions accumulate. If deleted nodes were allowed into the result heap, they would consume the fixed ef_search slots and push out live neighbours — in the worst case an ef_search window made entirely of deleted documents would return nothing. Skipping them during the walk means the same slots fill with live results instead, so a 10-hit page stays full even after the nearest documents are deleted. The exact tiny-allow-set scan above and the Flat/IVF inline paths apply the same deletion check.

The fast path is preserved: when a search has neither a filter nor any deletions, the traversal runs the original loop unchanged, paying nothing for the per-neighbour admission bookkeeping.

Allow-Set Representation

The allow-set is a typed structure chosen by shape: a Roaring bitmap for dense filters and a hash set for sparse ones. For a filtered hybrid search the lexical side already produced the matching document set as a bitmap (see the lexical Filter Result Cache); that bitmap is handed to the vector side as-is, so the set is materialised once for the whole query instead of being rebuilt for each side. This is an internal optimisation — the public filter API is unchanged.

Filter with Numeric Range

#![allow(unused)]
fn main() {
use laurus::lexical::NumericRangeQuery;
use laurus::lexical::core::field::NumericType;

let request = SearchRequestBuilder::new()
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "embedding".to_string(),
                payload: DataValue::Text("type systems".to_string()),
                weight: 1.0,
            },
        ])
    )
    .filter_query(Box::new(NumericRangeQuery::new(
        "year", NumericType::Integer,
        Some(2020.0), Some(2024.0), true, true
    )))
    .limit(10)
    .build();
}

Late-Interaction Rescore

A late-interaction rescore (Issue #1345) reorders the top candidates of any search — lexical, vector or hybrid — with ColBERT-style late interaction. Each document keeps one vector per token in a multi-vector field, the query is likewise a set of token vectors, and a document scores the sum, over the query vectors, of each one’s best match among the document’s vectors (MaxSim):

score(q, d) = Σ_i max_j (q_i · d_j)

Matching token by token keeps the details that a single document embedding averages away, while the first stage keeps the search fast: only the top candidates are rescored, so the multi-vector field needs no ANN index.

query ─→ first stage: lexical / vector / hybrid (as usual)
              │ top window_size candidates (default 100)
              ▼
         rescore: MaxSim against the multi-vector field
              ▼
         offset / limit → documents

When the multi-vector field has a token-level embedder (a candle_colbert one, see CandleColbertEmbedder), documents can be indexed as text and the query given as text:

#![allow(unused)]
fn main() {
use laurus::{RescoreOptions, SearchRequestBuilder};

let request = SearchRequestBuilder::new()
    .query_dsl("body:lifetimes body_vec:\"how do lifetimes work\"")
    .rescore(
        RescoreOptions::late_interaction_text("body_colbert", "how do lifetimes work")
            .window_size(100),
    )
    .limit(10)
    .build();
let results = engine.search(request).await?;
}

Without one, pass the query’s token vectors, computed with the same model that produced the documents’ token vectors:

#![allow(unused)]
fn main() {
use laurus::RescoreOptions;
use laurus::vector::Vector;

let query_tokens: Vec<Vector> = colbert_query_vectors("how do lifetimes work");
let rescore = RescoreOptions::late_interaction("body_colbert", query_tokens);
}

Besides the Rust API, the rescore stage is available over gRPC and the HTTP gateway, in the MCP search tool and in laurus search, and in the Python, Ruby, PHP, Node.js and WASM bindings.

Options

A text query is embedded by the field’s token-level embedder, as a query, before the first stage runs; with EngineBuilder::embedding_cache_capacity set, a repeated query (for example the next page) is served from the cache.

The query must hold between 1 and 1,024 vectors, each of the field’s dimension and with finite values. The request fails with LaurusError::InvalidArgument before any search runs when the options are invalid, when the field is not a multi-vector field, when a text query is blank or the field has no token-level embedder, or when the request also sorts by a field (sort_by with SortField::Field): a field sort does not rank by score, so there is nothing to rescore.

Ordering and Scores

1. The window — the top window_size first-stage candidates — is sorted by MaxSim, highest first. Equal scores are ordered by internal document ID, so the order is deterministic.
2. Window candidates without token vectors in the field follow, in their first-stage order.
3. Candidates beyond the window follow last, unchanged.

A rescored hit’s score is its MaxSim; the other hits keep their first-stage score. The two kinds of score are not comparable, which is why the groups are ordered by position rather than by score.

The first stage fetches max(window_size, offset + limit) candidates, so every page is cut from the same rescored ranking: pages fetched one by one concatenate to the ranking a single large page returns, even when a page reaches past the window. A larger window lets the rescore promote documents the first stage ranked lower, at a cost linear in the window; it cannot recover a document the first stage did not retrieve at all.

Similarity

The similarity of a query vector and a document vector is their dot product. In a field with DistanceMetric::Cosine (the default), document vectors are L2-normalized when written and query vectors when searched, so each pair contributes its cosine similarity and a score is at most the number of query vectors (up to rounding in a compressed kind). With DistanceMetric::DotProduct the vectors are used as given.

The field’s stored vectors may also be compressed: its storage option can encode them as F16, Int8 or Binary instead of the exact F32 default, trading some score precision for a smaller index — F16 carries ~2⁻¹¹ relative error per element, Int8’s error is bounded by its per-vector scale (max(abs(vector)) / 127), and Binary keeps one bit per element against a center the field learns, at a measurable cost in nDCG@10. The query vectors are never compressed.

Cost

Rescoring takes window_size × query vectors × document vectors dot products, so a field’s pool_factor cuts it by about that factor. On native builds the candidates are scored in parallel on rayon’s global thread pool; on wasm32 they are scored one by one. As a guide, 100 candidates of 300 vectors with 128 dimensions and a 32-vector query (about 123 million multiply-adds) took about 1.4 ms per search on an Apple M4, and about 5.5 ms on one thread.

Vectors of 64, 96 or 128 dimensions, the widths of common ColBERT-style models, are scored by a copy of the MaxSim loop made for that width. Other dimensions score four document vectors per pass over a query vector, which costs more per pair than the fixed-width copies but 31 to 36% less than scoring one document vector at a time: on one thread of an Apple M4, a pair retired about 54, 73 and 100 instructions at 64, 96 and 128 dimensions, and about 114 at 100 dimensions. The scores are the same through every loop.

A Binary field is scored straight from its bits. For each query vector, the search first builds one 256-entry table per byte of a row, holding the byte’s contribution to the dot product for each of its bit patterns. A query vector and a document vector then cost ceil(dimension / 8) table lookups instead of a decode and dimension multiply-adds. The tables take ceil(dimension / 8) KiB per query vector (384 KiB for 32 vectors of 96 dimensions) and are built once per search. A query that would need more than 4 MiB of tables decodes its Binary candidates instead. On the SciFact and NFCorpus token vectors of answerai-colbert-small-v1 (96 dimensions, Apple M4), a pair retires about 50 instructions, against about 74 for F32 and about 224 when the rows are decoded first.

Distance Metrics

The distance metric is configured per field in the schema (see Vector Indexing):

Code Example: Complete Vector Search

use std::sync::Arc;
use laurus::{Document, Engine, Schema, SearchRequestBuilder, PerFieldEmbedder};
use laurus::lexical::TextOption;
use laurus::vector::HnswOption;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_hnsw_field("text_vec", HnswOption {
            dimension: 384,
            ..Default::default()
        })
        .build();

    // Set up per-field embedder
    let embedder = Arc::new(my_embedder);
    let pfe = PerFieldEmbedder::new(embedder.clone());
    pfe.add_embedder("text_vec", embedder.clone());

    let engine = Engine::builder(storage, schema)
        .embedder(Arc::new(pfe))
        .build()
        .await?;

    // Index documents (text in vector field is auto-embedded)
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming")
        .add_text("text_vec", "Rust is a systems programming language.")
        .build()
    ).await?;
    engine.commit().await?;

    // Search by semantic similarity
    let results = engine.search(
        SearchRequestBuilder::new()
            .vector_query(
                VectorSearchQuery::Payloads(vec![
                    QueryPayload {
                        field: "text_vec".to_string(),
                        payload: DataValue::Text("systems language".to_string()),
                        weight: 1.0,
                    },
                ])
            )
            .limit(5)
            .build()
    ).await?;

    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }

    Ok(())
}

Next Steps

• Combine with keyword search: Hybrid Search
• Return each chunked document once: Result Grouping
• DSL syntax for vector queries: Query DSL

Hybrid Search

Hybrid search combines lexical search (keyword matching) with vector search (semantic similarity) to deliver results that are both precise and semantically relevant. This is Laurus’s most powerful search mode.

Why Hybrid Search?

How It Works

sequenceDiagram
    participant User
    participant Engine
    participant Lexical as LexicalStore
    participant Vector as VectorStore
    participant Fusion

    User->>Engine: SearchRequest\n(lexical + vector)

    par Execute in parallel
        Engine->>Lexical: BM25 keyword search
        Lexical-->>Engine: Ranked hits (by relevance)
    and
        Engine->>Vector: ANN similarity search
        Vector-->>Engine: Ranked hits (by distance)
    end

    Engine->>Fusion: Merge two result sets
    Note over Fusion: RRF or WeightedSum
    Fusion-->>Engine: Unified ranked list
    Engine-->>User: Vec of SearchResult

The lexical and vector searches are independent, so the engine runs them concurrently rather than one after the other. Any query embedding is computed first (the vector search depends on it), then both searches execute in parallel. As a result the hybrid latency is bounded by the slower of the two searches (max(lexical, vector)) instead of their sum, which matters most when both sides do substantial work. Fusion is order-independent, so concurrent execution returns exactly the same ranked list as a sequential run.

Basic Usage

Builder API

#![allow(unused)]
fn main() {
use laurus::{SearchRequestBuilder, FusionAlgorithm};
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

let request = SearchRequestBuilder::new()
    // Lexical component
    .lexical_query(
        LexicalSearchQuery::Obj(
            Box::new(TermQuery::new("body", "rust"))
        )
    )
    // Vector component
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "text_vec".to_string(),
                payload: DataValue::Text("systems programming".to_string()),
                weight: 1.0,
            },
        ])
    )
    // Fusion algorithm
    .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

Query DSL

Mix lexical and vector clauses in a single query string:

#![allow(unused)]
fn main() {
use laurus::UnifiedQueryParser;
use laurus::lexical::QueryParser;
use laurus::vector::VectorQueryParser;

let unified = UnifiedQueryParser::new(
    QueryParser::new(analyzer).with_default_field("body"),
    VectorQueryParser::new(embedder),
);

// Lexical + vector in one query
let request = unified.parse(r#"body:rust text_vec:"systems programming""#).await?;
let results = engine.search(request).await?;
}

The parser uses the schema to identify vector clauses by field type. Fields defined as vector fields (e.g., HNSW) are parsed as vector queries; everything else is parsed as lexical.

Fusion Algorithms

When both lexical and vector results exist, they must be merged into a single ranked list. Laurus supports two fusion algorithms:

RRF (Reciprocal Rank Fusion)

The default algorithm. Combines results based on their rank positions rather than raw scores.

score(doc) = sum( 1 / (k + rank_i) )

Where rank_i is the position of the document in each result list, and k is a smoothing parameter (default 60).

#![allow(unused)]
fn main() {
use laurus::FusionAlgorithm;

let fusion = FusionAlgorithm::RRF { k: 60.0 };
}

Advantages:

• Robust to different score distributions between lexical and vector results
• No need to tune weights
• Works well out of the box

WeightedSum

Linearly combines normalized lexical and vector scores:

score(doc) = lexical_weight * lexical_score + vector_weight * vector_score

#![allow(unused)]
fn main() {
use laurus::FusionAlgorithm;

let fusion = FusionAlgorithm::WeightedSum {
    lexical_weight: 0.3,
    vector_weight: 0.7,
};
}

When to use:

• When you want explicit control over the balance between lexical and vector relevance
• When you know one signal is more important than the other

SearchRequest Fields

SearchResult

Each result contains:

Highlighting in hybrid search is driven by the request’s lexical query, so a hit that came only from the vector side has an empty highlights map. See Highlighting.

Filtered Hybrid Search

Apply a filter to restrict both lexical and vector results:

#![allow(unused)]
fn main() {
let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust")))
    )
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "text_vec".to_string(),
                payload: DataValue::Text("systems programming".to_string()),
                weight: 1.0,
            },
        ])
    )
    // Only search within "tutorial" category
    .filter_query(Box::new(TermQuery::new("category", "tutorial")))
    .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
    .limit(10)
    .build();
}

How Filtering Works

1. The filter query runs on the lexical index to produce a set of allowed document IDs
2. For lexical search: the filter is combined with the user query as a boolean AND
3. For vector search: the allowed IDs are passed to restrict the ANN search

Pagination

Use offset and limit for pagination:

#![allow(unused)]
fn main() {
// Page 1: results 0-9
let page1 = SearchRequestBuilder::new()
    .lexical_query(/* ... */)
    .vector_query(/* ... */)
    .offset(0)
    .limit(10)
    .build();

// Page 2: results 10-19
let page2 = SearchRequestBuilder::new()
    .lexical_query(/* ... */)
    .vector_query(/* ... */)
    .offset(10)
    .limit(10)
    .build();
}

Rescoring the Fused Results

Hybrid search makes a good first stage for a late-interaction rescore: fusion gathers candidates that match either the keywords or the meaning, and the rescore then reorders the top window_size of them by token-level MaxSim against a multi-vector field:

#![allow(unused)]
fn main() {
use laurus::RescoreOptions;

let request = SearchRequestBuilder::new()
    .lexical_query(/* ... */)
    .vector_query(/* ... */)
    .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
    .rescore(RescoreOptions::late_interaction("body_colbert", query_tokens))
    .limit(10)
    .build();
}

The rescore runs after fusion and before offset / limit, so pagination works the same with or without it. See Late-Interaction Rescore for the options, the ordering rules and the cost.

Complete Example

use std::sync::Arc;
use laurus::{
    Document, Engine, Schema, SearchRequestBuilder,
    FusionAlgorithm, PerFieldEmbedder,
};
use laurus::lexical::{TextOption, TermQuery};
use laurus::lexical::core::field::IntegerOption;
use laurus::lexical::search::searcher::LexicalSearchQuery;
use laurus::vector::HnswOption;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    // Schema with both lexical and vector fields
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_text_field("category", TextOption::default())
        .add_integer_field("year", IntegerOption::default())
        .add_hnsw_field("body_vec", HnswOption {
            dimension: 384,
            ..Default::default()
        })
        .build();

    // Configure analyzer and embedder (see Text Analysis and Embeddings docs)
    // let analyzer = Arc::new(StandardAnalyzer::new()?);
    // let embedder = Arc::new(CandleBertEmbedder::new("sentence-transformers/all-MiniLM-L6-v2")?);
    let engine = Engine::builder(storage, schema)
        // .analyzer(analyzer)
        // .embedder(embedder)
        .build()
        .await?;

    // Index documents with both text and vector fields
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming Guide")
        .add_text("body", "Rust is a systems programming language.")
        .add_text("category", "programming")
        .add_integer("year", 2024)
        .add_text("body_vec", "Rust is a systems programming language.")
        .build()
    ).await?;
    engine.commit().await?;

    // Hybrid search: keyword "rust" + semantic "systems language"
    let results = engine.search(
        SearchRequestBuilder::new()
            .lexical_query(
                LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust")))
            )
            .vector_query(
                VectorSearchQuery::Payloads(vec![
                    QueryPayload {
                        field: "body_vec".to_string(),
                        payload: DataValue::Text("systems language".to_string()),
                        weight: 1.0,
                    },
                ])
            )
            .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
            .limit(10)
            .build()
    ).await?;

    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }

    Ok(())
}

Next Steps

• Full query syntax reference: Query DSL
• Return each chunked document once: Result Grouping
• Understand ID resolution: ID Management
• Data durability: Persistence & WAL

Result Grouping

When a long document is indexed as several chunks, the chunks of one document often fill the top of the ranking and push other documents off the page. Result grouping (Issue #1297) folds the hits that share a field value into one group and keeps the best few hits of each group, like Elasticsearch’s collapse or Qdrant’s group_by. It works the same way for lexical, vector and hybrid searches.

Basic Usage

Chunks added with Engine::add_document share their document’s _id, so grouping by _id returns each document once, represented by its best chunk:

#![allow(unused)]
fn main() {
use laurus::{GroupBy, SearchRequestBuilder};

engine.add_document("manual", chapter_1).await?;
engine.add_document("manual", chapter_2).await?;
engine.add_document("faq", faq).await?;
engine.commit().await?;

let request = SearchRequestBuilder::new()
    .query_dsl("body:lifetimes")
    .group_by(GroupBy::new("_id").group_size(3))
    .limit(10)
    .build();
for result in engine.search(request).await? {
    // The group's best chunk, then up to two more of the same document.
    println!("{} {}", result.id, result.score);
    for hit in &result.inner_hits {
        println!("  {}", hit.score);
    }
}
}

Besides the Rust API, grouping is available over gRPC and the HTTP gateway, in the MCP search and search_batch tools, in laurus search --group-by, and in the Python, Node.js, Ruby, PHP and WASM bindings.

How It Works

query ─→ first stage: lexical / vector / hybrid (as usual)
              ▼
         rescore (when requested)
              ▼
         group the ranked hits by the field's value
              ▼
         offset / limit over groups → documents

Grouping runs on the final ranking, after fusion and rescore and before the page is cut:

1. Groups are ordered by the position of their best hit in the ranking.
2. Within a group, hits keep their ranking order, up to group_size hits.
3. The first hit of a group is the returned result; the others are its inner_hits.

Grouping follows positions in the ranking and never compares scores, so it also follows a lexical sort_by and a rescore, whose rescored and unrescored hits have scores of different kinds.

Options

Grouping Fields

A grouping field is _id, or a schema field that is:

• of type Text, Integer, Float, Boolean or DateTime;
• single-valued (multi_valued: false);
• stored, with doc_values enabled (both are on by default).

Values are compared as stored: a Text value is compared as the whole original string, not as analyzed terms. Float values that compare equal (0.0 and -0.0) share a group, and so do NaN values. A document with no value for the field forms a group of its own, with no group_key, so no hit is dropped and unrelated documents are never merged.

The search fails with LaurusError::InvalidArgument before any search runs when the field is unknown, of another type, multi-valued, not stored or without doc_values, when group_size is outside 1..=100, or when offset + limit exceeds 10,000. A limit of 0 returns no results without searching.

Results

limit and offset count groups, not hits. Each returned SearchResult is a group’s best hit, with two fields filled:

Each inner hit carries the same group_key and has no inner hits of its own. Highlights are filled for inner hits too. An ungrouped search leaves both fields empty.

Candidate Pool

The first stage of a grouped search ranks (offset + limit) × group_size × 2 candidates, or the rescore window when that is larger. When those candidates hold fewer than offset + limit groups, the engine doubles the pool and runs the first stage again, until the pool holds enough groups, the first stage has no more hits, or the pool reaches 10,000 candidates. Query embedding runs once; a rescore scores each candidate once across rounds. Doubling keeps the total work within about twice that of the last round.

The number of groups is complete within that bound. The number of inner hits is not: a group’s further hits are taken from the same pool, so a group can hold fewer than group_size hits even when more of its documents match.

lexical_timeout_ms limits the lexical time of all rounds together: when widening runs past it, the search fails as an ungrouped search would.

Group values are read from the field’s DocValues column. The first grouped search on a field loads that column of each segment it reaches into memory, where it stays for later searches, as for sorting by the field. For 1,000,000 documents with 12-character _id values, the _id columns of every segment took about 111 MB (about 111 bytes per document).

Next Steps

• Lexical Search
• Vector Search
• Hybrid Search

Query DSL

Laurus provides a unified query DSL (Domain Specific Language) that allows lexical (keyword) and vector (semantic) search in a single query string. The UnifiedQueryParser splits the input into lexical and vector portions and delegates to the appropriate sub-parser.

Overview

title:hello AND content:"cute kitten"^0.8
|--- lexical --|    |--- vector --------|

The field type in the schema determines whether a clause is lexical or vector. If the field is a vector field (e.g., HNSW), the clause is treated as a vector query. Everything else is treated as a lexical query.

Field validation

Every field:value clause is validated against the schema at parse time. A query that references a field not declared in the schema is rejected with an error rather than returning silently-empty results. This catches typos early (e.g. titl:hello instead of title:hello).

If you want the engine to accept documents with previously-unknown fields, set the schema’s dynamic_field_policy so that the field gets added during ingestion. Once a field is part of the schema, queries referencing it succeed.

Lexical Query Syntax

Lexical queries search the inverted index using exact or approximate keyword matching.

Term Query

Match a single term against a field (or the default field):

hello
title:hello

If the field’s analyzer splits a bare term into multiple tokens — for example, a Japanese sentence under a morphological (Lindera) analyzer, or a hyphenated word like rust-lang under the default \w+ tokenizer — the resulting tokens are combined with OR (matching Lucene’s match query), not required to appear as an exact phrase. Use a quoted Phrase Query when exact adjacency is required.

Boolean Operators

Combine clauses with AND and OR (case-insensitive):

title:hello AND body:world
title:hello OR title:goodbye

AND is symmetric — it makes the clauses on both sides required (Must). For example, title:hello AND body:world returns only documents that match both clauses. The same is true for chains: a AND b AND c requires all three. A clause that was already explicitly marked with + (required) or - (prohibited) keeps that intent — AND does not override an explicit prefix.

Space-separated clauses without an explicit operator use implicit boolean (behaves like OR with scoring), so a b AND c reads as “optionally match a, and require both b and c”.

Required / Prohibited Clauses

Use + (must match) and - (must not match):

+title:hello -title:goodbye

Phrase Query

Match an exact phrase using double quotes. Optional proximity (~N) allows N words between terms:

"hello world"
"hello world"~2

On a multi-valued text field the phrase (or its ~N proximity window) never crosses from one element into the next unless N reaches the field’s position_increment_gap (default 100).

A quoted value that the field’s analyzer turns into a single token is an exact term match, the same as a Term Query (any ~N is ignored). This is how to search a keyword field for a value the unquoted form does not accept, such as a path, a URL, or an ID containing ., /, or ::

path:"file:///notes/a.md"
links:"https://example.com/a"

When the field’s analyzer stacks several tokens at one position, as SynonymGraphFilter does, they are alternatives at that position. With big and large as synonyms, "big" matches documents that contain either word, and "a big dog" also matches “a large dog”. A single position’s alternatives are scored as one blended term (a SynonymQuery), not summed, so a document containing both big and large does not score higher than one containing big twice. A multi-word synonym makes one phrase per alternative, and the value matches when any of them does: with ml and machine learning as synonyms, "ml" matches ml or the phrase machine learning. These phrases are matched together through the graph, not one by one, so there is no limit on how many a value makes, and a document gets one phrase score however many of them it contains. See Synonym Expansion.

A phrase of two or more positions compares term positions, which a field stores only with term_vectors: true (the default). On a field with term_vectors: false the search is rejected with a query error instead of returning no hits. This includes an unfielded phrase that expands over the default fields when one of them has term_vectors: false, and a quoted word whose multi-word synonym makes it a phrase, such as "ml" above.

Fuzzy Query

Approximate matching with edit distance. Append ~ and optionally the maximum edit distance:

roam~
roam~2

Wildcard Query

Use ? (single character) and * (zero or more characters):

te?t
test*

Range Query

Inclusive [] or exclusive {} ranges, useful for numeric and date fields:

price:[100 TO 500]
date:{2024-01-01 TO 2024-12-31}
price:[* TO 100]

Date and datetime bounds

A range whose bounds are datetime literals builds a DateTimeRangeQuery, evaluated against the field’s BKD tree with constant scoring like a numeric range. A bound that is not a bare number is tried as, in order:

# Date-only bounds (upper bound is 2024-12-31T00:00:00Z, see below)
created_at:[2024-01-01 TO 2024-12-31]

# RFC 3339 with an offset (09:00+09:00 is 00:00Z)
created_at:[2024-06-15T09:00:00+09:00 TO 2024-06-16T09:00:00+09:00]

# Naive datetime is UTC; * leaves the upper side open
created_at:[2024-06-15T12:34:56 TO *]

# Exclusive on both sides
created_at:{2024-01-01 TO 2025-01-01}

• A date-only bound is midnight UTC, so [2024-01-01 TO 2024-12-31] includes 2024-12-31T00:00:00Z and nothing later that day. To cover the whole year, give the upper bound a time ([2024-01-01 TO 2024-12-31T23:59:59.999999]) or end an exclusive range on the next day ({2024-01-01 TO 2025-01-01} — note that {} also excludes the exact instant 2024-01-01T00:00:00Z).
• * leaves a side open.
• Bare numbers are still numeric bounds: on a DateTime field they are epoch seconds and build a NumericRangeQuery.
• Mixing a numeric bound with a datetime bound ([1704067200 TO 2024-12-31]), or a datetime bound with a string that is not a datetime literal ([2024-01-01 TO tomorrow]), is a parse error rather than a silent empty result. A range whose bounds are neither numbers nor datetime literals (title:[A TO Z]) still parses but matches nothing, as before.

Document ingestion coerces text into a DateTime field only from RFC 3339; the naive and date-only forms are accepted for query literals only.

2D Geographic Queries (geo_*)

Two function-style forms target Geo (2D latitude / longitude) fields. All arguments are signed floats; latitudes / longitudes are in degrees and the distance is in metres:

location:geo_distance(lat, lon, distance_m)
location:geo_bbox(min_lat, min_lon, max_lat, max_lon)

Examples:

# Within 10 km (= 10 000 m) of Tokyo (35.6895, 139.6917)
location:geo_distance(35.6895, 139.6917, 10000)

# Inside an axis-aligned lat/lon bounding box
location:geo_bbox(35.0, 139.0, 36.0, 140.0)

The query field must be declared as a Geo field in the schema. Latitudes must lie in [-90, 90] and longitudes in [-180, 180]; the parser rejects out-of-range values.

3D Geographic Queries (geo3d_*)

Three function-style forms target Geo3d (3D ECEF Cartesian) fields. All arguments are signed floats in metres, except k (an unsigned integer):

position:geo3d_distance(x, y, z, distance_m)
position:geo3d_bbox(min_x, min_y, min_z, max_x, max_y, max_z)
position:geo3d_nearest(x, y, z, k)

Examples:

# Within 5 km of Tokyo Tower (ECEF coordinates)
position:geo3d_distance(-3955182, 3350553, 3700276, 5000)

# Inside an axis-aligned ECEF bounding box
position:geo3d_bbox(-4000000, 3300000, 3650000, -3900000, 3400000, 3750000)

# 10 nearest indexed points
position:geo3d_nearest(-3955182, 3350553, 3700276, 10)

The query field must be declared as a Geo3d field in the schema. See 3D Geographic Search for the underlying coordinate system, WGS84 conversion helpers, and detailed semantics.

Boost

Increase the weight of a clause with ^:

title:hello^2
"important phrase"^1.5

Grouping

Use parentheses for sub-expressions:

(title:hello OR title:hi) AND body:world

Lexical PEG Grammar

The full lexical grammar (parser.pest):

query          = { SOI ~ boolean_query ~ EOI }
boolean_query  = { clause ~ (boolean_op ~ clause | clause)* }
clause         = { required_clause | prohibited_clause | sub_clause }
required_clause   = { "+" ~ sub_clause }
prohibited_clause = { "-" ~ sub_clause }
sub_clause     = { grouped_query | field_query | term_query }
grouped_query  = { "(" ~ boolean_query ~ ")" ~ boost? }
boolean_op     = { ^"AND" | ^"OR" }
field_query    = { field ~ ":" ~ field_value }
field_value    = { geo3d_query | geo_query | range_query | phrase_query
                 | fuzzy_term | wildcard_term | simple_term }
geo3d_query    = { geo3d_distance | geo3d_bbox | geo3d_nearest }
geo3d_distance = { ^"geo3d_distance" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ signed_float ~ ")" }
geo3d_bbox     = { ^"geo3d_bbox" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ signed_float ~ ","
                 ~ signed_float ~ "," ~ signed_float ~ ")" }
geo3d_nearest  = { ^"geo3d_nearest" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ unsigned_int ~ ")" }
geo_query      = { geo_distance | geo_bbox }
geo_distance   = { ^"geo_distance" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ ")" }
geo_bbox       = { ^"geo_bbox" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ signed_float ~ ")" }
phrase_query   = { "\"" ~ phrase_content ~ "\"" ~ proximity? ~ boost? }
proximity      = { "~" ~ number }
fuzzy_term     = { term ~ "~" ~ fuzziness? ~ boost? }
wildcard_term  = { wildcard_pattern ~ boost? }
simple_term    = { term ~ boost? }
boost          = { "^" ~ boost_value }

Vector Query Syntax

Vector queries embed text into vectors at parse time and perform similarity search.

Basic Syntax

field:"text"
field:text
field:"text"^weight

The field name must refer to a vector field defined in the schema. The parser uses the schema to determine whether a clause is a vector query.

Vector Query Examples

# Single field (quoted text)
content:"cute kitten"

# Unquoted text
content:python

# With boost weight
content:"cute kitten"^0.8

# Multiple clauses
content:"cats" image:"dogs"^0.5

# Nested field name (dot notation)
metadata.embedding:"text"

Multiple Clauses

Multiple vector clauses are space-separated. All clauses are executed and their scores are combined using the score_mode (default: WeightedSum):

content:"cats" image:"dogs"^0.5

This produces:

score = similarity("cats", content) * 1.0
      + similarity("dogs", image)   * 0.5

There are no AND/OR operators in the vector DSL. Vector search is inherently a ranking operation, and the weight (^) controls the contribution of each clause.

Score Modes

Score mode cannot be set from DSL syntax. Use the Rust API to override:

#![allow(unused)]
fn main() {
let mut request = parser.parse(r#"content:"cats" image:"dogs""#).await?;
request.vector_options.score_mode = VectorScoreMode::MaxSim;
}

Vector PEG Grammar

The full vector grammar (parser.pest):

query          = { SOI ~ vector_clause+ ~ EOI }
vector_clause  = { field_prefix ~ (quoted_text | unquoted_text) ~ boost? }
field_prefix   = { field_name ~ ":" }
field_name     = @{ (ASCII_ALPHA | "_") ~ (ASCII_ALPHANUMERIC | "_" | ".")* }
quoted_text    = ${ "\"" ~ inner_text ~ "\"" }
inner_text     = @{ (!("\"") ~ ANY)* }
unquoted_text  = @{ (!(" " | "^" | "\"") ~ ANY)+ }
boost          = { "^" ~ float_value }
float_value    = @{ ASCII_DIGIT+ ~ ("." ~ ASCII_DIGIT+)? }

Unified (Hybrid) Query Syntax

The UnifiedQueryParser allows mixing lexical and vector clauses freely in a single query string:

title:hello content:"cute kitten"^0.8

How It Works

1. Split: The parser checks each field name against the schema. Fields defined as vector fields (e.g., HNSW, Flat, IVF) are routed to the vector parser; all other fields are routed to the lexical parser. The whole field name must match: with a vector field content, subcontent:x or my.content:x stays a lexical clause (Issue #1373).
2. Delegate: Vector portion goes to VectorQueryParser, remainder goes to lexical QueryParser.
3. Fuse: If both lexical and vector results exist, they are combined using a fusion algorithm.

Disambiguation

The parser uses the schema’s field type information to distinguish vector clauses from lexical clauses. A clause like content:"cute kitten" is a vector query if content is a vector field, or a phrase query if content is a text field. Lexical ~ syntax (e.g., roam~2 for fuzzy, "hello world"~10 for proximity) is unaffected.

Fusion Algorithms

When a query contains both lexical and vector clauses, results are fused:

Note: The fusion algorithm cannot be specified in the DSL syntax. It is configured when constructing the UnifiedQueryParser via .with_fusion(). The default is RRF (k=60). See Custom Fusion for a code example.

Hybrid AND/OR Semantics (the + Prefix)

By default, hybrid queries use union (OR) — documents appearing in either the lexical results or the vector results are included. You can switch to intersection (AND) by prefixing a vector clause with +, which requires documents to appear in both result sets.

Rules:

• When no vector clause carries the + prefix, the fusion produces a union (OR) of lexical and vector results.
• When at least one vector clause carries the + prefix, the fusion switches to intersection (AND) — only documents present in both the lexical and vector result sets are returned.
• + on a lexical field (e.g., +title:Rust) is interpreted by the lexical query parser as a required clause, which is the existing Tantivy/Lucene-style behaviour. It does not by itself trigger intersection mode for the hybrid fusion.

Unified Query Examples

# Lexical only — no fusion
title:hello AND body:world

# Vector only — no fusion
content:"cute kitten"

# Hybrid — fusion applied automatically (OR / union)
title:hello content:"cute kitten"

# Hybrid with AND / intersection — only docs in both result sets
title:hello +content:"cute kitten"

# Hybrid with boolean operators
title:hello AND category:animal content:"cute kitten"^0.8

# Multiple vector clauses + lexical
category:animal content:"cats" image:"dogs"^0.5

# Unquoted vector text
category:animal content:python

Code Examples

Lexical Search with DSL

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::analysis::analyzer::standard::StandardAnalyzer;
use laurus::lexical::query::QueryParser;

let analyzer = Arc::new(StandardAnalyzer::new()?);
let parser = QueryParser::new(analyzer)
    .with_default_field("title");

let query = parser.parse("title:hello AND body:world")?;
}

Vector Search with DSL

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::vector::query::VectorQueryParser;

let parser = VectorQueryParser::new(embedder)
    .with_default_field("content");

let request = parser.parse(r#"content:"cute kitten"^0.8"#).await?;
}

Hybrid Search with Unified DSL

#![allow(unused)]
fn main() {
use laurus::engine::query::UnifiedQueryParser;

let unified = UnifiedQueryParser::new(lexical_parser, vector_parser);

let request = unified.parse(
    r#"title:hello content:"cute kitten"^0.8"#
).await?;
// request.query              -> SearchQuery::Hybrid { lexical, vector }
// request.fusion_algorithm   -> Some(RRF)  — fusion algorithm
}

Custom Fusion

#![allow(unused)]
fn main() {
use laurus::engine::search::FusionAlgorithm;

let unified = UnifiedQueryParser::new(lexical_parser, vector_parser)
    .with_fusion(FusionAlgorithm::WeightedSum {
        lexical_weight: 0.3,
        vector_weight: 0.7,
    });
}

BKD-Tree

Laurus stores numeric, datetime, and geographic point data in a BKD-Tree (Block KD-Tree) — a disk-resident, multi-dimensional index that supports range, bounding-box, distance, and k-nearest-neighbour queries on the same underlying file format.

The BKD primitive is shared by every “spatial-shaped” field type:

Adding a new spatial field type therefore boils down to picking a dimensionality and writing a query-side IntersectVisitor — the writer, reader, and on-disk layout are reused unchanged.

File Format (Version 4)

A .bkd segment file is a self-contained binary blob made of three regions, followed by a checksum footer:

+----------------------------------------+
| File Header                            |   fixed-size, version-tagged
+----------------------------------------+
| Leaf Blocks                            |   bit-packed points + doc_ids
|   leaf 0                               |
|   leaf 1                               |
|   ...                                  |
+----------------------------------------+
| Index Nodes                            |   internal navigation nodes
|   node N-1                             |
|   ...                                  |
|   node 0  (root, written last)         |
+----------------------------------------+
| Footer                                 |   CRC-32 of all of the above
|                                        |   + magic "LCRC" (8 bytes)
+----------------------------------------+

The header records where the leaves and the index ended up, so the writer reserves it, writes everything after it, and fills it in last. The footer’s CRC still covers the bytes in file order (Issue #1214). A file written before the footer existed ends in a 4-byte trailer instead.

The header (BKDFileHeader) records magic, version (currently 4), num_dims, bytes_per_dim, the total point count, the number of leaf blocks, block_size (the writer’s configured max points per leaf, used to bound a leaf’s count on read — Issue #1142), the per-axis global min/max for the whole tree, and offsets to the index region and the root node.

Leaf Block Layout

Each leaf block stores the points that fall inside its subtree, bit-packed (Issue #549) instead of raw, and — since Issue #1142 — in doc_id-ascending order rather than the spatial split order the recursive build otherwise leaves them in:

count               u32               — number of points in the leaf
leaf_min            [f64; num_dims]   — leaf-level AABB minimum
leaf_max            [f64; num_dims]   — leaf-level AABB maximum
doc_id_base         u64               — the leaf's minimum doc_id
doc_id_bits         u8                — bit width of the largest consecutive doc_id gap
packed_dim[0]       byte-aligned      — bit-packed `sortable(point) - sortable(leaf_min[d])`
packed_dim[1..]     ...               — one section per dimension
packed_doc_ids      byte-aligned      — bit-packed consecutive deltas (see below)

Each dimension’s bit width is derived from leaf_min[d]/leaf_max[d] (mapped into an IEEE-754 total-order-preserving u64, the same order f64::total_cmp computes) rather than stored — the writer and reader compute it with the same formula, so it cannot drift between the two sides. A dimension that is constant across the whole leaf derives a 0-bit width and contributes no bytes at all.

packed_doc_ids stores consecutive deltas, not independent deltas from doc_id_base: value i is doc_id[i-1] + delta, with doc_id[-1] := doc_id_base, so the very first packed delta is always 0 (the reader treats a nonzero leading delta as corruption). doc_id_bits is therefore bits_needed(max consecutive gap) rather than bits_needed(max - min) — never larger (consecutive gaps always telescope-sum to exactly max - min), and often much smaller when a leaf’s doc_ids are locally clustered rather than spread evenly across the leaf’s full range. doc_id_bits is the one width actually stored on disk, since there is no equivalent “doc_id_max” header field to derive it from; the reader rejects a value greater than 64 as corruption. Sorting by doc_id before packing is a stable sort, so multiple points sharing a doc_id (a multi-valued numeric, datetime or geo field) keep their original relative order — this is depended on by GeoBoxPointsVisitor’s “first point seen wins” deduplication.

Measured space savings depend on how correlated the data is. Relative to the raw (pre-#549) leaf format: roughly 2.16x/1.67x/1.50x smaller for uniformly random 1D/2D/3D points, and around 2.66x for a monotonically increasing field (timestamps, auto-incrementing counters) — improved from the v3 (pre-#1142) figures of 1.96x/1.59x/1.45x/2.28x for the same data, because consecutive-delta doc_id packing captures locally-clustered doc_id distributions that independent-delta-from-a-single-anchor packing could not. This is a lossless, delta-based packing scheme, not quantization — every point round-trips bit-for-bit, including ±0.0 and ±Infinity.

The per-leaf AABB lets the reader prune the leaf without decoding any of its points when the query region is fully outside (Outside) or fully inside (Inside) the leaf; on Inside, the packed point sections are skipped with a single seek and only doc_ids are decoded.

Internal Index Node Layout

Internal nodes carry both the split decision and the per-child AABB:

split_dim           u32                       — axis to split on
split_value         f64                       — split threshold on that axis
left_min            [f64; num_dims]           — left subtree AABB min
left_max            [f64; num_dims]           — left subtree AABB max
right_min           [f64; num_dims]           — right subtree AABB min
right_max           [f64; num_dims]           — right subtree AABB max
left_offset         u64                       — file offset of left child
right_offset        u64                       — file offset of right child

Per-node AABBs (added in v2) replace the v1 layout that only carried the split value. They make Inside / Outside pruning a constant-time rectangle test instead of a recursive descent.

Build Algorithm

BKDWriter::write builds the tree from a flat row-major point buffer plus a parallel doc_ids buffer. Construction is driven by the widest-axis split heuristic:

1. Compute the AABB of the input subset.
2. Pick the axis whose (max - min) range is the widest (ties broken by lower dimension index for determinism).
3. Sort the index permutation by that axis and split at the median.
4. Recurse on the two halves until a subtree fits in block_size points (default 512); emit it as a leaf.
5. Back-patch each parent’s left_offset / right_offset once the children have been flushed.

The builder sorts an index permutation rather than the point/doc-id buffers themselves, so it does not allocate per-point storage no matter how many points are written.

Numeric Robustness

Coordinates must be totally orderable. BKDWriter::write rejects NaN explicitly because NaN has no defined ordering and would corrupt the split decisions and per-node AABB invariants. Both ±INFINITY are accepted and act as natural sentinels for “unbounded” semantics in queries.

Query: The IntersectVisitor Protocol

Queries against a BKD index are expressed as an IntersectVisitor implementation. The reader walks the tree and asks the visitor three things:

#![allow(unused)]
fn main() {
pub enum CellRelation {
    Inside,   // entire subtree is a hit — collect without per-point checks
    Outside,  // entire subtree can be skipped
    Crosses,  // recurse, or filter the leaf per-point
}

pub trait IntersectVisitor {
    fn compare(&self, cell: &AABB) -> CellRelation;
    fn visit_inside(&mut self, doc_id: u64);
    fn visit(&mut self, doc_id: u64, point: &[f64]);
}
}

The reader’s traversal is:

graph TD
    A["compare(node.aabb)"]
    A -->|Inside| B["visit_inside(doc_id) for every doc<br/>under the subtree — no point read"]
    A -->|Outside| C["skip subtree"]
    A -->|Crosses, internal| D["recurse into children"]
    A -->|Crosses, leaf| E["visit(doc_id, point) per point<br/>visitor decides what is a hit"]

This three-valued classification is what unlocks pruning. A visitor that always returns Crosses would still produce correct results — it would just degrade to a full leaf scan.

Range Queries

The legacy BKDTree::range_search API is now a thin wrapper around intersect: it constructs a RangeQueryVisitor from the half-open / closed range parameters and converts unbounded None slots into ±INFINITY. The visitor handles inclusive-vs-exclusive boundary checks itself.

3D Geographic Queries

Three additional visitors live in laurus::lexical::query::geo3d and target Geo3d (3D ECEF) fields:

The same primitive could host any future spatial query — for example, polygon queries or great-circle 2D Geo queries — by writing a new visitor.

Reader Internals

BKDReader::intersect uses a single per-query scratch buffer (IntersectScratch) that grows to the size of the largest leaf encountered and is then reused for every subsequent leaf, so a query touches the allocator at most a handful of times regardless of how many leaves it visits.

Single-leaf trees (very small fields) are handled as a special case: the “root offset” points directly at the only leaf and the reader skips the internal-node descent entirely.

See Also

• 3D Geographic Search — concrete BKD-backed visitors for ECEF distance, bounding-box, and k-NN queries.
• Lexical Indexing — where the .bkd segment file fits within the broader segment layout.
• Lexical Search — NumericRangeQuery, GeoDistanceQuery / GeoBoundingBoxQuery, and Geo3dDistanceQuery programmatic entry points.

3D Geographic Search (ECEF)

Laurus offers two geographic field types that target different problems:

Both share the same BKD-Tree primitive; Geo3d simply adds a third dimension and a different query vocabulary.

Why ECEF?

(latitude, longitude, altitude) triples are convenient for humans but make Euclidean distance unusable: a degree of longitude is ~111 km at the equator and 0 km at the poles, and “altitude” is curved with the Earth’s surface.

ECEF (Earth-Centered Earth-Fixed) flattens this into a Cartesian frame:

• The origin is the Earth’s centre of mass.
• The +X axis points through the equator at 0° longitude.
• The +Y axis points through the equator at 90° E longitude.
• The +Z axis points through the geographic North Pole.
• All three axes are in metres.

In this frame the straight-line distance between two points is just the Euclidean norm — no spherical trigonometry, no pole singularity, no longitude-wrap. That property is what makes Geo3d queries usable from a 3D BKD-Tree without bespoke spatial code.

Defining a Geo3d Field

#![allow(unused)]
fn main() {
use laurus::Schema;
use laurus::lexical::core::field::Geo3dOption;

let schema = Schema::builder()
    .add_geo3d_field("position", Geo3dOption::default())
    .build();
}

Geo3dOption exposes the same indexed / stored switches as the other lexical field options. Indexed values populate the field’s 3D BKD-Tree; stored values are returned by get_documents.

The TOML form (used by laurus-cli):

[fields.position.Geo3d]
indexed = true
stored = true

Indexing 3D Points

Use DocumentBuilder::add_geo_ecef to add a point in raw Cartesian metres:

#![allow(unused)]
fn main() {
use laurus::Document;

// A point at lat=35.6586°, lon=139.7454°, height=250 m
// (already converted to ECEF — see "Coordinate Conversion" below).
let doc = Document::builder()
    .add_geo_ecef("position", -3_955_182.0, 3_350_553.0, 3_700_276.0)
    .build();

engine.put_document("tokyo-tower", doc).await?;
engine.commit().await?;
}

If the value already exists as a GeoEcefPoint, build it through the unified DataValue API instead:

#![allow(unused)]
fn main() {
use laurus::{DataValue, GeoEcefPoint};

let p = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);
let doc = Document::builder()
    .add_field("position", DataValue::GeoEcef(p))
    .build();
}

Coordinate Conversion (WGS84 ↔ ECEF)

Most input arrives as (latitude°, longitude°, height_m). Laurus exposes the canonical pair of conversions in laurus::util::ecef:

#![allow(unused)]
fn main() {
use laurus::util::ecef::{wgs84_to_ecef, ecef_to_wgs84};

// Forward: lat/lon/height → ECEF
let p = wgs84_to_ecef(35.6586, 139.7454, 250.0);

// Reverse: ECEF → lat/lon/height
let (lat, lon, height) = ecef_to_wgs84(&p);
}

The reverse conversion round-trips with sub-µm precision on every axis from sub-surface up to far beyond LEO altitudes. Near the poles the implementation switches from the standard h = p / cos(lat) - N formula (which diverges at cos(lat) → 0) to the meridian form h = z / sin(lat) - N(1 - e²).

The WGS84 ellipsoid constants used internally are also exposed as pub consts (WGS84_A, WGS84_F, WGS84_B, WGS84_E2, WGS84_E_PRIME_SQ) so external code can reference the exact values laurus uses.

Query Types

Three query types target Geo3d fields. All three are IntersectVisitor implementations on top of the shared 3D BKD-Tree.

Sphere (radius) — Geo3dDistanceQuery

Find every document whose stored point lies within distance_m metres of a centre point. Score is 1 - distance / radius, clamped to [0, 1].

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo3d::Geo3dDistanceQuery;
use laurus::GeoEcefPoint;

let centre = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);
let query = Geo3dDistanceQuery::new("position", centre, 5_000.0); // 5 km radius
}

The visitor uses AABB::min_distance_sq_to_point for the Outside test and AABB::max_distance_sq_to_point for the Inside test, so both cell-vs-sphere checks run in squared-distance space and avoid sqrt.

Bounding Box — Geo3dBoundingBoxQuery

Find every document whose stored point lies inside an axis-aligned 3D box defined by (min_x, min_y, min_z) and (max_x, max_y, max_z).

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo3d::Geo3dBoundingBoxQuery;
use laurus::GeoEcefPoint;

let min = GeoEcefPoint::new(-4_000_000.0, 3_300_000.0, 3_650_000.0);
let max = GeoEcefPoint::new(-3_900_000.0, 3_400_000.0, 3_750_000.0);
let query = Geo3dBoundingBoxQuery::new("position", min, max)?;
}

The constructor returns Result because every min[i] must be ≤ max[i]; mismatched bounds are rejected at construction time rather than producing silently-empty results.

Caveat. The query box is axis-aligned in ECEF Cartesian space. A box built from two (lat, lon, height) corners by converting each corner separately is not the same shape as the spherical-shell region a user might intuitively expect. For region queries on the Earth’s surface, Geo3dDistanceQuery (a true 3D sphere) usually matches user intuition better.

k-Nearest Neighbours — Geo3dNearestQuery

Find the k nearest documents to a query point. The query starts from a small probe sphere (default 1 km) and doubles the radius until either:

1. it has collected at least k distinct candidates,
2. doubling makes no progress,
3. the radius reaches max_radius_m (default 1.0e10 m, i.e. 10⁷ km).

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo3d::Geo3dNearestQuery;
use laurus::GeoEcefPoint;

let centre = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);
let query = Geo3dNearestQuery::new("position", centre, 10) // 10 nearest
    .with_initial_radius(500.0)         // start at 500 m
    .with_max_radius(1_000_000.0);      // cap at 1 000 km
}

A smaller-than-k result simply means the index does not contain k points within max_radius_m. Score is normalised so that the closest hit gets 1.0 and the farthest hit in the returned set gets 0.0.

The k-NN visitor never returns CellRelation::Inside from compare — only Outside or Crosses — so every candidate is delivered to visit with its exact coordinates, ensuring k-NN ordering uses the true distance.

Query DSL

Geo3d queries are also exposed in the unified Query DSL (see Query DSL → Geo3d Functions):

position:geo3d_distance(-3955182, 3350553, 3700276, 5000)
position:geo3d_bbox(-4000000, 3300000, 3650000, -3900000, 3400000, 3750000)
position:geo3d_nearest(-3955182, 3350553, 3700276, 10)

All numeric arguments are signed floats; the k of geo3d_nearest is an unsigned integer.

Wire Formats

gRPC

Protocol Buffers expose 3D geo as a dedicated Value variant alongside a Geo3dPoint message and a Geo3dOption schema option:

message Geo3dPoint {
  double x = 1;
  double y = 2;
  double z = 3;
}

message Value {
  oneof kind {
    // ...
    Geo3dPoint geo3d_value = 12;
  }
}

message Geo3dOption {
  bool indexed = 1;
  bool stored = 2;
}

See common.proto and index.proto for the full definitions.

HTTP gateway / MCP

JSON documents accept Geo3d values as a { "x": …, "y": …, "z": … } object:

{
  "position": { "x": -3955182.0, "y": 3350553.0, "z": 3700276.0 }
}

The HTTP gateway converts this to DataValue::GeoEcef before forwarding to the engine; the MCP server uses the same convention.

When not to Use Geo3d

• Pure 2D map queries (radius from a coordinate on the surface, simple bounding boxes on a tile map) — keep using Geo. The 2D BKD is one dimension lighter and the WGS84 input matches user intuition.
• Approximate semantic similarity of a learned location embedding — that is a vector field (Hnsw, Flat, Ivf), not a Geo3d field.

See Also

• BKD-Tree — the underlying multi-dimensional index that also backs Integer, Float, DateTime, and 2D Geo fields.
• Schema & Fields — Geo3d alongside the rest of the field type table.
• Query DSL — full geo3d DSL grammar.
• Lexical Search — programmatic entry points for every lexical query type.
• laurus-wasm/examples/geo3d/ — browser-side demo: live satellite positions on a CesiumJS 3D globe with geo3d_bbox and geo3d_nearest queries.

Library Overview

The laurus crate is the core search engine library. It provides lexical search (keyword matching via inverted index), vector search (semantic similarity via embeddings), and hybrid search (combining both) through a unified API.

Module Structure

graph TB
    LIB["laurus (lib.rs)"]

    LIB --> engine["engine\nEngine, EngineBuilder\nSearchRequest, FusionAlgorithm"]
    LIB --> analysis["analysis\nAnalyzer, Tokenizer\nToken Filters, Char Filters"]
    LIB --> lexical["lexical\nInverted Index, BM25\nQuery Types, Faceting, Highlighting"]
    LIB --> vector["vector\nFlat, HNSW, IVF\nDistance Metrics, Quantization"]
    LIB --> embedding["embedding\nCandle BERT, OpenAI\nCLIP, Precomputed"]
    LIB --> storage["storage\nMemory, File, Mmap\nColumnStorage"]
    LIB --> store["store\nDocumentLog (WAL)"]
    LIB --> spelling["spelling\nSpelling Correction\nSuggestion Engine"]
    LIB --> util["util\nShared helpers and macros"]
    LIB --> data["data\nDataValue, Document"]
    LIB --> error["error\nLaurusError, Result"]

Key Types

Feature Flags

The laurus crate enables the native feature by default. Enable embedding support as needed:

# Lexical search only (no embedding)
[dependencies]
laurus = "0.13"

# With local BERT embeddings
[dependencies]
laurus = { version = "0.13", features = ["embeddings-candle"] }

# All features
[dependencies]
laurus = { version = "0.13", features = ["embeddings-all"] }

Sections

• Engine – Engine and EngineBuilder internals
• Scoring & Ranking – BM25, TF-IDF, and vector similarity scoring
• Faceting – Hierarchical facet search
• Highlighting – Search result highlighting
• Spelling Correction – Spelling suggestions and auto-correction
• ID Management – Dual-tiered document identity
• Persistence & WAL – Write-ahead logging and durability
• Deletions & Compaction – Logical deletion and space reclamation
• Error Handling – LaurusError and Result types
• Extensibility – Custom analyzers, embedders, and storage backends
• API Reference – Key types and methods at a glance

Engine

The Engine is the central type in Laurus. It coordinates the lexical index, vector index, and document log behind a single async API.

Engine Struct

#![allow(unused)]
fn main() {
pub struct Engine {
    schema: Schema,
    lexical: LexicalStore,
    vector: VectorStore,
    log: Arc<DocumentLog>,
}
}

EngineBuilder

Use EngineBuilder to configure and create an Engine:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::{Engine, Schema};
use laurus::lexical::TextOption;
use laurus::storage::memory::MemoryStorage;

let storage = Arc::new(MemoryStorage::new(Default::default()));
let schema = Schema::builder()
    .add_text_field("title", TextOption::default())
    .add_text_field("body", TextOption::default())
    .add_default_field("body")
    .build();

let engine = Engine::builder(storage, schema)
    .analyzer(my_analyzer)            // optional: custom text analyzer
    .embedder(my_embedder)            // optional: vector embedder
    .embedding_cache_capacity(1024)   // optional: cache query embeddings
    .build()
    .await?;
}

Builder Methods

Query embedding cache

embedding_cache_capacity(n) enables an LRU cache for query-time embeddings (document-ingestion embedding is unaffected). When a vector or hybrid search embeds a query payload, the result is cached keyed by (field, embedder name, payload hash) and reused on subsequent identical queries — both the DSL path (e.g. content:"cute kitten") and the pre-embedded Payloads path share the same cache.

This avoids repeated model inference for local embedders, or network round trips for remote ones, on repeated-query workloads (autocomplete, dashboard refreshes, A/B evaluation). It is disabled by default; pick a capacity that bounds memory to your working set of distinct queries.

Within a single query, all of its payloads are embedded in one Embedder::embed_batch call (cache misses only, grouped per field for PerFieldEmbedder), so a batch-capable embedder pays one round trip for a multi-vector query instead of one per payload (Issue #671).

The token vectors of a late-interaction rescore’s text query are cached too, in a separate LRU of the same capacity, keyed by field, the field’s own embedder, role and query text, so their larger entries never evict single-vector ones (Issue #1349). Adding, deleting or rebuilding a vector field empties both, since the embedder behind a field name may have changed.

Build Lifecycle

When build() is called, the following steps occur:

sequenceDiagram
    participant User
    participant EngineBuilder
    participant Engine

    User->>EngineBuilder: .build().await
    EngineBuilder->>EngineBuilder: split_schema()
    Note over EngineBuilder: Separate fields into<br/>LexicalIndexConfig<br/>+ VectorIndexConfig
    EngineBuilder->>Engine: Create PrefixedStorage (lexical/, vector/, documents/)
    EngineBuilder->>Engine: Create LexicalStore
    EngineBuilder->>Engine: Create VectorStore
    EngineBuilder->>Engine: Create DocumentLog
    EngineBuilder->>Engine: Recover from WAL
    EngineBuilder-->>User: Engine ready

1. Split schema – Lexical fields (Text, Integer, Float, etc.) go to LexicalIndexConfig, vector fields (HNSW, Flat, IVF) go to VectorIndexConfig
2. Create prefixed storage – Each component gets an isolated namespace (lexical/, vector/, documents/)
3. Initialize stores – LexicalStore and VectorStore are created with their respective configs
4. Recover from WAL – Any uncommitted operations from a previous session are replayed

Schema Splitting

The Schema contains both lexical and vector fields. At build time, split_schema() separates them:

graph LR
    S["Schema<br/>title: Text<br/>body: Text<br/>page: Integer<br/>content_vec: HNSW"]

    S --> LC["LexicalIndexConfig<br/>title: TextOption<br/>body: TextOption<br/>page: IntegerOption<br/>_id: KeywordAnalyzer"]

    S --> VC["VectorIndexConfig<br/>content_vec: HnswOption<br/>(dim=384, m=16, ef=200)"]

The reserved _id field is always added to the lexical config with KeywordAnalyzer for exact match lookups.

Per-Field Dispatch

PerFieldAnalyzer

When a PerFieldAnalyzer is provided, text analysis is dispatched to field-specific analyzers:

graph LR
    PFA["PerFieldAnalyzer"]
    PFA -->|"title"| KA["KeywordAnalyzer"]
    PFA -->|"body"| SA["StandardAnalyzer"]
    PFA -->|"description"| JA["JapaneseAnalyzer"]
    PFA -->|"_id"| KA2["KeywordAnalyzer<br/>(always)"]
    PFA -->|other fields| DEF["Default Analyzer"]

PerFieldEmbedder

Similarly, PerFieldEmbedder routes embedding to field-specific embedders:

graph LR
    PFE["PerFieldEmbedder"]
    PFE -->|"text_vec"| BERT["CandleBertEmbedder<br/>(384 dim)"]
    PFE -->|"image_vec"| CLIP["CandleClipEmbedder<br/>(512 dim)"]
    PFE -->|other fields| DEF["Default Embedder"]

Engine Methods

Document Operations

Search

The search() method accepts a SearchRequest which can contain a lexical query, a vector query, or both. When both are present, results are merged using the specified FusionAlgorithm.

#![allow(unused)]
fn main() {
use laurus::{SearchRequestBuilder, FusionAlgorithm};
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

// Lexical-only search
let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust")))
    )
    .limit(10)
    .build();

// Hybrid search with RRF fusion
let request = SearchRequestBuilder::new()
    .lexical_query(lexical_query)
    .vector_query(vector_query)
    .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

Warmup

engine.warmup() (Issue #677) primes the vector searcher so the first vector / hybrid query does not pay one-time setup costs. Call it once after building the engine, before serving traffic:

let engine = builder.build()?;
engine.warmup()?;   // optional: move first-query latency to startup

It eagerly builds and caches the vector searcher — loading the reader (file → memory for InMemory, the offset table for Mmap) — and, for HNSW indexes in Mmap mode, pre-faults the on-disk vector data into the OS page cache. The HNSW graph is always loaded eagerly, so only the vector data needs warming. warmup() is safe to call multiple times and is a no-op for lexical-only workloads.

SearchRequest

LexicalSearchOptions

FusionAlgorithm

See also: Architecture for the high-level data flow diagrams.

Scoring & Ranking

Laurus uses BM25 for lexical search, distance-based similarity for vector search, and a configurable fusion algorithm to combine the two for hybrid search. This page describes each scoring path and how to influence it from the public API.

Lexical Scoring

BM25 (Default)

BM25 is the lexical scoring function. It balances term frequency with document length normalization:

score = IDF * (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (doc_len / avg_doc_len)))

Where:

• tf — term frequency in the document.
• IDF — inverse document frequency (rarity of the term across all documents).
• k1 — term-frequency saturation parameter. Laurus uses 1.2.
• b — document-length normalization factor. Laurus uses 0.75.
• doc_len / avg_doc_len — ratio of document length to average document length. doc_len counts field positions, not tokens: a synonym the analyzer stacks on another token (position_increment = 0) shares its position and does not add to the length, matching Lucene’s discountOverlaps.

The (k1, b) parameters are fixed at the implementation defaults today. The values match Lucene / Elasticsearch defaults, so BM25 scores from Laurus are directly comparable to those engines for tuning intuition.

A quoted or bare value whose analyzer stacks several alternatives at one position (synonyms) is scored as one blended term rather than as a sum of independent term scores, so a document is not rewarded for holding every alternative — see Searching with synonyms.

Field Boosts

Per-field score multipliers are configured on the search request, not in a separate scoring struct:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;

let request = SearchRequestBuilder::new()
    .query_dsl("rust programming")
    .add_field_boost("title", 2.0) // title matches score 2x
    .add_field_boost("body", 1.0)  // body matches score 1x (the default)
    .limit(10)
    .build();
}

The boost is multiplied into the BM25 score contribution of matches in that field. A boost of 1.0 is a no-op; boosts apply only to fields named in the query (or in the schema’s default-search fields). The exception is a CombinedFields multi-field query, where a boost is a BM25F weight inside the score instead.

Over gRPC and HTTP, the same setting is exposed as SearchRequest.field_boosts (map<string, float>). See gRPC API → SearchRequest.

Multi-field Queries

MultiFieldQuery searches one term, unanalyzed like a TermQuery, across several fields, each with its own boost. Its MultiFieldQueryType decides how the fields combine:

BestFields and MostFields score each field with that field’s own statistics. A term that is rare in one field gets a high IDF there, so the same text scores differently depending on which field it lands in. CombinedFields blends the statistics and scores each document once, with BM25 over:

tf          = Σ w_f · tf_f                     (the document's term frequency in each field)
doc_len     = Σ w_f · len_f                    (every field, whether the term occurs in it or not)
avg_doc_len = Σ w_f · avg_len_f · doc_count_f / max(doc_count_f)
df          = max(df_f)                        (for the IDF)

Here w_f is the field’s boost. It acts as a weight inside BM25F: 2.0 counts the field as if its text were indexed twice, rather than multiplying the field’s score. Every boost must be finite and greater than 0.0; any other value makes the search return InvalidArgument. Boosts below 1.0 are accepted, although Lucene and Elasticsearch reject them.

As with TermQuery, the document frequency covers the whole index, while on a multi-segment index the average length is each segment’s own.

#![allow(unused)]
fn main() {
use laurus::lexical::query::advanced_query::{MultiFieldQuery, MultiFieldQueryType};

let query = MultiFieldQuery::new("smith".to_string())
    .add_field("first_name".to_string(), 1.0)
    .add_field("last_name".to_string(), 1.0)
    .query_type(MultiFieldQueryType::CombinedFields);
}

Vector Scoring

Vector search ranks results by distance-based similarity. The distance metric is configured per field on the vector index (HNSW / Flat / IVF):

Distances are converted to similarity scores so that “higher is better” holds across both lexical and vector results, which is what the fusion algorithms below assume.

Hybrid Search Fusion

When a search request contains both lexical and vector clauses, the two result lists need to be merged. Laurus exposes two fusion algorithms via FusionAlgorithm.

RRF (Reciprocal Rank Fusion)

RRF avoids score normalisation entirely by combining ranks instead of raw scores:

rrf_score(doc) = Σ 1 / (k + rank_i(doc))

The sum runs over each result list the document appears in. The k parameter (default 60.0) smooths the distribution — higher k flattens the contribution of top-ranked results.

#![allow(unused)]
fn main() {
use laurus::{FusionAlgorithm, SearchRequestBuilder};

let request = SearchRequestBuilder::new()
    .query_dsl("title:rust ~\"systems programming\"")
    .fusion_algorithm(FusionAlgorithm::Rrf { k: 60.0 })
    .build();
}

WeightedSum

WeightedSum first min-max normalises each list of scores independently, then takes a weighted linear combination:

norm(score)  = (score - min) / (max - min)
final(doc)   = lexical_weight * norm(lexical_score(doc))
             + vector_weight  * norm(vector_score(doc))

#![allow(unused)]
fn main() {
use laurus::{FusionAlgorithm, SearchRequestBuilder};

let request = SearchRequestBuilder::new()
    .query_dsl("title:rust ~\"systems programming\"")
    .fusion_algorithm(FusionAlgorithm::WeightedSum {
        lexical_weight: 0.6,
        vector_weight: 0.4,
    })
    .build();
}

Both weights are clamped to [0.0, 1.0]. Use RRF when you do not have a calibrated reason to pick specific weights — it is parameter-light and robust to scale differences between lists.

See Also

• API Reference → FusionAlgorithm — variant signatures
• Hybrid Search — when to pick which fusion algorithm
• Vector Search — distance metric trade-offs

Faceting

Faceting enables counting and categorizing search results by field values. It is commonly used to build navigation filters in search UIs (e.g., “Electronics (42)”, “Books (18)”).

Concepts

FacetPath

A FacetPath represents a hierarchical facet value. For example, a product category “Electronics > Computers > Laptops” is a facet path with three levels.

#![allow(unused)]
fn main() {
use laurus::lexical::search::features::facet::FacetPath;

// Single-level facet
let facet = FacetPath::from_value("category".into(), "Electronics".into());

// Hierarchical facet from components
let facet = FacetPath::new("category".into(), vec![
    "Electronics".to_string(),
    "Computers".to_string(),
    "Laptops".to_string(),
]);

// From a delimited string
let facet = FacetPath::from_delimited("category".into(), "Electronics/Computers/Laptops", "/");
}

from_delimited drops empty components, as the collector does, so "/Electronics//Computers" gives ["Electronics", "Computers"].

FacetPath Methods

FacetCount

FacetCount represents the result of a facet aggregation:

#![allow(unused)]
fn main() {
pub struct FacetCount {
    pub path: FacetPath,
    pub count: u64,
    pub children: Vec<FacetCount>,
}
}

FacetConfig

FacetConfig controls what the collector counts and returns:

Values that no collected document has are never returned, so a min_count of 0 behaves like 1.

Collecting facets

Feed each matching document to a FacetCollector, then call finalize:

#![allow(unused)]
fn main() {
use laurus::lexical::search::features::facet::{FacetCollector, FacetConfig};

let mut collector = FacetCollector::new(FacetConfig::default(), vec!["category".to_string()]);
for doc_id in matching_doc_ids {
    collector.collect_doc(doc_id, reader.as_ref())?;
}
let results = collector.finalize()?;

for facet in results.get_field_facets("category").into_iter().flatten() {
    println!("{} ({})", facet.path.to_string_with_delimiter("/"), facet.count);
}
}

collect_doc returns an error when a facet field has to be read from the stored document and that read fails. The collector’s counts are incomplete after such an error, so discard it.

Hierarchical facets

A Text value containing / is a hierarchical path: Electronics/Computers/Laptops has three levels. The collector counts the path and each of its ancestors, once per document, and finalize returns one tree per field: the top-level values in get_field_facets(field), and each deeper level in the children of its parent.

Category
├── Electronics (42)
│   ├── Computers (18)
│   │   ├── Laptops (12)
│   │   └── Desktops (6)
│   └── Phones (24)
└── Books (35)
    ├── Fiction (20)
    └── Non-Fiction (15)

A node’s count is the number of documents whose value is that path or lies under it, so a child never outnumbers its parent. A flat value and the root of a hierarchical one are the same node. For example, one document with cat = "a" and another with cat = "a/b" give a single a (2) with one child b (1).

Each level is filtered, sorted and truncated on its own:

• min_count drops a value together with its whole subtree.
• max_facets_per_field applies to the top level and to the children of each node separately, so an ancestor never uses up its descendants’ budget.
• Values with equal counts are ordered by label, so which values are kept never depends on hash order. This matches Lucene and Tantivy.

Empty components are dropped: "/a/b" and "a//b" both give a/b, "a/" gives a, and "" or "/" count nothing. Lucene rejects such components when indexing; here facets are derived at search time from ordinary text values, so they are dropped instead of failing the search.

Use Cases

• E-commerce: Filter by category, brand, price range, rating
• Document search: Filter by author, department, date range, document type
• Content management: Filter by tags, topics, content status

Multi-valued fields

A multi-valued field (multi_valued = true, see Multi-valued fields) stores an array per document, and the collector expands it: every element becomes its own facet path (Issue #1187). A TextArray element containing / is split into a hierarchical path exactly like a scalar Text value, so tags = ["rust", "search"] counts rust and search once each, and cat = ["a/b", "a/c"] counts a/b, a/c and their shared ancestor a.

Counts are per document, following Lucene’s SortedSetDocValuesFacetCounts: an element that appears twice in one document (["rust", "rust"]) counts once, and so does an ancestor reached from two elements (a above is 1, not 2). FacetCount::count is therefore always the number of matching documents. An empty array contributes nothing.

Array elements are rendered exactly like the scalar of the same type:

Null, geo points (Geo, GeoEcef and their arrays), Vector and Bytes are not facetable and contribute nothing. A DocValues hit that yields no facet value does not fall back to the stored document.

Performance

Facet counts are read from each field’s DocValues column, not from the stored document. For every collected hit the collector reads only the facet field’s value via the per-field DocValues lookup, so it never decodes or clones the whole stored-fields blob when every faceted field has a DocValues column (the default for any stored: true field, unless its type is excluded — see below — or its doc_values option is explicitly set to false). A field that lacks DocValues — because it opted out, isn’t stored, or is a Bytes/Vector value, which DocValues never carries regardless of the setting — transparently falls back to the stored document, so results are identical either way; only the read path changes. Array values of multi-valued fields are stored whole in DocValues (one entry per document) and split into elements at facet time, so expansion adds no DocValues reads.

Setting doc_values: false on a field that is never sorted or faceted on shrinks its segment footprint, since the value is then written once (to the stored document) instead of twice.

Highlighting

Highlighting marks matching terms in search results, helping users see why a document matched their query. Laurus generates highlighted text fragments with configurable HTML tags.

Highlighting search results

The easiest way to get highlights is through the search API itself: ask for them on a SearchRequest, and each hit’s SearchResult::highlights comes back filled in.

#![allow(unused)]
fn main() {
use laurus::{HighlightConfig, SearchRequestBuilder};

let request = SearchRequestBuilder::new()
    .lexical_query(query)
    .highlight(vec!["body".to_string()])
    .highlight_config(HighlightConfig::default().tag("em".to_string()))
    .build();

let results = engine.search(request).await?;
for result in &results {
    if let Some(fragments) = result.highlights.get("body") {
        println!("{}", fragments.join(" ... "));
    }
}
}

highlight(fields) and highlight_config(config) are independent SearchRequestBuilder methods and compose in either order; calling highlight again replaces the field list, and the same holds for highlight_config. Omitting highlight (or requesting no fields) leaves every hit’s highlights map empty — the engine does no highlighting work in that case.

Semantics to keep in mind:

• Field selection. Only fields named in highlight(fields) are considered, and only when they are stored: true text fields. A field that is absent from the document, not stored, or not a text field is silently skipped — it never appears as a key in highlights. A multi-valued text field (multi_valued: true, Issue #1175) is highlighted element by element: only the elements that match contribute fragments, a fragment never straddles two elements, and the concatenated fragment list honours max_fragments.
• Analyzer. Each field is tokenized with its own index-time analyzer (per-field analyzers apply automatically), so highlighting reflects what actually matched at index time — not a generic tokenizer.
• Which query is highlighted. The request’s lexical query drives highlighting, including in hybrid search — a vector-only request produces no highlights. Terms contributed only by the request-level filter_query are never highlighted, since they describe eligibility, not relevance to what was searched for.
• Empty result. A field with no highlight fragments (no match, or return_entire_field_if_no_highlight not set) is omitted from highlights entirely rather than mapped to an empty list.
• Cost. Highlighting runs after pagination, so its cost scales with limit × len(fields), not with the total number of matches.

For direct control over highlighting outside the search API — for example to highlight text that never went through Engine::search — use Highlighter as described below.

HighlightConfig

HighlightConfig controls how highlights are generated:

#![allow(unused)]
fn main() {
use laurus::lexical::search::features::highlight::HighlightConfig;

let config = HighlightConfig::default()
    .tag("mark")
    .css_class("highlight")
    .max_fragments(3)
    .fragment_size(200);
}

Configuration Options

Builder Methods

Supported Queries

Highlight terms come from the query tree (Query::collect_highlight_terms), not from the query’s description string, and are matched against the tokens produced by the highlighter’s analyzer (StandardAnalyzer by default; use Highlighter::with_analyzer to match the field’s analyzer, for example for Japanese text).