Laurus

A fast, featureful hybrid search library for Rust.

Laurus is a pure-Rust library that combines lexical search (keyword matching via inverted index) and vector search (semantic similarity via embeddings) into a single, unified engine. It is designed to be embedded directly into your Rust application — no external server required.

Key Features

How It Works

graph LR
    subgraph Your Application
        D["Document"]
        Q["Query"]
    end

    subgraph Laurus Engine
        SCH["Schema"]
        AN["Analyzer"]
        EM["Embedder"]
        LI["Lexical Index\n(Inverted Index)"]
        VI["Vector Index\n(HNSW / Flat / IVF)"]
        FU["Fusion\n(RRF / WeightedSum)"]
    end

    D --> SCH
    SCH --> AN --> LI
    SCH --> EM --> VI
    Q --> LI --> FU
    Q --> VI --> FU
    FU --> R["Ranked Results"]

1. Define a Schema — declare your fields and their types (text, integer, vector, etc.)
2. Build an Engine — attach an analyzer for text and an embedder for vectors
3. Index Documents — the engine routes each field to the correct index automatically
4. Search — run lexical, vector, or hybrid queries and get ranked results

Document Map

Quick Example

use std::sync::Arc;
use laurus::{Document, Engine, Schema, SearchRequestBuilder, Result};
use laurus::lexical::{TextOption, TermQuery};
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> Result<()> {
    // 1. Storage
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    // 2. Schema
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_default_field("body")
        .build();

    // 3. Engine
    let engine = Engine::builder(storage, schema).build().await?;

    // 4. Index a document
    let doc = Document::builder()
        .add_text("title", "Hello Laurus")
        .add_text("body", "A fast search library for Rust")
        .build();
    engine.add_document("doc-1", doc).await?;
    engine.commit().await?;

    // 5. Search
    let request = SearchRequestBuilder::new()
        .lexical_search_request(
            laurus::LexicalSearchRequest::new(
                Box::new(TermQuery::new("body", "rust"))
            )
        )
        .limit(10)
        .build();
    let results = engine.search(request).await?;

    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }
    Ok(())
}

License

Laurus is licensed under the MIT License.

Architecture

This page explains how Laurus is structured internally. Understanding the architecture will help you make better decisions about schema design, analyzer selection, and search strategies.

Project Structure

Laurus is organized as a Cargo workspace with five crates:

graph TB
    CLI["laurus-cli\n(Binary Crate)\nCLI + REPL"]
    SRV["laurus-server\n(Library + Binary)\ngRPC Server + HTTP Gateway"]
    MCP["laurus-mcp\n(Library + Binary)\nMCP Server"]
    PY["laurus-python\n(cdylib)\nPython Bindings"]
    LIB["laurus\n(Library Crate)\nCore Search Engine"]

    CLI --> LIB
    CLI --> SRV
    CLI --> MCP
    SRV --> LIB
    MCP --> SRV
    MCP --> LIB
    PY --> LIB

For details on each crate, see:

• Library Overview
• CLI Overview
• Server Overview
• MCP Server Overview
• Python Bindings Overview
• Node.js Bindings Overview
• WASM Bindings Overview
• Ruby Bindings Overview
• PHP Bindings Overview

High-Level Overview

Laurus is organized around a single Engine that coordinates four internal components:

graph TB
    subgraph Engine
        SCH["Schema"]
        LS["LexicalStore\n(Inverted Index)"]
        VS["VectorStore\n(HNSW / Flat / IVF)"]
        DL["DocumentLog\n(WAL + Document Storage)"]
    end

    Storage["Storage (trait)\nMemory / File / File+Mmap"]

    LS --- Storage
    VS --- Storage
    DL --- Storage

All three stores share a single Storage backend, isolated by key prefixes (lexical/, vector/, documents/).

Engine Lifecycle

Building an Engine

The EngineBuilder assembles the engine from its parts:

#![allow(unused)]
fn main() {
let engine = Engine::builder(storage, schema)
    .analyzer(analyzer)      // optional: for text fields
    .embedder(embedder)      // optional: for vector fields
    .build()
    .await?;
}

sequenceDiagram
    participant User
    participant EngineBuilder
    participant Engine

    User->>EngineBuilder: new(storage, schema)
    User->>EngineBuilder: .analyzer(analyzer)
    User->>EngineBuilder: .embedder(embedder)
    User->>EngineBuilder: .build().await
    EngineBuilder->>EngineBuilder: split_schema()
    Note over EngineBuilder: Separate fields into\nLexicalIndexConfig\n+ VectorIndexConfig
    EngineBuilder->>Engine: Create LexicalStore
    EngineBuilder->>Engine: Create VectorStore
    EngineBuilder->>Engine: Create DocumentLog
    EngineBuilder->>Engine: Recover from WAL
    EngineBuilder-->>User: Engine ready

During build(), the engine:

1. Splits the schema — lexical fields go to LexicalIndexConfig, vector fields go to VectorIndexConfig
2. Creates prefixed storage — each component gets an isolated namespace (lexical/, vector/, documents/)
3. Initializes stores — LexicalStore and VectorStore are constructed with their configs
4. Recovers from WAL — replays any uncommitted operations from a previous session

Schema Splitting

The Schema contains both lexical and vector fields. At build time, split_schema() separates them:

graph LR
    S["Schema\ntitle: Text\nbody: Text\ncategory: Text\npage: Integer\ncontent_vec: HNSW"]

    S --> LC["LexicalIndexConfig\ntitle: TextOption\nbody: TextOption\ncategory: TextOption\npage: IntegerOption\n_id: KeywordAnalyzer"]

    S --> VC["VectorIndexConfig\ncontent_vec: HnswOption\n(dim=384, m=16, ef=200)"]

Key details:

• The reserved _id field is always added to the lexical config with KeywordAnalyzer (exact match)
• A PerFieldAnalyzer wraps per-field analyzer settings; if you pass a simple StandardAnalyzer, it becomes the default for all text fields
• A PerFieldEmbedder works the same way for vector fields

Indexing Data Flow

When you call engine.add_document(id, doc):

sequenceDiagram
    participant User
    participant Engine
    participant WAL as DocumentLog (WAL)
    participant Lexical as LexicalStore
    participant Vector as VectorStore

    User->>Engine: add_document("doc-1", doc)
    Engine->>WAL: Append to WAL
    Engine->>Engine: Assign internal ID (u64)

    loop For each field in document
        alt Lexical field (text, integer, etc.)
            Engine->>Lexical: Analyze + index field
        else Vector field
            Engine->>Vector: Embed + index field
        end
    end

    Note over Engine: Document is buffered\nbut NOT yet searchable

    User->>Engine: commit()
    Engine->>Lexical: Flush segments to storage
    Engine->>Vector: Flush segments to storage
    Engine->>WAL: Truncate WAL
    Note over Engine: Documents are\nnow searchable

Key points:

• WAL-first: every write is logged before modifying in-memory structures
• Dual indexing: each field is routed to either the lexical or vector store based on the schema
• Commit required: documents become searchable only after commit()

Search Data Flow

When you call engine.search(request):

sequenceDiagram
    participant User
    participant Engine
    participant Lexical as LexicalStore
    participant Vector as VectorStore
    participant Fusion

    User->>Engine: search(request)

    opt Filter query present
        Engine->>Lexical: Execute filter query
        Lexical-->>Engine: Allowed document IDs
    end

    par Lexical search
        Engine->>Lexical: Execute lexical query
        Lexical-->>Engine: Ranked hits (BM25)
    and Vector search
        Engine->>Vector: Execute vector query
        Vector-->>Engine: Ranked hits (similarity)
    end

    alt Both lexical and vector results
        Engine->>Fusion: Fuse results (RRF or WeightedSum)
        Fusion-->>Engine: Merged ranked list
    end

    Engine->>Engine: Apply offset + limit
    Engine-->>User: Vec of SearchResult

The search pipeline has three stages:

1. Filter (optional) — execute a filter query on the lexical index to get a set of allowed document IDs
2. Search — run lexical and/or vector queries in parallel
3. Fusion — if both query types are present, merge results using RRF (default, k=60) or WeightedSum

Storage Architecture

All components share a single Storage trait implementation, but use key prefixes to isolate their data:

graph TB
    Engine --> PS1["PrefixedStorage\nprefix: 'lexical/'"]
    Engine --> PS2["PrefixedStorage\nprefix: 'vector/'"]
    Engine --> PS3["PrefixedStorage\nprefix: 'documents/'"]

    PS1 --> S["Storage Backend\n(Memory / File / File+Mmap)"]
    PS2 --> S
    PS3 --> S

Per-Field Dispatch

When a PerFieldAnalyzer is provided, the engine dispatches analysis to field-specific analyzers. The same pattern applies to PerFieldEmbedder.

graph LR
    PFA["PerFieldAnalyzer"]
    PFA -->|"title"| KA["KeywordAnalyzer"]
    PFA -->|"body"| SA["StandardAnalyzer"]
    PFA -->|"description"| JA["JapaneseAnalyzer"]
    PFA -->|"_id"| KA2["KeywordAnalyzer\n(always)"]
    PFA -->|other fields| DEF["Default Analyzer\n(StandardAnalyzer)"]

This allows different fields to use different analysis strategies within the same engine.

Summary

Next Steps

• Understand field types and schema design: Schema & Fields
• Learn about text analysis: Text Analysis
• Learn about embeddings: Embeddings

Getting Started

Welcome to Laurus! This section will help you install the library and run your first search.

What You Will Build

By the end of this guide, you will have a working search engine that can:

• Index text documents
• Perform keyword (lexical) search
• Perform semantic (vector) search
• Combine both with hybrid search

Prerequisites

• Rust 1.85 or later (edition 2024)
• Cargo (included with Rust)
• Tokio runtime (Laurus uses async APIs)

Steps

1. Installation — Add Laurus to your project and choose feature flags
2. Quick Start — Build a complete search engine in 5 steps

Workflow Overview

Building a search application with Laurus follows a consistent pattern:

graph LR
    A["1. Create\nStorage"] --> B["2. Define\nSchema"]
    B --> C["3. Build\nEngine"]
    C --> D["4. Index\nDocuments"]
    D --> E["5. Search"]

Installation

Add Laurus to Your Project

Add laurus and tokio (async runtime) to your Cargo.toml:

[dependencies]
laurus = "0.9"
tokio = { version = "1", features = ["full"] }

Feature Flags

Laurus ships with a minimal default feature set. Enable additional features as needed:

Examples

Lexical search only (no embeddings needed):

[dependencies]
laurus = "0.9"

Vector search with local model (no API key required):

[dependencies]
laurus = { version = "0.9", features = ["embeddings-candle"] }

Vector search with OpenAI:

[dependencies]
laurus = { version = "0.9", features = ["embeddings-openai"] }

Everything:

[dependencies]
laurus = { version = "0.9", features = ["embeddings-all"] }

Verify Installation

Create a minimal program to verify that Laurus compiles:

use laurus::Result;

#[tokio::main]
async fn main() -> Result<()> {
    println!("Laurus version: {}", laurus::VERSION);
    Ok(())
}

cargo run

If you see the version printed, you are ready to proceed to the Quick Start.

Quick Start

This tutorial walks you through building a complete search engine in 5 steps. By the end, you will be able to index documents and search them by keyword.

Step 1 — Create Storage

Storage determines where Laurus persists index data. For development and testing, use MemoryStorage:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::storage::memory::MemoryStorage;
use laurus::Storage;

let storage: Arc<dyn Storage> = Arc::new(
    MemoryStorage::new(Default::default())
);
}

Tip: For production, consider FileStorage (with optional use_mmap for memory-mapped I/O). See Storage for details.

Step 2 — Define a Schema

A Schema declares the fields in your documents and how each field should be indexed:

#![allow(unused)]
fn main() {
use laurus::Schema;
use laurus::lexical::TextOption;

let schema = Schema::builder()
    .add_text_field("title", TextOption::default())
    .add_text_field("body", TextOption::default())
    .add_default_field("body")  // used when no field is specified in a query
    .build();
}

Each field has a type. Common types include:

See Schema & Fields for the full list.

Step 3 — Build an Engine

The Engine ties storage, schema, and runtime components together:

#![allow(unused)]
fn main() {
use laurus::Engine;

let engine = Engine::builder(storage, schema)
    .build()
    .await?;
}

When you only use text fields, the default StandardAnalyzer is used automatically. To customize analysis or add vector embeddings, see Architecture.

Step 4 — Index Documents

Create documents with the DocumentBuilder and add them to the engine:

#![allow(unused)]
fn main() {
use laurus::Document;

// Each document needs a unique external ID (string)
let doc = Document::builder()
    .add_text("title", "Introduction to Rust")
    .add_text("body", "Rust is a systems programming language focused on safety and performance.")
    .build();
engine.add_document("doc-1", doc).await?;

let doc = Document::builder()
    .add_text("title", "Python for Data Science")
    .add_text("body", "Python is widely used in machine learning and data analysis.")
    .build();
engine.add_document("doc-2", doc).await?;

let doc = Document::builder()
    .add_text("title", "Web Development with JavaScript")
    .add_text("body", "JavaScript powers interactive web applications and server-side code with Node.js.")
    .build();
engine.add_document("doc-3", doc).await?;

// Commit to make documents searchable
engine.commit().await?;
}

Important: Documents are not searchable until commit() is called.

Step 5 — Search

Use SearchRequestBuilder with a query to search the index:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

// Search for "rust" in the "body" field
let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(
            Box::new(TermQuery::new("body", "rust"))
        )
    )
    .limit(10)
    .build();

let results = engine.search(request).await?;

for result in &results {
    println!("ID: {}, Score: {:.4}", result.id, result.score);
    if let Some(doc) = &result.document {
        if let Some(title) = doc.get("title") {
            println!("  Title: {:?}", title);
        }
    }
}
}

Complete Example

Here is the full program that you can copy, paste, and run:

use std::sync::Arc;
use laurus::{
    Document, Engine,
    Result, Schema, SearchRequestBuilder,
};
use laurus::lexical::{TextOption, TermQuery};
use laurus::lexical::search::searcher::LexicalSearchQuery;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> Result<()> {
    // 1. Storage
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    // 2. Schema
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_default_field("body")
        .build();

    // 3. Engine
    let engine = Engine::builder(storage, schema).build().await?;

    // 4. Index documents
    for (id, title, body) in [
        ("doc-1", "Introduction to Rust", "Rust is a systems programming language focused on safety."),
        ("doc-2", "Python for Data Science", "Python is widely used in machine learning."),
        ("doc-3", "Web Development", "JavaScript powers interactive web applications."),
    ] {
        let doc = Document::builder()
            .add_text("title", title)
            .add_text("body", body)
            .build();
        engine.add_document(id, doc).await?;
    }
    engine.commit().await?;

    // 5. Search
    let request = SearchRequestBuilder::new()
        .lexical_query(
            LexicalSearchQuery::Obj(
                Box::new(TermQuery::new("body", "rust"))
            )
        )
        .limit(10)
        .build();

    let results = engine.search(request).await?;
    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }

    Ok(())
}

Next Steps

• Learn how the Engine works internally: Architecture
• Understand Schema and field types: Schema & Fields
• Add vector search: Vector Search
• Combine lexical + vector: Hybrid Search

Examples

The laurus/examples/ directory contains runnable examples demonstrating different features of the library.

Running Examples

# Run an example without feature flags
cargo run --example <name>

# Run an example with a feature flag
cargo run --example <name> --features <flag>

Available Examples

quickstart

A minimal example showing the basic workflow: create storage, define a schema, build an engine, index documents, and search.

cargo run --example quickstart

Demonstrates: In-memory storage, TextOption, TermQuery, LexicalSearchRequest.

lexical_search

Comprehensive example of all lexical query types, using both the Builder API and the QueryParser DSL.

cargo run --example lexical_search

Demonstrates: TermQuery, PhraseQuery, FuzzyQuery, WildcardQuery, NumericRangeQuery, GeoQuery, BooleanQuery, SpanQuery.

vector_search

Vector search with a mock embedder, including filtered vector search and DSL syntax.

cargo run --example vector_search

Demonstrates: PerFieldEmbedder, VectorSearchRequestBuilder, filtered search, DSL syntax (field:"query").

hybrid_search

Combining lexical and vector search with different fusion algorithms.

cargo run --example hybrid_search

Demonstrates: Lexical-only, vector-only, and hybrid search. Both RRF and WeightedSum fusion algorithms. Builder API and DSL.

geo3d_search

3D Earth-Centered Earth-Fixed (ECEF) geographic search. Indexes six landmarks (Tokyo Tower, Tokyo Skytree, Mt. Fuji summit, Statue of Liberty, Sydney Opera House, an ISS sample point) computed via wgs84_to_ecef.

cargo run --example geo3d_search

Demonstrates: Geo3dDistanceQuery (sphere), Geo3dBoundingBoxQuery (3D AABB), Geo3dNearestQuery (k-NN), and the wgs84_to_ecef conversion utility. The bounding-box query intentionally excludes the ISS sample to highlight that altitude is a first-class third axis.

search_with_candle

Vector search using real BERT embeddings via Hugging Face Candle. The model is downloaded automatically on first run (~80 MB).

cargo run --example search_with_candle --features embeddings-candle

Requires: embeddings-candle feature flag.

Demonstrates: CandleBertEmbedder with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions).

search_with_openai

Vector search using the OpenAI Embeddings API.

export OPENAI_API_KEY=your-api-key
cargo run --example search_with_openai --features embeddings-openai

Requires: embeddings-openai feature flag, OPENAI_API_KEY environment variable.

Demonstrates: OpenAIEmbedder with text-embedding-3-small (1536 dimensions).

multimodal_search

Multimodal (text + image) search using a CLIP model.

cargo run --example multimodal_search --features embeddings-multimodal

Requires: embeddings-multimodal feature flag.

Demonstrates: CandleClipEmbedder, indexing images from the filesystem, text-to-image and image-to-image queries.

synonym_graph_filter

Demonstrates the SynonymGraphFilter for token expansion during analysis.

cargo run --example synonym_graph_filter

Demonstrates: Synonym dictionary creation, synonym-based token expansion, boost application, token position and position_length attributes.

Helper Module: common.rs

The common.rs file provides shared utilities used by the examples:

• memory_storage() – Create an in-memory storage instance
• per_field_analyzer() – Create a PerFieldAnalyzer with KeywordAnalyzer for specific fields
• MockEmbedder – A mock Embedder implementation for testing vector search without a real model

Schema & Fields

The Schema defines the structure of your documents — what fields exist and how each field is indexed. It is the single source of truth for the Engine.

For the TOML file format used by the CLI, see Schema Format Reference.

Schema

A Schema is a collection of named fields. Each field is either a lexical field (for keyword search) or a vector field (for similarity search).

#![allow(unused)]
fn main() {
use laurus::Schema;
use laurus::lexical::TextOption;
use laurus::lexical::core::field::IntegerOption;
use laurus::vector::HnswOption;

let schema = Schema::builder()
    .add_text_field("title", TextOption::default())
    .add_text_field("body", TextOption::default())
    .add_integer_field("year", IntegerOption::default())
    .add_hnsw_field("embedding", HnswOption::default())
    .add_default_field("body")
    .build();
}

Default Fields

add_default_field() specifies which field(s) are searched when a query does not explicitly name a field. This is used by the Query DSL parser.

Field Types

graph TB
    FO["FieldOption"]

    FO --> T["Text"]
    FO --> I["Integer"]
    FO --> FL["Float"]
    FO --> B["Boolean"]
    FO --> DT["DateTime"]
    FO --> G["Geo"]
    FO --> G3["Geo3d"]
    FO --> BY["Bytes"]

    FO --> FLAT["Flat"]
    FO --> HNSW["HNSW"]
    FO --> IVF["IVF"]

Lexical Fields

Lexical fields are indexed using an inverted index and support keyword-based queries.

Text Field Options

TextOption controls how text is indexed:

#![allow(unused)]
fn main() {
use laurus::lexical::TextOption;

// Default: indexed + stored + term vectors (all true)
let opt = TextOption::default();

// Customize
let opt = TextOption::default()
    .indexed(true)
    .stored(true)
    .term_vectors(true);
}

Vector Fields

Vector fields are indexed using vector indexes for approximate nearest neighbor (ANN) search.

HNSW Field Options (most common)

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::distance::DistanceMetric;

use laurus::vector::core::quantization::QuantizationMethod;

let opt = HnswOption {
    dimension: 384,                                  // vector dimensions
    distance: DistanceMetric::Cosine,                // distance metric
    m: 16,                                           // max connections per layer
    ef_construction: 200,                            // construction search width
    default_ef_search: Some(100),                    // schema-level ef_search default (issue #644)
    base_weight: 1.0,                                // default scoring weight
    quantizer: QuantizationMethod::Scalar8Bit,       // mandatory; default Scalar8Bit
    embedder: None,                                  // optional named embedder
};
}

default_ef_search: the search-time recall knob

ef_search controls the dynamic candidate list size during query time (distinct from ef_construction, which only affects index build). Higher values explore more graph neighbours and yield higher recall at the cost of latency.

• Schema-level default: set HnswOption.default_ef_search = Some(ef) to raise the per-field default. When None, the searcher falls back to its built-in 50.
• Per-query override: search requests honour SearchRequestBuilder::vector_ef_search. The per-query value takes precedence over the schema default.
• Auto-lifting: regardless of which source provides ef_search, the searcher lifts the effective value to at least top_k (and top_k * rerank_factor when both are set) so the candidate heap is never undersized for the requested top_k.
• Tracked under Issue #644.

See Vector Indexing for detailed parameter guidance.

Document

A Document is a collection of named field values. Use DocumentBuilder to construct documents:

#![allow(unused)]
fn main() {
use laurus::Document;

let doc = Document::builder()
    .add_text("title", "Introduction to Rust")
    .add_text("body", "Rust is a systems programming language.")
    .add_integer("year", 2024)
    .add_float("rating", 4.8)
    .add_boolean("published", true)
    .build();
}

Indexing Documents

The Engine provides two methods for adding documents, each with different semantics:

#![allow(unused)]
fn main() {
// Upsert: replaces any existing document with id "doc1"
engine.put_document("doc1", doc).await?;

// Append: adds another chunk under the same id "doc1"
engine.add_document("doc1", chunk2).await?;

// Always commit after indexing
engine.commit().await?;
}

Retrieving Documents

Use get_documents to retrieve all documents (including chunks) by external ID:

#![allow(unused)]
fn main() {
let docs = engine.get_documents("doc1").await?;
for doc in &docs {
    if let Some(title) = doc.get("title") {
        println!("Title: {:?}", title);
    }
}
}

Deleting Documents

Delete all documents and chunks sharing an external ID:

#![allow(unused)]
fn main() {
engine.delete_documents("doc1").await?;
engine.commit().await?;
}

Document Lifecycle

graph LR
    A["Build Document"] --> B["put/add_document()"]
    B --> C["WAL"]
    C --> D["commit()"]
    D --> E["Searchable"]
    E --> F["get_documents()"]
    E --> G["delete_documents()"]

Important: Documents are not searchable until commit() is called.

DocumentBuilder Methods

DataValue

DataValue is the unified value enum that represents any field value in Laurus:

#![allow(unused)]
fn main() {
pub enum DataValue {
    Null,
    Bool(bool),
    Int64(i64),
    Float64(f64),
    Text(String),
    Bytes(Vec<u8>, Option<String>),  // (data, optional MIME type)
    Vector(Vec<f32>),
    DateTime(DateTime<Utc>),
    Geo(GeoPoint),                   // 2D WGS84 point (latitude, longitude)
    GeoEcef(GeoEcefPoint),           // 3D ECEF Cartesian point (x, y, z) in metres
    Int64Array(Vec<i64>),            // multi-valued integer field
    Float64Array(Vec<f64>),          // multi-valued float field
}
}

DataValue implements From<T> for common types, so you can use .into() conversions:

#![allow(unused)]
fn main() {
use laurus::DataValue;

let v: DataValue = "hello".into();       // Text
let v: DataValue = 42i64.into();         // Int64
let v: DataValue = 3.14f64.into();       // Float64
let v: DataValue = true.into();          // Bool
let v: DataValue = vec![0.1f32, 0.2].into(); // Vector
}

Reserved Fields

Any field name starting with an underscore (_) is reserved for the engine. User code cannot declare fields with such names, and documents that carry user-supplied _-prefixed keys are rejected at ingest time.

The only _-prefixed name that is accepted is the allow-listed _id system field described below.

_id — external document identifier

Stores the external document ID supplied to put_document / add_document. It is injected automatically and indexed with KeywordAnalyzer (exact match). You do not need to add it to your schema.

Dynamic Schema

Laurus can accept documents even when some of their fields have not been declared in the schema. The behaviour is controlled by the DynamicFieldPolicy attached to the schema:

Set the policy on the builder:

#![allow(unused)]
fn main() {
use laurus::{DynamicFieldPolicy, Schema};

let schema = Schema::builder()
    .dynamic_field_policy(DynamicFieldPolicy::Dynamic)
    .build();
}

Type inference rules (Dynamic policy)

Vector fields (Hnsw, Flat, Ivf) and Bytes are never inferred: they must be declared in the schema explicitly. Mixing 2D (lat/lon) and 3D (x/y/z) markers in a single object is rejected as ambiguous; use either shape, not both.

Multi-valued numeric fields

Integer and Float fields can be declared with multi_valued = true to hold multiple values per document. A range query matches a document if any of its values satisfies the predicate (Lucene-style “any match” semantics with constant scoring — there is no per-match BM25 weighting).

Single values sent to a multi-valued field are auto-wrapped into a one-element array; arrays sent to a single-valued field are rejected rather than silently truncating.

Type conflicts

When a value arrives for a field that is already declared, Laurus attempts to coerce the value to the declared type. The coercion rules are:

Coercion errors interact with the policy:

• Strict: error is returned immediately.
• Dynamic: error is returned — the coercion layer already applied every conversion that is considered safe.
• Ignore: the offending field is dropped; the rest of the document is indexed.

⚠️ Warning: silent information loss is possible.

Several coercions throw away information without reporting an error:

• An Integer field truncates incoming Float values (3.14 → 3, -3.9 → -3). Ingest does not fail.
• A Float field may lose precision for very large integers that do not fit in an f64 mantissa.
• A Text field accepts any scalar by stringifying it, losing the original type.
• Ignore drops incompatible fields quietly.

If the correctness of your data matters more than the convenience of schema-less ingestion, use DynamicFieldPolicy::Strict (or declare every field up-front). The Dynamic policy prioritises keeping the document ingestable over preserving every bit of incoming data.

Query DSL and undeclared fields

Once the schema is settled, the query parser validates that every field:value clause references a declared field. Typos such as titl:hello (for title:hello) produce a clear parse error instead of returning silently-empty results.

Dynamic Field Management

Fields can be added to or removed from a running engine at runtime. Type changes are not supported—remove the field and re-add it with the new type instead.

Adding a Field

Use Engine::add_field() to add a new field to the schema.

Adding a Lexical Field

let updated_schema = engine.add_field(
    "category",
    FieldOption::Text(TextOption::default()),
).await?;

Adding a Vector Field

let updated_schema = engine.add_field(
    "embedding",
    FieldOption::Flat(FlatOption::default().dimension(384)),
).await?;

Existing documents are unaffected—they simply have no value for the new field. The returned Schema should be persisted (e.g., to schema.toml) by the caller.

Removing a Field

Use Engine::delete_field() to remove a field from the schema.

let updated_schema = engine.delete_field("category").await?;

When a field is deleted:

• The field definition is removed from the schema.
• Existing indexed data for the field remains in the index but becomes inaccessible through queries.
• If the field was listed in default_fields, it is automatically removed.
• Any per-field analyzer or embedder registered for the field is unregistered.

Schema Design Tips

1. Separate lexical and vector fields — a field is either lexical or vector, never both. For hybrid search, create separate fields (e.g., body for text, body_vec for vector).

2. Use KeywordAnalyzer for exact-match fields — category, status, and tag fields should use KeywordAnalyzer via PerFieldAnalyzer to avoid tokenization.

3. Choose the right vector index — use HNSW for most cases, Flat for small datasets, IVF for very large datasets. See Vector Indexing.

4. Set default fields — if you use the Query DSL, set default fields so users can write hello instead of body:hello.

5. Use the schema generator — run laurus create schema to interactively build a schema TOML file instead of writing it by hand. See CLI Commands.

Text Analysis

Text analysis is the process of converting raw text into searchable tokens. When a document is indexed, the analyzer breaks text fields into individual terms; when a query is executed, the same analyzer processes the query text to ensure consistency.

The Analysis Pipeline

graph LR
    Input["Raw Text\n'The quick brown FOX jumps!'"]
    CF["UnicodeNormalizationCharFilter"]
    T["Tokenizer\nSplit into words"]
    F1["LowercaseFilter"]
    F2["StopFilter"]
    F3["StemFilter"]
    Output["Terms\n'quick', 'brown', 'fox', 'jump'"]

    Input --> CF --> T --> F1 --> F2 --> F3 --> Output

The analysis pipeline consists of:

1. Char Filters — normalize raw text at the character level before tokenization
2. Tokenizer — splits text into raw tokens (words, characters, n-grams)
3. Token Filters — transform, remove, or expand tokens (lowercase, stop words, stemming, synonyms)

The Analyzer Trait

All analyzers implement the Analyzer trait:

#![allow(unused)]
fn main() {
pub trait Analyzer: Send + Sync + Debug {
    fn analyze(&self, text: &str) -> Result<TokenStream>;
    fn name(&self) -> &str;
    fn as_any(&self) -> &dyn Any;
}
}

TokenStream is a Box<dyn Iterator<Item = Token> + Send> — a lazy iterator over tokens.

A Token contains:

Built-in Analyzers

StandardAnalyzer

The default analyzer. Suitable for most Western languages.

Pipeline: RegexTokenizer (Unicode word boundaries) → LowercaseFilter → StopFilter (128 common English stop words)

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::standard::StandardAnalyzer;

let analyzer = StandardAnalyzer::default();
// "The Quick Brown Fox" → ["quick", "brown", "fox"]
// ("The" is removed by stop word filtering)
}

JapaneseAnalyzer

Uses morphological analysis for Japanese text segmentation.

Pipeline: UnicodeNormalizationCharFilter (NFKC) → JapaneseIterationMarkCharFilter → LinderaTokenizer → LowercaseFilter → StopFilter (Japanese stop words)

JapaneseAnalyzer::new takes the same arguments as LinderaTokenizer::new: the segmentation mode, a path to a Lindera dictionary directory, and an optional user dictionary path. laurus does not enable Lindera’s embed-* features by default, so a real filesystem path (typically an IPADIC build) is required at runtime.

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::language::japanese::JapaneseAnalyzer;

// Pass the path where you have unpacked the Lindera dictionary.
let analyzer = JapaneseAnalyzer::new(
    "normal",
    "/var/lib/lindera/ipadic",
    None,
)?;
// "東京都に住んでいる" → ["東京", "都", "住ん", "いる"]
}

When the analyzer is referenced from a Schema, supply the parameters through the structured AnalyzerSpec form (see PerFieldAnalyzer below).

KeywordAnalyzer

Treats the entire input as a single token. No tokenization or normalization.

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::keyword::KeywordAnalyzer;

let analyzer = KeywordAnalyzer::new();
// "Hello World" → ["Hello World"]
}

Use this for fields that should match exactly (categories, tags, status codes).

SimpleAnalyzer

Tokenizes text without any filtering. The original case and all tokens are preserved. Useful when you need complete control over the analysis pipeline or want to test a tokenizer in isolation.

Pipeline: User-specified Tokenizer only (no char filters, no token filters)

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::simple::SimpleAnalyzer;
use laurus::analysis::tokenizer::regex::RegexTokenizer;
use std::sync::Arc;

let tokenizer = Arc::new(RegexTokenizer::new()?);
let analyzer = SimpleAnalyzer::new(tokenizer);
// "Hello World" → ["Hello", "World"]
// (no lowercasing, no stop word removal)
}

Use this for testing tokenizers, or when you want to apply token filters manually in a separate step.

EnglishAnalyzer

An English-specific analyzer. Tokenizes, lowercases, and removes common English stop words.

Pipeline: RegexTokenizer (Unicode word boundaries) → LowercaseFilter → StopFilter (128 common English stop words)

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::language::english::EnglishAnalyzer;

let analyzer = EnglishAnalyzer::new()?;
// "The Quick Brown Fox" → ["quick", "brown", "fox"]
// ("The" is removed by stop word filtering, remaining tokens are lowercased)
}

PipelineAnalyzer

Build a custom pipeline by combining any char filters, a tokenizer, and any sequence of token filters:

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::pipeline::PipelineAnalyzer;
use laurus::analysis::char_filter::unicode_normalize::{
    NormalizationForm, UnicodeNormalizationCharFilter,
};
use laurus::analysis::tokenizer::regex::RegexTokenizer;
use laurus::analysis::token_filter::lowercase::LowercaseFilter;
use laurus::analysis::token_filter::stop::StopFilter;
use laurus::analysis::token_filter::stem::StemFilter;

let analyzer = PipelineAnalyzer::new(Arc::new(RegexTokenizer::new()?))
    .add_char_filter(Arc::new(UnicodeNormalizationCharFilter::new(NormalizationForm::NFKC)))
    .add_filter(Arc::new(LowercaseFilter::new()))
    .add_filter(Arc::new(StopFilter::new()))
    .add_filter(Arc::new(StemFilter::new()));  // Porter stemmer
}

PerFieldAnalyzer

PerFieldAnalyzer lets you assign different analyzers to different fields within the same engine:

graph LR
    PFA["PerFieldAnalyzer"]
    PFA -->|"title"| KW["KeywordAnalyzer"]
    PFA -->|"body"| STD["StandardAnalyzer"]
    PFA -->|"description_ja"| JP["JapaneseAnalyzer"]
    PFA -->|other fields| DEF["Default\n(StandardAnalyzer)"]

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::analysis::analyzer::standard::StandardAnalyzer;
use laurus::analysis::analyzer::keyword::KeywordAnalyzer;
use laurus::analysis::analyzer::per_field::PerFieldAnalyzer;

// Default analyzer for fields not explicitly configured
let per_field = PerFieldAnalyzer::new(
    Arc::new(StandardAnalyzer::default())
);

// Use KeywordAnalyzer for exact-match fields
per_field.add_analyzer("category", Arc::new(KeywordAnalyzer::new()));
per_field.add_analyzer("status", Arc::new(KeywordAnalyzer::new()));

let engine = Engine::builder(storage, schema)
    .analyzer(Arc::new(per_field))
    .build()
    .await?;
}

Note: The _id field is always analyzed with KeywordAnalyzer regardless of configuration.

Configuring per-field analyzers from a Schema

Most callers configure analyzers declaratively on the schema rather than wiring them up by hand. The analyzer setting on a text field accepts two shapes:

// 1. A bare name for a parameter-less built-in or a user-registered analyzer.
{ "analyzer": "standard" }
{ "analyzer": "english" }
{ "analyzer": "my_custom_pipeline" }

// 2. A structured object for a parameterised built-in preset. Today only
//    the Japanese preset uses this form (it requires a Lindera dictionary
//    path).
{
  "analyzer": {
    "language": "japanese",
    "mode": "normal",
    "dict": "/var/lib/lindera/ipadic"
  }
}

The bare string "japanese" is rejected because the preset cannot be constructed without a dictionary. Schemas that previously stored "analyzer": "japanese" must migrate to the structured form above.

For full pipelines that do not fit a preset, register the pipeline under schema.analyzers as an AnalyzerDefinition and reference it by name.

Char Filters

Char filters operate on the raw input text before it reaches the tokenizer. They perform character-level normalization such as Unicode normalization, character mapping, and pattern-based replacement. This ensures that the tokenizer receives clean, normalized text.

All char filters implement the CharFilter trait:

#![allow(unused)]
fn main() {
pub trait CharFilter: Send + Sync {
    fn filter(&self, input: &str) -> (String, Vec<Transformation>);
    fn name(&self) -> &'static str;
}
}

The Transformation records describe how character positions shifted, allowing the engine to map token positions back to the original text.

UnicodeNormalizationCharFilter

Applies Unicode normalization to the input text. NFKC is recommended for search use cases because it normalizes both compatibility characters and composed forms.

#![allow(unused)]
fn main() {
use laurus::analysis::char_filter::unicode_normalize::{
    NormalizationForm, UnicodeNormalizationCharFilter,
};

let filter = UnicodeNormalizationCharFilter::new(NormalizationForm::NFKC);
// "Ｓｏｎｙ" (fullwidth) → "Sony" (halfwidth)
// "㌂" → "アンペア"
}

MappingCharFilter

Replaces character sequences using a dictionary. Matches are found using the Aho-Corasick algorithm (leftmost-longest match).

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use laurus::analysis::char_filter::mapping::MappingCharFilter;

let mut mapping = HashMap::new();
mapping.insert("ph".to_string(), "f".to_string());
mapping.insert("qu".to_string(), "k".to_string());

let filter = MappingCharFilter::new(mapping)?;
// "phone queue" → "fone keue"
}

PatternReplaceCharFilter

Replaces all occurrences of a regex pattern with a fixed string.

#![allow(unused)]
fn main() {
use laurus::analysis::char_filter::pattern_replace::PatternReplaceCharFilter;

// Remove hyphens
let filter = PatternReplaceCharFilter::new(r"-", "")?;
// "123-456-789" → "123456789"

// Normalize numbers
let filter = PatternReplaceCharFilter::new(r"\d+", "NUM")?;
// "Year 2024" → "Year NUM"
}

JapaneseIterationMarkCharFilter

Expands Japanese iteration marks (踊り字) to their base characters. Supports kanji (々), hiragana (ゝ, ゞ), and katakana (ヽ, ヾ) iteration marks.

#![allow(unused)]
fn main() {
use laurus::analysis::char_filter::japanese_iteration_mark::JapaneseIterationMarkCharFilter;

let filter = JapaneseIterationMarkCharFilter::new(
    true,  // normalize kanji iteration marks
    true,  // normalize kana iteration marks
);
// "佐々木" → "佐佐木"
// "いすゞ" → "いすず"
}

Using Char Filters in a Pipeline

Add char filters to a PipelineAnalyzer with add_char_filter(). Multiple char filters are applied in the order they are added, all before the tokenizer runs.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::analysis::analyzer::pipeline::PipelineAnalyzer;
use laurus::analysis::char_filter::unicode_normalize::{
    NormalizationForm, UnicodeNormalizationCharFilter,
};
use laurus::analysis::char_filter::pattern_replace::PatternReplaceCharFilter;
use laurus::analysis::tokenizer::regex::RegexTokenizer;
use laurus::analysis::token_filter::lowercase::LowercaseFilter;

let analyzer = PipelineAnalyzer::new(Arc::new(RegexTokenizer::new()?))
    .add_char_filter(Arc::new(
        UnicodeNormalizationCharFilter::new(NormalizationForm::NFKC),
    ))
    .add_char_filter(Arc::new(
        PatternReplaceCharFilter::new(r"-", "")?,
    ))
    .add_filter(Arc::new(LowercaseFilter::new()));
// "Ｔｏｋｙｏ-2024" → NFKC → "Tokyo-2024" → remove hyphens → "Tokyo2024" → tokenize → lowercase → ["tokyo2024"]
}

Tokenizers

Token Filters

Synonym Expansion

The SynonymGraphFilter expands terms using a synonym dictionary:

#![allow(unused)]
fn main() {
use laurus::analysis::synonym::dictionary::SynonymDictionary;
use laurus::analysis::token_filter::synonym_graph::SynonymGraphFilter;

let mut dict = SynonymDictionary::new(None)?;
dict.add_synonym_group(vec!["ml".into(), "machine learning".into()]);
dict.add_synonym_group(vec!["ai".into(), "artificial intelligence".into()]);

// keep_original=true means original token is preserved alongside synonyms
let filter = SynonymGraphFilter::new(dict, true)
    .with_boost(0.8);  // synonyms get 80% weight
}

The boost parameter controls how much weight synonyms receive relative to original tokens. A value of 0.8 means synonym matches contribute 80% as much to the score as exact matches.

Embeddings

Embeddings convert text (or images) into dense numeric vectors that capture semantic meaning. Two texts with similar meanings produce vectors that are close together in vector space, enabling similarity-based search.

The Embedder Trait

All embedders implement the Embedder trait:

#![allow(unused)]
fn main() {
#[async_trait]
pub trait Embedder: Send + Sync + Debug {
    async fn embed(&self, input: &EmbedInput<'_>) -> Result<Vector>;
    async fn embed_batch(&self, inputs: &[EmbedInput<'_>]) -> Result<Vec<Vector>>;
    fn supported_input_types(&self) -> Vec<EmbedInputType>;
    fn name(&self) -> &str;
    fn as_any(&self) -> &dyn Any;
}
}

The embed() method returns a Vector (a struct wrapping Vec<f32>).

EmbedInput supports two modalities:

Built-in Embedders

CandleBertEmbedder

Runs a BERT model locally using Hugging Face Candle. No API key required.

Feature flag: embeddings-candle

#![allow(unused)]
fn main() {
use laurus::CandleBertEmbedder;

// Downloads model on first run (~80MB)
let embedder = CandleBertEmbedder::new(
    "sentence-transformers/all-MiniLM-L6-v2"  // model name
)?;
// Output: 384-dimensional vector
}

OpenAIEmbedder

Calls the OpenAI Embeddings API. Requires an API key.

Feature flag: embeddings-openai

#![allow(unused)]
fn main() {
use laurus::OpenAIEmbedder;

let embedder = OpenAIEmbedder::new(
    api_key,
    "text-embedding-3-small".to_string()
).await?;
// Output: 1536-dimensional vector
}

CandleClipEmbedder

Runs a CLIP model locally for multimodal (text + image) embeddings.

Feature flag: embeddings-multimodal

#![allow(unused)]
fn main() {
use laurus::CandleClipEmbedder;

let embedder = CandleClipEmbedder::new(
    "openai/clip-vit-base-patch32"
)?;
// Text or images → 512-dimensional vector
}

PrecomputedEmbedder

Use pre-computed vectors directly without any embedding computation. Useful when vectors are generated externally.

#![allow(unused)]
fn main() {
use laurus::PrecomputedEmbedder;

let embedder = PrecomputedEmbedder::new();  // no parameters needed
}

When using PrecomputedEmbedder, you provide vectors directly in documents instead of text for embedding:

#![allow(unused)]
fn main() {
let doc = Document::builder()
    .add_vector("embedding", vec![0.1, 0.2, 0.3, ...])
    .build();
}

PerFieldEmbedder

PerFieldEmbedder routes embedding requests to field-specific embedders:

graph LR
    PFE["PerFieldEmbedder"]
    PFE -->|"text_vec"| BERT["CandleBertEmbedder\n(384 dim)"]
    PFE -->|"image_vec"| CLIP["CandleClipEmbedder\n(512 dim)"]
    PFE -->|other fields| DEF["Default Embedder"]

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::PerFieldEmbedder;

let bert = Arc::new(CandleBertEmbedder::new("...")?);
let clip = Arc::new(CandleClipEmbedder::new("...")?);


let per_field = PerFieldEmbedder::new(bert.clone());
per_field.add_embedder("text_vec", bert.clone());
per_field.add_embedder("image_vec", clip.clone());

let engine = Engine::builder(storage, schema)
    .embedder(Arc::new(per_field))
    .build()
    .await?;
}

This is especially useful when:

• Different vector fields need different models (e.g., BERT for text, CLIP for images)
• Different fields have different vector dimensions
• You want to mix local and remote embedders

How Embeddings Are Used

At Index Time

When you add a text value to a vector field, the engine automatically embeds it:

#![allow(unused)]
fn main() {
let doc = Document::builder()
    .add_text("text_vec", "Rust is a systems programming language")
    .build();
engine.add_document("doc-1", doc).await?;
// The embedder converts the text to a vector before indexing
}

At Search Time

When you search with text, the engine embeds the query text as well:

#![allow(unused)]
fn main() {
// Builder API
let request = VectorSearchRequestBuilder::new()
    .add_text("text_vec", "systems programming")
    .build();

// Query DSL
let request = vector_parser.parse(r#"text_vec:"systems programming""#).await?;
}

Both approaches embed the query text using the same embedder that was used at index time, ensuring consistent vector spaces.

Feature Flags Summary

Each embedder requires a specific feature flag to be enabled in Cargo.toml:

The embeddings-all feature enables all embedding features at once. See Feature Flags for details.

Choosing an Embedder

Storage

Laurus uses a pluggable storage layer that abstracts how and where index data is persisted. All components — lexical index, vector index, and document log — share a single storage backend.

The Storage Trait

All backends implement the Storage trait:

#![allow(unused)]
fn main() {
pub trait Storage: Send + Sync + Debug {
    fn loading_mode(&self) -> LoadingMode;
    fn open_input(&self, name: &str) -> Result<Box<dyn StorageInput>>;
    fn create_output(&self, name: &str) -> Result<Box<dyn StorageOutput>>;
    fn file_exists(&self, name: &str) -> bool;
    fn delete_file(&self, name: &str) -> Result<()>;
    fn list_files(&self) -> Result<Vec<String>>;
    fn file_size(&self, name: &str) -> Result<u64>;
    // ... additional methods
}
}

This interface is file-oriented: all data (index segments, metadata, WAL entries, documents) is stored as named files accessed through streaming StorageInput / StorageOutput handles.

Storage Backends

MemoryStorage

All data lives in memory. Fast and simple, but not durable.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::Storage;
use laurus::storage::memory::MemoryStorage;

let storage: Arc<dyn Storage> = Arc::new(
    MemoryStorage::new(Default::default())
);
}

FileStorage

Standard file-system based persistence. Each key maps to a file on disk.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::Storage;
use laurus::storage::file::{FileStorage, FileStorageConfig};

let config = FileStorageConfig::new("/tmp/laurus-data");
let storage: Arc<dyn Storage> = Arc::new(FileStorage::new("/tmp/laurus-data", config)?);
}

FileStorage with Memory Mapping

FileStorage supports memory-mapped file access via the use_mmap configuration flag. When enabled, the OS manages paging between memory and disk; the lexical posting decoder (Issue #504) takes a zero-copy path through StorageInput::as_slice, handing PFOR-bit-packed blocks directly to bitpacking::decompress* instead of allocating an intermediate Vec<u8> and copying through Read.

Default is platform-specific:

• *Unix (Linux / macOS / BSD): true as of Issue #504. Set the LAURUS_NO_MMAP=1 environment variable when constructing the config (via FileStorageConfig::new) to fall back to buffered file I/O for debug sessions or hosts where mmap misbehaves.
• Windows: false as of Issue #508. Windows holds an exclusive lock on memory-mapped files (ERROR_USER_MAPPED_FILE, os error 1224) which prevents the writer from truncating / deleting a segment file while a reader still holds an mmap. The current segment-file lifecycle is incompatible with that lock. Set LAURUS_USE_MMAP=1 to opt in for read-only / read-mostly workloads where commit frequency is low. Full Windows mmap support is tracked in Issue #508.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::Storage;
use laurus::storage::file::{FileStorage, FileStorageConfig};

// mmap is on by default on Unix; on Windows it is off unless
// LAURUS_USE_MMAP=1 is set.
let config = FileStorageConfig::new("/tmp/laurus-data");
let storage: Arc<dyn Storage> = Arc::new(FileStorage::new("/tmp/laurus-data", config)?);

// Explicit opt-out without touching the env var (works on any OS).
let mut buffered_config = FileStorageConfig::new("/tmp/laurus-data");
buffered_config.use_mmap = false;

// Explicit opt-in (works on any OS, including Windows).
let mut mmap_config = FileStorageConfig::new("/tmp/laurus-data");
mmap_config.use_mmap = true;
}

StorageFactory

You can also create storage via configuration:

#![allow(unused)]
fn main() {
use laurus::storage::{StorageConfig, StorageFactory};
use laurus::storage::memory::MemoryStorageConfig;

let storage = StorageFactory::create(
    StorageConfig::Memory(MemoryStorageConfig::default())
)?;
}

PrefixedStorage

The engine uses PrefixedStorage to isolate components within a single storage backend:

graph TB
    E["Engine"]
    E --> P1["PrefixedStorage\nprefix = 'lexical/'"]
    E --> P2["PrefixedStorage\nprefix = 'vector/'"]
    E --> P3["PrefixedStorage\nprefix = 'documents/'"]
    P1 --> S["Storage Backend"]
    P2 --> S
    P3 --> S

When the lexical store writes a key segments/seg-001.dict, it is actually stored as lexical/segments/seg-001.dict in the underlying backend. This ensures no key collisions between components.

You do not need to create PrefixedStorage yourself — the EngineBuilder handles this automatically.

ColumnStorage

In addition to the primary storage backends, Laurus provides a ColumnStorage layer for fast field-level access. This is used internally for operations like faceting, sorting, and aggregation, where accessing individual field values without deserializing entire documents is important.

ColumnValue

ColumnValue represents a single stored column value:

ColumnStorage is managed internally by the Engine – you do not need to interact with it directly.

Choosing a Backend

Recommendations

• Development / Testing: Use MemoryStorage for fast iteration without file cleanup
• Production (general): Use FileStorage for reliable persistence
• Production (large scale): Use FileStorage with use_mmap = true when you have large indexes and want to leverage OS page cache

Next Steps

• Learn how the lexical index works: Lexical Indexing
• Learn how the vector index works: Vector Indexing

Indexing

This section explains how Laurus stores and organizes data internally. Understanding the indexing layer will help you choose the right field types and tune performance.

Topics

Lexical Indexing

How text, numeric, and geographic fields are indexed using an inverted index. Covers:

• The inverted index structure (term dictionary, posting lists)
• BKD trees for numeric range queries
• Segment files and their formats
• BM25 scoring

Vector Indexing

How vector fields are indexed for approximate nearest neighbor search. Covers:

• Index types: Flat, HNSW, IVF
• Parameter tuning (m, ef_construction, n_clusters, n_probe)
• Distance metrics (Cosine, Euclidean, DotProduct)
• Quantization (SQ8, PQ)

Lexical Indexing

Lexical indexing powers keyword-based search. When a document’s text field is indexed, Laurus builds an inverted index — a data structure that maps terms to the documents containing them.

How Lexical Indexing Works

sequenceDiagram
    participant Doc as Document
    participant Analyzer
    participant Writer as IndexWriter
    participant Seg as Segment

    Doc->>Analyzer: "The quick brown fox"
    Analyzer->>Analyzer: Tokenize + Filter
    Analyzer-->>Writer: ["quick", "brown", "fox"]
    Writer->>Writer: Buffer in memory
    Writer->>Seg: Flush to segment on commit()

Step by Step

1. Analyze: The text passes through the configured analyzer (tokenizer + filters), producing a stream of normalized terms
2. Buffer: Terms are stored in an in-memory write buffer, organized by field
3. Commit: On commit(), the buffer is flushed to a new segment on storage

The Inverted Index

An inverted index is essentially a map from terms to document lists:

graph LR
    subgraph "Term Dictionary"
        T1["'brown'"]
        T2["'fox'"]
        T3["'quick'"]
        T4["'rust'"]
    end

    subgraph "Posting Lists"
        P1["doc_1, doc_3"]
        P2["doc_1"]
        P3["doc_1, doc_2"]
        P4["doc_2, doc_3"]
    end

    T1 --> P1
    T2 --> P2
    T3 --> P3
    T4 --> P4

Posting List Contents

Each entry in a posting list contains:

On-Disk Posting Layout

Posting lists are stored in a structure-of-arrays layout with each field written as its own contiguous section. Document IDs and term frequencies are encoded in fixed-size 128-int blocks using bit-packing (Frame-of-Reference plus sorted-delta for doc IDs), with any partial trailing block falling back to varint. This matches the on-disk format used by Tantivy and Lucene 9 and yields fast SIMD-accelerated decoding through the bitpacking crate.

[term, total_frequency, doc_frequency, posting_count N, any_positions]
[Skip levels              — v2 only: num_levels + per-level (len + u32 doc_ids)]
[Section 1: doc_ids       — N/128 bit-packed blocks + varint tail]
[Section 2: frequencies   — N/128 bit-packed blocks + varint tail]
[Section 3: weights       — N raw f32 values]
[Section 4: positions     — per-posting flag + varint deltas (only if any)]

Per-segment doc IDs must fit in u32. Encoding a value beyond u32::MAX fails fast with a clear error to prevent silent corruption of the bit-packed segment.

The decoder exposes a SoA-native fast path (PostingList::decode_soa) that produces parallel Vec<u32> slices for doc_id and frequency directly from disk without the intermediate Vec<Posting> reassembly. The query iterator stores those slices, so next() advances a single u32 cursor over a dense slice instead of striding across a 40-byte Posting struct.

Multi-Level Skip Table

A posting list of N ≥ 8 postings carries a Lucene-90-style multi-level skip table immediately after the header (v2 format, introduced for skip_to performance on seek-heavy workloads). Each level stores the “last doc id” of a fixed-stride window over doc_ids; level 0 has one entry per SKIP_INTERVAL = 8 postings, level 1 covers 8² postings, and so on until the top level collapses to a single entry.

PostingIterator::skip_to(target) walks the table top-down: at each level a single partition_point narrows the search window by a factor of SKIP_INTERVAL, descends one level, and finishes with a linear scan over the bottom-level window of at most 8 postings. Total work is O(log_8 N + SKIP_INTERVAL) per call — for N = 1 M that is ~25 comparisons, versus the linear O(N / block_size) walk the single-level block_cache paid before.

The table is co-located with the posting list rather than stored in a separate file, matching Lucene 9 / Tantivy. Segments written in the legacy v1 format (without the on-disk skip table) remain readable — the SoA decoder rebuilds the table from doc_ids at load time.

Term Dictionary Storage Layers

The dictionary uses a two-layer representation that decouples on-disk compactness from in-memory query speed:

• Disk layer — the .dict file uses a Lucene BlockTreeTermsWriter-style block-tree layout (magic LTDD, schema v1). This produces a compact file: at 100k unique 5-10 byte terms a .dict is ~12.5 bytes / term, roughly 70 % smaller than the prior parallel-array on-disk format.
• In-memory query layer — at build / load time the dictionary populates an AHashMap<term, ordinal> index, an ordinal-indexed sorted Vec<String>, and a single-copy Arc<[TermInfo]>. get / iter / find_prefix / find_range operate exclusively on these structures, so per-query cost matches the prior parallel-array implementation (no per-call FST traversal or in-block linear scan).

The disk-format scratch (FST + BlockSection bytes) is retained only so [BlockTermDictionary::write_to_storage] can re-serialise the dictionary on segment merge without re-encoding from scratch.

[Header                ]  magic "LTDD" + version
[FstSection            ]  fst::Map<u64> keyed by each block's last term,
                          valued by that block's start byte offset
[BlockSection          ]  concatenated 128-term blocks, each containing
                          front-coded term bytes, a bit-packed
                          fixed-size TermInfo block, and a
                          variable-length per-term Block-Max-WAND
                          metadata array
[Footer                ]  total term count + block count

Lookup walks the FST once (O(|term|)) to identify the block whose last term is ≥ target, then performs a linear front-coded scan inside the block (≤ 128 entries). Iteration walks the BlockSection sequentially without consulting the FST, decoding each block’s front-coded prefix once and reusing the buffer across terms. Prefix scans combine the two: FST seek to the first matching block, then sequential block walk until the prefix no longer matches.

Compared to a flat per-term FST the block-head FST is one to two orders of magnitude smaller, and the front-coded block bytes plus bit-packed TermInfo typically cut the on-disk size by 50-80 % at production scale.

Numeric and Date Fields

Integer, float, and datetime fields are indexed using a BKD tree — a space-partitioning data structure optimized for range queries:

graph TB
    Root["BKD Root"]
    Root --> L["values < 50"]
    Root --> R["values >= 50"]
    L --> LL["values < 25"]
    L --> LR["25 <= values < 50"]
    R --> RL["50 <= values < 75"]
    R --> RR["values >= 75"]

BKD trees allow efficient evaluation of range queries like price:[10 TO 100] or date:[2024-01-01 TO 2024-12-31].

Geo Fields

Geographic fields come in two flavours, both backed by the same multi-dimensional BKD-Tree primitive:

Geo3d is the right choice when altitude is a first-class dimension — drones, satellites, indoor 3D positioning, or anything else that a 2D Geo field would lose or distort near the poles. See 3D Geographic Search (ECEF) for the coordinate system, WGS84 conversion helpers, and DSL syntax.

Segments

The lexical index is organized into segments. Each segment is an immutable, self-contained mini-index:

graph TB
    LI["Lexical Index"]
    LI --> S1["Segment 0"]
    LI --> S2["Segment 1"]
    LI --> S3["Segment 2"]

    S1 --- F1[".dict (terms)"]
    S1 --- F2[".post (postings)"]
    S1 --- F3[".bkd (numerics)"]
    S1 --- F4[".docs (doc store)"]
    S1 --- F5[".dv (doc values)"]
    S1 --- F6[".meta (metadata)"]
    S1 --- F7[".lens (field lengths)"]

Segment Lifecycle

1. Create: A new segment is created each time commit() is called
2. Search: All segments are searched in parallel and results are merged
3. Merge: After each commit(), an auto-merge merges the smallest segments once the count exceeds max_segments, keeping the segment count bounded; a manual optimize() force-merges everything into one segment
4. Delete: When a document is deleted, its ID is added to a deletion bitmap rather than physically removed (see Deletions & Compaction)

BM25 Scoring

Laurus uses the BM25 algorithm to score lexical search results. BM25 considers:

• Term Frequency (TF): how often the term appears in the document (more = better, with diminishing returns)
• Inverse Document Frequency (IDF): how rare the term is across all documents (rarer = more important)
• Field Length Normalization: shorter fields are boosted relative to longer ones

The formula:

score(q, d) = IDF(q) * (TF(q, d) * (k1 + 1)) / (TF(q, d) + k1 * (1 - b + b * |d| / avgdl))

Where k1 = 1.2 and b = 0.75 are the default tuning parameters.

SIMD Optimization

Vector distance calculations leverage SIMD (Single Instruction, Multiple Data) instructions when available, providing significant speedups for similarity computations in vector search.

Code Example

use std::sync::Arc;
use laurus::{Document, Engine, Schema};
use laurus::lexical::TextOption;
use laurus::lexical::core::field::IntegerOption;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_integer_field("year", IntegerOption::default())
        .build();

    let engine = Engine::builder(storage, schema).build().await?;

    // Index documents
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming")
        .add_text("body", "Rust is a systems programming language.")
        .add_integer("year", 2024)
        .build()
    ).await?;

    // Commit to flush segments to storage
    engine.commit().await?;

    Ok(())
}

Next Steps

• Learn how vector indexes work: Vector Indexing
• Run queries against the lexical index: Lexical Search

Vector Indexing

Vector indexing powers similarity-based search. When a document’s vector field is indexed, Laurus stores the embedding vector in a specialized index structure that enables fast approximate nearest neighbor (ANN) retrieval.

How Vector Indexing Works

sequenceDiagram
    participant Doc as Document
    participant Embedder
    participant Normalize as Normalizer
    participant Index as Vector Index

    Doc->>Embedder: "Rust is a systems language"
    Embedder-->>Normalize: [0.12, -0.45, 0.78, ...]
    Normalize->>Normalize: L2 normalize
    Normalize-->>Index: [0.14, -0.52, 0.90, ...]
    Index->>Index: Insert into index structure

Step by Step

1. Embed: The text (or image) is converted to a vector by the configured embedder
2. Normalize: The vector is L2-normalized (for cosine similarity)
3. Index: The vector is inserted into the configured index structure (Flat, HNSW, or IVF)
4. Commit: On commit(), the index is flushed to persistent storage

Index Types

Laurus supports three vector index types, each with different performance characteristics:

Comparison

Flat Index

The simplest index. Compares the query vector against every stored vector (brute-force).

#![allow(unused)]
fn main() {
use laurus::vector::FlatOption;
use laurus::vector::core::distance::DistanceMetric;

let opt = FlatOption {
    dimension: 384,
    distance: DistanceMetric::Cosine,
    ..Default::default()
};
}

• Pros: 100% recall (exact results), simple, low memory
• Cons: Slow for large datasets (linear scan)
• Use when: You have fewer than ~10,000 vectors, or you need exact results

HNSW Index

Hierarchical Navigable Small World graph. The default and most commonly used index type.

graph TB
    subgraph "Layer 2 (sparse)"
        A2["A"] --- C2["C"]
    end

    subgraph "Layer 1 (medium)"
        A1["A"] --- B1["B"]
        A1 --- C1["C"]
        B1 --- D1["D"]
        C1 --- D1
    end

    subgraph "Layer 0 (dense - all vectors)"
        A0["A"] --- B0["B"]
        A0 --- C0["C"]
        B0 --- D0["D"]
        B0 --- E0["E"]
        C0 --- D0
        C0 --- F0["F"]
        D0 --- E0
        E0 --- F0
    end

    A2 -.->|"entry point"| A1
    A1 -.-> A0
    C2 -.-> C1
    C1 -.-> C0
    B1 -.-> B0
    D1 -.-> D0

The HNSW algorithm searches from the top (sparse) layer down to the bottom (dense) layer, narrowing the search space at each level.

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::distance::DistanceMetric;

let opt = HnswOption {
    dimension: 384,
    distance: DistanceMetric::Cosine,
    m: 16,                  // max connections per node per layer
    ef_construction: 200,   // search width during index building
    ..Default::default()
};
}

HNSW Parameters

Tuning tips:

• Increase m (e.g., 32 or 64) for higher recall at the cost of memory
• Increase ef_construction (e.g., 400) for better index quality at the cost of build time
• At search time, the ef_search parameter (set in the search request) controls the search width

IVF Index

Inverted File Index. Partitions vectors into clusters, then only searches relevant clusters.

graph TB
    Q["Query Vector"]
    Q --> C1["Cluster 1\n(centroid)"]
    Q --> C2["Cluster 2\n(centroid)"]

    C1 --> V1["vec_3"]
    C1 --> V2["vec_7"]
    C1 --> V3["vec_12"]

    C2 --> V4["vec_1"]
    C2 --> V5["vec_9"]
    C2 --> V6["vec_15"]

    style C1 fill:#f9f,stroke:#333
    style C2 fill:#f9f,stroke:#333

#![allow(unused)]
fn main() {
use laurus::vector::IvfOption;
use laurus::vector::core::distance::DistanceMetric;

let opt = IvfOption {
    dimension: 384,
    distance: DistanceMetric::Cosine,
    n_clusters: 100,   // number of clusters
    n_probe: 10,       // clusters to search at query time
    ..Default::default()
};
}

IVF Parameters

Tuning tips:

• Set n_clusters to roughly sqrt(n) where n is the number of vectors
• Set n_probe to 5-20% of n_clusters for a good recall/speed trade-off
• IVF requires a training phase — initial indexing may be slower

Distance Metrics

#![allow(unused)]
fn main() {
use laurus::vector::core::distance::DistanceMetric;

let metric = DistanceMetric::Cosine;      // Default for text
let metric = DistanceMetric::Euclidean;    // For spatial data
let metric = DistanceMetric::Manhattan;    // L1 distance
let metric = DistanceMetric::DotProduct;   // For pre-normalized vectors
let metric = DistanceMetric::Angular;      // Angular distance
}

Note: For cosine similarity, vectors are automatically L2-normalized before indexing. Lower distance = more similar.

Quantization

Vectors are stored on disk as 8-bit scalar-quantized integers (Issue #481 Stage 1). Compared to the previous 32-bit float storage this is ~4x smaller with negligible recall loss in practice (Recall@10 remains ≥ 0.95 against the f32 ground truth — see the recall test at laurus/tests/vector_recall_test.rs).

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::quantization::QuantizationMethod;

// `quantizer` defaults to `Scalar8Bit`; the explicit form below is
// equivalent to `HnswOption { dimension: 384, ..Default::default() }`.
let opt = HnswOption {
    dimension: 384,
    quantizer: QuantizationMethod::Scalar8Bit,
    ..Default::default()
};
}

Breaking change (Issue #481 Stage 1): the quantizer field is no longer Option<QuantizationMethod>; quantization is mandatory and defaults to Scalar8Bit. There is no longer an unquantized (f32) on-disk format. Existing pre-Stage-1 vector indexes are intentionally not readable by this version — rebuild the index from source data.

How Scalar8Bit works

• Each segment trains a single global (offset, scale) pair from its f32 vectors at flush time (offset = min, scale = (max - min) / 255).
• Each f32 element is encoded as u8 = clamp(round((v - offset) / scale), 0, 255).
• Per-vector metadata (sum_q: u32, norm_q: f32) is precomputed and persisted alongside the int8 payload so the cosine search hot loop collapses to one int8 SIMD multiply-accumulate plus three scalar corrections — no per-element dequantization at search time.
• Segment files start with the LVS1 magic + a 16-byte header so the reader can detect the format at load time.

Two-stage rerank (Issue #481 Stage 2)

Stage 1 stores vectors as int8 only. The graph search runs entirely against int8 distances, which is fast but introduces a small quantization error. Stage 2 adds an optional per-field f32 sidecar so the searcher can rescore the top candidates against the original full-precision vectors:

1. The HNSW int8 graph search returns up to ef_search candidates ranked by quantized cosine distance.
2. The top top_k * rerank_factor candidates are rescored against the f32 vectors loaded from the LRS1 sidecar (*.hnsw.f32).
3. The new ranking is truncated to top_k and returned.

Stage 2 is opt-in per field via HnswOption.rerank_storage:

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::rerank::RerankStorageKind;

let opt = HnswOption {
    rerank_storage: Some(RerankStorageKind::F32),
    ..HnswOption::default()
};
}

Queries pass the rerank factor through VectorIndexQuery::rerank_factor (low-level), SearchRequestBuilder::vector_rerank_factor (engine), or the gRPC / JSON VectorParams.rerank_factor field.

Fields without rerank_storage enabled silently fall back to the Stage 1 int8 ranking even when rerank_factor is set — there is no f32 information to recover from a Stage 1 segment.

LRS1 rerank sidecar

The sidecar is a separate file written next to the LVS1 segment when rerank_storage is enabled:

offset  size  field
------  ----  -------------------------------------------
     0     4  magic         ASCII "LRS1"
     4     2  version       u16 LE  (current = 1)
     6     2  storage_kind  u16 LE  (1 = F32; 0 reserved; 2.. future)
     8     8  reserved      zero-padded
    16     4  dim           u32 LE
    20     4  vector_count  u32 LE
    24     -  payload       vector_count * dim * bytes_per_element

Vectors are written in the same (doc_id, field_name) order as the matching LVS1 segment, so a (sidecar position) → (LVS1 position) mapping is the identity. The HNSW reader loads the sidecar into a RerankStoragePool at init time when the storage loading mode is Eager; Lazy mode skips the sidecar to honor its memory-savings promise (Stage 2 segments opened in Lazy mode silently degrade to Stage 1).

Recall vs speed trade-off

rerank_factor lets you exchange a small per-query rerank cost (~top_k * rerank_factor exact-distance calls — a few µs at dim 128) for higher Recall@10. The gain depends on the corpus and the graph search budget (ef_search):

• Real clustered embedding data (text-embedding-3, BERT, etc.) reaches Recall@10 ≥ 0.99 at low ef_search; rerank polishes the ranking with negligible latency overhead.
• Synthetic random unit-norm data (the worst case for HNSW recall recovery) needs a higher ef_search for the int8 graph to visit enough true top-10 candidates; rerank then re-orders the visited set but cannot retrieve candidates the graph never reached.

The recall acceptance is split into two CI gates so the rerank kernel and the full HNSW pipeline can fail independently:

• stage2_brute_force_rerank_recall_at_10_meets_kernel_gate asserts Recall@10 ≥ 0.99. Bypasses the HNSW graph entirely (brute-force int8 over the corpus, widen to top_k * rerank_factor, rescore with f32) so any miss is a rerank-kernel regression.
• hnsw_quantized_recall_at_10_with_rerank_meets_stage2_recall_gate asserts Recall@10 ≥ 0.98. Adds the HNSW graph-construction non-determinism that an f32 HNSW baseline would also contribute; the looser gate matches the observed run-to-run variance band on this synthetic adversarial distribution. Real clustered embedding data and a stronger HNSW config (m=32, ef_construction=500) reach ≥ 0.99 on this path too — see the diagnostic sweep below.

The companion stage2_recall_sweep_diagnostic (opt-in via LAURUS_STAGE2_SWEEP=1) sweeps (ef_search, rerank_factor) across three corpus / query distributions and two HNSW configs so production deployments can calibrate the budget for their actual embedding distribution.

Real-data validation (Issue #498)

A third opt-in CI gate validates Stage 2 against a real ANN benchmark dataset (SIFT1M from TEXMEX) so the synthetic-data gates above do not become the only signal:

• hnsw_quantized_recall_at_10_with_rerank_on_sift_meets_stage2_real_data_recall_gate asserts Recall@10 ≥ 0.99 on a 50 000-vector SIFT1M subsample at (m=16, ef_construction=200, ef_search=200, rerank_factor=5).
• The companion bench bench_hnsw_graph_search_rerank_real_data (in laurus/benches/vector_search_bench.rs) measures end-to-end Stage 2 latency on the same fixture. The accompanying example laurus/examples/sift_rerank_probe.rs runs a full (ef_search × rerank_factor × HNSW config) sweep with (Recall, latency) per cell so operators can pick the operating point for their own data shape.

Both are gated on LAURUS_REAL_BENCHMARK=1 AND the presence of the SIFT1M .fvecs files under .cache/sift/sift/. Default CI runs are unchanged. To enable locally:

./scripts/fetch-sift.sh --large   # ~478MB
LAURUS_REAL_BENCHMARK=1 cargo test --release \
    --test vector_recall_test \
    hnsw_quantized_recall_at_10_with_rerank_on_sift_meets_stage2_real_data_recall_gate \
    -- --nocapture
LAURUS_REAL_BENCHMARK=1 cargo bench --bench vector_search_bench \
    -- "HNSW Graph Search Rerank Real"

The Issue #481 wording originally asked for “≥ 3× speedup vs the pre-Stage-1 f32 baseline.” Cross-branch Criterion measurements on SIFT1M-50k (median over 30 samples, same (m, ef_construction, ef_search)) gave 625 µs/query on the pre-Stage-1 f32 HNSW path versus 323 µs/query on the Stage 2 int8 + rerank path — a 1.94× speedup. Issue #498 reduces the real-data gate to ≥ 1.5× accordingly, with the recall side held at the original 0.99. The gap between the original 3× target and the measured 1.94× comes from rerank only re-ordering candidates the int8 graph traversal already visited — a lower ef_search does not pay back through rerank when the candidate set itself becomes too narrow, which a follow-up could address by widening the graph search budget independently of ef_search (the Lucene 99 pattern).

Product Quantization with rerank (Issue #481 Stage 3)

Stage 3 adds an opt-in Product Quantization path for the HNSW index. Each segment trains a per-field codebook of M sub-vectors × K = 256 centroids using Lloyd k-means with k-means++ initialisation, and stores every vector as M bytes (one centroid index per sub-vector). The search hot loop replaces the int8 SIMD kernel with asymmetric distance computation (ADC): per-query the searcher builds an M × K look-up table of squared distances between the query’s sub-vectors and the codebook entries, then scores each candidate as Σ_m lut[m][codes[m]] — M table lookups + M − 1 adds per candidate.

PQ is enabled per field via HnswOption.quantizer:

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::quantization::QuantizationMethod;
use laurus::vector::core::rerank::RerankStorageKind;

let opt = HnswOption {
    dimension: 128,
    quantizer: QuantizationMethod::ProductQuantization { subvector_count: 32 },
    // PQ-only Recall@10 caps out around 0.78-0.92 on SIFT1M, so
    // production deployments should pair PQ with the LRS1 rerank
    // sidecar — same Stage 2 mechanism, just driven by PQ
    // candidate generation instead of int8.
    rerank_storage: Some(RerankStorageKind::F32),
    ..Default::default()
};
}

subvector_count must divide dimension. Common choices for dim = 128: M ∈ {8, 16, 32} (sub_dim 16 / 8 / 4). Larger M trades on-disk compression for higher recall — Issue #481 Stage 3 ships only the 8-bit (K = 256) variant; the on-disk format reserves a 4-bit (K = 16) slot for a future PR.

On-disk format

PQ segments use the same LVS1 header as Scalar8Bit (quant_kind = 2) and carry the codebook inside the per-segment metadata block:

[ Fixed header           16 bytes ]
[ PQ params               8 bytes ]    m / k / sub_dim / padding (u16 × 4)
[ Codebook                m × k × sub_dim × 4 bytes ]
[ Per-vector codes        num_vectors × m bytes ]

For dim = 128, M = 32, K = 256: codebook = 32 × 256 × 4 × 4 = 131 072 bytes (128 KB) per segment plus 32 bytes per vector.

Recall and speed gates (Issue #481 Stage 3)

• Kernel-level test — synthetic 5 000-vector / dim 128 / 100 queries at (m=16, ef_construction=200, ef_search=200, rerank_factor=10, M=32): hnsw_pq_rerank_recall_at_10_meets_stage3_recall_gate asserts Recall@10 ≥ 0.95. Measured 0.9660.
• Real-data test — SIFT1M-50k subsample (opt-in via LAURUS_REAL_BENCHMARK=1, same config): hnsw_pq_rerank_recall_at_10_on_sift_meets_stage3_real_data_recall_gate asserts Recall@10 ≥ 0.95. Measured 0.9965.
• Real-data speed bench — bench_hnsw_graph_search_pq_rerank_real_data (opt-in, Criterion). Cross-branch measurement at the same SIFT1M-50k config: pre-Stage-1 f32 HNSW = 625.21 µs/query (PR #500); Stage 3 PQ + rerank = 299.54 µs/query = 2.09× speedup.

Issue #481 originally asked for ≥ 5× speedup at Recall ≥ 0.95. The PR established that target is not reachable on SIFT1M with the current implementation — both Stage 2 (#500) and this Stage 3 PR have measured speedups in the 1.9-2.1× band because rerank dominates the wall-clock once the candidate set is wide enough to recover recall. The gate was reduced to ≥ 1.5× accordingly. A follow-up could pursue the Lucene 99 pattern (independent graph search budget) and / or a 4-bit PQ variant to close the gap.

Segment Files

Each vector index type stores its data in a single segment file:

Code Example

use std::sync::Arc;
use laurus::{Document, Engine, Schema};
use laurus::lexical::TextOption;
use laurus::vector::HnswOption;
use laurus::vector::core::distance::DistanceMetric;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_hnsw_field("embedding", HnswOption {
            dimension: 384,
            distance: DistanceMetric::Cosine,
            m: 16,
            ef_construction: 200,
            ..Default::default()
        })
        .build();

    // With an embedder, text in vector fields is automatically embedded
    let engine = Engine::builder(storage, schema)
        .embedder(my_embedder)
        .build()
        .await?;

    // Add text to the vector field — it will be embedded automatically
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming")
        .add_text("embedding", "Rust is a systems programming language.")
        .build()
    ).await?;

    engine.commit().await?;

    Ok(())
}

Next Steps

• Search the vector index: Vector Search
• Combine with lexical search: Hybrid Search

Search

This section covers how to query your indexed data. Laurus supports three search modes that can be used independently or combined.

Topics

Lexical Search

Keyword-based search using an inverted index. Covers:

• All query types: Term, Phrase, Boolean, Fuzzy, Wildcard, Range, Geo, Span
• BM25 scoring and field boosts
• Using the Query DSL for text-based queries

Vector Search

Semantic similarity search using vector embeddings. Covers:

• VectorSearchRequestBuilder API
• Multi-field vector search and score modes
• Filtered vector search

Hybrid Search

Combining lexical and vector search for best-of-both-worlds results. Covers:

• SearchRequestBuilder API
• Fusion algorithms (RRF, WeightedSum)
• Filtered hybrid search
• Pagination with offset/limit

For spelling correction, see Spelling Correction in the Library section.

Lexical Search

Lexical search finds documents by matching keywords against an inverted index. Laurus provides a rich set of query types that cover exact matching, phrase matching, fuzzy matching, and more.

Basic Usage

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(
            Box::new(TermQuery::new("body", "rust"))
        )
    )
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

Query Types

TermQuery

Matches documents containing an exact term in a specific field.

#![allow(unused)]
fn main() {
use laurus::lexical::TermQuery;

// Find documents where "body" contains the term "rust"
let query = TermQuery::new("body", "rust");
}

Note: Terms are matched after analysis. If the field uses StandardAnalyzer, both the indexed text and the query term are lowercased, so TermQuery::new("body", "rust") will match “Rust” in the original text.

PhraseQuery

Matches documents containing an exact sequence of terms.

#![allow(unused)]
fn main() {
use laurus::lexical::query::phrase::PhraseQuery;

// Find documents containing the exact phrase "machine learning"
let query = PhraseQuery::new("body", vec!["machine".to_string(), "learning".to_string()]);

// Or use the convenience method from a phrase string:
let query = PhraseQuery::from_phrase("body", "machine learning");
}

Phrase queries require term positions to be stored (the default for TextOption).

BooleanQuery

Combines multiple queries with boolean logic.

#![allow(unused)]
fn main() {
use laurus::lexical::query::boolean::{BooleanQuery, BooleanQueryBuilder, Occur};

let query = BooleanQueryBuilder::new()
    .must(Box::new(TermQuery::new("body", "rust")))       // AND
    .must(Box::new(TermQuery::new("body", "programming"))) // AND
    .must_not(Box::new(TermQuery::new("body", "python")))  // NOT
    .build();
}

FuzzyQuery

Matches terms within a specified edit distance (Levenshtein distance).

#![allow(unused)]
fn main() {
use laurus::lexical::query::fuzzy::FuzzyQuery;

// Find documents matching "programing" within edit distance 2
// This will match "programming", "programing", etc.
let query = FuzzyQuery::new("body", "programing");  // default max_edits = 2
}

WildcardQuery

Matches terms using wildcard patterns.

#![allow(unused)]
fn main() {
use laurus::lexical::query::wildcard::WildcardQuery;

// '?' matches exactly one character, '*' matches zero or more
let query = WildcardQuery::new("filename", "*.pdf")?;
let query = WildcardQuery::new("body", "pro*")?;
let query = WildcardQuery::new("body", "col?r")?;  // matches "color" and "colour"
}

PrefixQuery

Matches documents containing terms that start with a specific prefix.

#![allow(unused)]
fn main() {
use laurus::lexical::query::prefix::PrefixQuery;

// Find documents where "body" contains terms starting with "pro"
// This matches "programming", "program", "production", etc.
let query = PrefixQuery::new("body", "pro");
}

RegexpQuery

Matches documents containing terms that match a regular expression pattern.

#![allow(unused)]
fn main() {
use laurus::lexical::query::regexp::RegexpQuery;

// Find documents where "body" contains terms matching the regex
let query = RegexpQuery::new("body", "^pro.*ing$")?;

// Match version-like patterns
let query = RegexpQuery::new("version", r"^v\d+\.\d+")?;
}

Note: RegexpQuery::new() returns Result because the regex pattern is validated at construction time. Invalid patterns will produce an error.

NumericRangeQuery

Matches documents with numeric field values within a range.

#![allow(unused)]
fn main() {
use laurus::lexical::NumericRangeQuery;
use laurus::lexical::core::field::NumericType;

// Find documents where "price" is between 10.0 and 100.0 (inclusive)
let query = NumericRangeQuery::new(
    "price",
    NumericType::Float,
    Some(10.0),   // min
    Some(100.0),  // max
    true,         // include min
    true,         // include max
);

// Open-ended range: price >= 50
let query = NumericRangeQuery::new(
    "price",
    NumericType::Float,
    Some(50.0),
    None,     // no upper bound
    true,
    false,
);
}

GeoQuery

Matches documents by 2D geographic location (WGS84 latitude / longitude).

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo::GeoQuery;

// Find documents within 10 km (= 10 000 m) of Tokyo Station (35.6812, 139.7671)
let query = GeoQuery::within_radius("location", 35.6812, 139.7671, 10_000.0)?; // distance in metres

// Find documents within a bounding box (min_lat, min_lon, max_lat, max_lon)
let query = GeoQuery::within_bounding_box(
    "location",
    35.0, 139.0,  // min (lat, lon)
    36.0, 140.0,  // max (lat, lon)
)?;
}

Geo3dDistanceQuery / Geo3dBoundingBoxQuery / Geo3dNearestQuery

Three queries target 3D Geo3d fields backed by ECEF Cartesian coordinates (metres). Use them when altitude matters or when a 2D Geo field would introduce pole singularities. See 3D Geographic Search for the coordinate system, WGS84 conversion helpers, and worked examples.

#![allow(unused)]
fn main() {
use laurus::GeoEcefPoint;
use laurus::lexical::query::geo3d::{
    Geo3dDistanceQuery, Geo3dBoundingBoxQuery, Geo3dNearestQuery,
};

let centre = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);

// Sphere: docs within 5 km of `centre`
let q = Geo3dDistanceQuery::new("position", centre, 5_000.0);

// Axis-aligned 3D bounding box (constructor validates min ≤ max per axis)
let min = GeoEcefPoint::new(-4_000_000.0, 3_300_000.0, 3_650_000.0);
let max = GeoEcefPoint::new(-3_900_000.0, 3_400_000.0, 3_750_000.0);
let q = Geo3dBoundingBoxQuery::new("position", min, max)?;

// k-NN: 10 nearest neighbours, with a custom radius schedule
let q = Geo3dNearestQuery::new("position", centre, 10)
    .with_initial_radius(500.0)
    .with_max_radius(1_000_000.0);
}

SpanQuery

Matches terms based on their proximity within a document. Use SpanTermQuery and SpanNearQuery to build proximity queries:

#![allow(unused)]
fn main() {
use laurus::lexical::query::span::{SpanQuery, SpanTermQuery, SpanNearQuery};

// Find documents where "quick" appears near "fox" (within 3 positions)
let query = SpanNearQuery::new(
    "body",
    vec![
        Box::new(SpanTermQuery::new("body", "quick")) as Box<dyn SpanQuery>,
        Box::new(SpanTermQuery::new("body", "fox")) as Box<dyn SpanQuery>,
    ],
    3,    // slop (max distance between terms)
    true, // in_order (terms must appear in order)
);
}

Scoring

Lexical search results are scored using BM25. The score reflects how relevant a document is to the query:

• Higher term frequency in the document increases the score
• Rarer terms across the index increase the score
• Shorter documents are boosted relative to longer ones

Field Boosts

You can boost specific fields to influence relevance using the SearchRequestBuilder:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

let request = SearchRequestBuilder::new()
    .lexical_query(LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust"))))
    .add_field_boost("title", 2.0)  // title matches count double
    .add_field_boost("body", 1.0)
    .build();
}

Lexical Search Options

Lexical search behavior is controlled via LexicalSearchOptions on the SearchRequest, or by using builder methods on SearchRequestBuilder:

Builder Methods

SearchRequestBuilder provides convenience methods for lexical options:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::{LexicalSearchQuery, SortField, SortOrder};

let request = SearchRequestBuilder::new()
    .lexical_query(LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust"))))
    .lexical_min_score(0.5)
    .lexical_timeout_ms(5000)
    .lexical_parallel(true)
    .sort_by(SortField::Field { name: "date".to_string(), order: SortOrder::Desc })
    .add_field_boost("title", 2.0)
    .add_field_boost("body", 1.0)
    .limit(20)
    .build();
}

Using the Query DSL

Instead of building queries programmatically, you can use the text-based Query DSL:

#![allow(unused)]
fn main() {
use laurus::lexical::QueryParser;
use laurus::analysis::analyzer::standard::StandardAnalyzer;
use std::sync::Arc;

let analyzer = Arc::new(StandardAnalyzer::default());
let parser = QueryParser::new(analyzer).with_default_field("body");

// Simple term
let query = parser.parse("rust")?;

// Boolean
let query = parser.parse("rust AND programming")?;

// Phrase
let query = parser.parse("\"machine learning\"")?;

// Field-specific
let query = parser.parse("title:rust AND body:programming")?;

// Fuzzy
let query = parser.parse("programing~2")?;

// Range
let query = parser.parse("year:[2020 TO 2024]")?;
}

See Query DSL for the complete syntax reference.

Filter Result Cache

Filter clauses — tenancy, category, status flags, and similar — are frequently reused across many requests. Re-evaluating them from scratch every time re-walks the same posting lists. Laurus memoises the set of document ids a filter matches so a repeated filter becomes a single lookup instead of a full posting walk.

• Snapshot-scoped, self-invalidating. The cache lives on the reader, which is rebuilt on every commit() / optimize() / refresh(). Each reader is a point-in-time snapshot, so the cache needs no manual invalidation: after the index changes, the next search starts from a fresh, empty cache and always reflects committed data.
• Score-independent. A filter selects documents without affecting relevance, so the cached value is a plain doc-id set (a Roaring bitmap). It is used for the filter_query of a hybrid / filtered search and feeds both the lexical and vector sides.
• Reused inside boolean queries. An Occur::Filter clause within a BooleanQuery (e.g. must(user_query).filter(tenant_filter)) also draws its matched set from the cache instead of re-walking postings — including the per-segment fan-out path on multi-segment indexes.
• Safe by construction. Only queries with a canonical key are cached. Term, phrase, prefix, wildcard, regexp, fuzzy, range, geo, and geo3d queries are cacheable, as are boolean queries composed entirely of cacheable clauses with at least one positive (Must / Should / Filter) clause. Boolean queries with no positive clause, span queries, and multi-field queries are evaluated fresh (never cached), so results are always correct.

The cache is enabled by default. Tune or disable it via the index config:

#![allow(unused)]
fn main() {
use laurus::lexical::store::config::LexicalIndexConfig;

let config = LexicalIndexConfig::builder()
    .query_filter_cache_capacity(4096) // entries per snapshot; 0 disables the cache
    .build();
}

Parsed Query Cache

Searching with a DSL string (SearchRequest::from_dsl, or a LexicalSearchQuery::Dsl) parses the string with the pest grammar and re-tokenises its terms with the analyzer on every call. Autocomplete and popular-query workloads repeat the same strings, so Laurus memoises dsl string → parsed query: a repeated DSL string is parsed once and then reused (a cheap clone of the parsed query tree).

Like the filter cache, it is snapshot-scoped: the cache lives on the searcher, which is rebuilt on every commit() / optimize() / refresh(). The analyzer and default fields are fixed for that searcher, so the DSL string alone is the key; a schema/analyzer change yields a fresh, empty cache. Enabled by default; tune or disable via the index config:

#![allow(unused)]
fn main() {
use laurus::lexical::store::config::LexicalIndexConfig;

let config = LexicalIndexConfig::builder()
    .parsed_query_cache_capacity(2048) // entries per snapshot; 0 disables the cache
    .build();
}

Posting Cache

Evaluating a term reads its posting list from the segment’s .post file and decodes it (varint doc-ids, deletion filtering, skip table). Without caching, every query for the same term repeats that read + decode — and on cloud/remote storage the read dominates. Each segment reader keeps a small cache of decoded, deletion-filtered posting lists, so a repeated (field, term) lookup within a snapshot reuses the decoded list.

Because a segment is immutable for a reader snapshot, the cached list is always consistent with its deletions; a commit builds new segment readers with empty caches. The cache is byte-budget bounded (posting lists vary widely in size) — least-recently-used lists are evicted once the budget is exceeded, and a single list larger than the whole budget is not cached. It is enabled by default and shares the max_cache_memory budget; control it via the index config:

#![allow(unused)]
fn main() {
use laurus::lexical::store::config::LexicalIndexConfig;
use laurus::lexical::index::config::InvertedIndexConfig;

let mut inverted = InvertedIndexConfig::default();
inverted.enable_posting_cache = false;        // disable entirely
inverted.max_cache_memory = 256 * 1024 * 1024; // or resize the cache budget (bytes)
let config = LexicalIndexConfig::Inverted(inverted);
}

Next Steps

• Semantic similarity search: Vector Search
• Combine lexical + vector: Hybrid Search
• Full DSL syntax reference: Query DSL

Vector Search

Vector search finds documents by semantic similarity. Instead of matching keywords, it compares the meaning of the query against document embeddings in vector space.

Basic Usage

Builder API

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

let request = SearchRequestBuilder::new()
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "embedding".to_string(),
                payload: DataValue::Text("systems programming language".to_string()),
                weight: 1.0,
            },
        ])
    )
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

The QueryPayload stores raw data (text, bytes, etc.) that will be embedded at search time using the configured embedder.

Query DSL

#![allow(unused)]
fn main() {
use laurus::vector::VectorQueryParser;

let parser = VectorQueryParser::new(embedder.clone())
    .with_default_field("embedding");

let request = parser.parse(r#"embedding:"systems programming""#).await?;
}

VectorSearchQuery

The vector search query is specified as a VectorSearchQuery enum:

QueryPayload

QueryVector

Examples

#![allow(unused)]
fn main() {
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::{QueryPayload, QueryVector};
use laurus::vector::core::vector::Vector;
use laurus::data::DataValue;

// Text query (will be embedded at search time)
let query = VectorSearchQuery::Payloads(vec![
    QueryPayload {
        field: "text_vec".to_string(),
        payload: DataValue::Text("machine learning".to_string()),
        weight: 1.0,
    },
]);

// Pre-computed vector
let query = VectorSearchQuery::Vectors(vec![
    QueryVector {
        vector: Vector::from(vec![0.1, 0.2, 0.3]),
        weight: 1.0,
        fields: Some(vec!["embedding".to_string()]),
    },
]);
}

Multi-Field Vector Search

You can search across multiple vector fields in a single request:

#![allow(unused)]
fn main() {
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

let query = VectorSearchQuery::Payloads(vec![
    QueryPayload {
        field: "text_vec".to_string(),
        payload: DataValue::Text("cute kitten".to_string()),
        weight: 1.0,
    },
    QueryPayload {
        field: "image_vec".to_string(),
        payload: DataValue::Text("fluffy cat".to_string()),
        weight: 1.0,
    },
]);
}

Each clause produces a vector that is searched against its respective field. Results are combined using the configured score mode.

Score Modes

Parallel Multi-Vector Execution

When a request carries multiple query vectors (e.g., ColBERT-style late interaction, multi-vector MaxSim, ensemble rerankers), Laurus dispatches the per-query similarity searches in parallel via rayon.

Behaviour:

• The parallelisation lives in VectorIndexSearcher::search_batch’s default implementation. On native builds (default native feature), once queries.len() reaches the searcher’s parallel_threshold (default 4, overridable per searcher), the per-query HNSW / Flat / IVF searches run on rayon’s global thread pool. Below that threshold the serial loop wins because rayon’s dispatch overhead (~1-2 µs) would otherwise dominate a single 50-200 µs query.
• On wasm32 targets the serial path is always used because rayon is unavailable.
• Aggregation (the score-mode merge) and the final sort run serially after the per-query phase. Score ties are broken by ascending doc_id so the results are deterministic regardless of rayon’s work-stealing schedule.

The external API surface (VectorStore::search, gRPC Search, REST POST /v1/search, all language bindings) is unchanged; parallel execution is purely an internal optimisation enabled by upgrading laurus. Speedups scale with the host’s available cores — on a 4-core / 8-thread laptop CPU the parallel path reaches roughly 2× throughput at B = 64 query vectors, limited by physical core count and HyperThreading sharing.

Parallel Brute-Force Scan

Flat and IVF indexes rank candidates with an exhaustive distance scan rather than a graph walk. When one query’s candidate count reaches an internal threshold (2048), that scan is dispatched across rayon’s global thread pool; below it the serial loop wins because rayon’s per-job dispatch (~1-2 µs) would dominate a small scan.

This is orthogonal to the per-query parallelism above: a batch parallelises across queries, and each large query further parallelises its own scan on the same pool, with work-stealing bounding total parallelism to the pool size (no OS-thread oversubscription). The distance kernel has no side effects, so results are collected in arbitrary order and then sorted, keeping output deterministic. On wasm32 (no rayon) the scan is always serial.

Speedup scales with the host’s physical cores and is largest for big Flat indexes or a wide IVF n_probe; scans below the threshold are unaffected.

IVF Cluster Selection

Before the distance scan, an IVF query first chooses which clusters to scan: it scores the query against every centroid and keeps the n_probe nearest. Because the probed clusters are then merged and re-ranked by similarity, the centroids’ relative order does not matter, so the nearest n_probe are taken with an O(K) partial selection (select_nth_unstable_by) over the K centroids rather than a full O(K log K) sort. The saving grows with the cluster count K; at K = 2048 it cut the per-query coarse step by roughly 18% (Issue #668). The centroid scan itself stays serial — each centroid is a single distance computation, so dispatching it to rayon costs more than it saves at realistic cluster counts (K ≈ √N).

Weights

Use the ^ boost syntax in DSL or weight in QueryVector to adjust how much each field contributes:

text_vec:"cute kitten"^1.0 image_vec:"fluffy cat"^0.5

This means text similarity counts twice as much as image similarity.

Field Routing

In a multi-field schema, each vector field has its own HNSW graph. By default a query searches every vector field; restricting it to named fields skips the others’ graph work entirely.

Two routing inputs are honored, in priority order:

1. Per-query — QueryVector.fields. The DSL parser sets this from the field a clause names, so image_vec:"fluffy cat" only searches image_vec.
2. Request-level — VectorSearchParams.fields, a list of selectors:
   • Exact("image_vec") — match a field by exact name.
   • Prefix("image_") — match every field whose name starts with the prefix (resolved against the index’s field names).

When neither is set, all fields are searched (the default). A query routed to a field never returns documents that lack a vector in that field.

You can apply lexical filters to narrow the vector search results:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;
use laurus::lexical::TermQuery;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

// Vector search with a category filter
let request = SearchRequestBuilder::new()
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "embedding".to_string(),
                payload: DataValue::Text("machine learning".to_string()),
                weight: 1.0,
            },
        ])
    )
    .filter_query(Box::new(TermQuery::new("category", "tutorial")))
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

The filter query runs first on the lexical index to identify allowed document IDs, then the vector search is restricted to those IDs.

Filter-Aware HNSW Traversal

For HNSW fields the allowed-ID set is pushed into the graph search itself, not just applied afterwards. During traversal the frontier still expands through every neighbour — including non-matching ones — so the search can cross non-matching regions of the graph, but only matching documents enter the result set.

This matters for selective filters. A plain post-filter inspects the fixed ef_search window of nearest neighbours and keeps whichever happen to match; when matches are rare they can fall entirely outside that window, so the query returns far fewer hits than exist — sometimes none, even though matching documents are reachable. Filter-aware traversal keeps searching until it has collected enough matches or hits an internal visit cap (a multiple of ef_search) that bounds latency for very selective filters.

The unfiltered path is unchanged: with no filter the traversal behaves exactly as before. Flat and IVF fields honour the allow-set inline — a candidate whose document ID is not in the set is skipped before the distance kernel runs, so a selective filter avoids the wasted distance computations a post-filter would incur. Their scan is exhaustive either way, so recall is unchanged; the store’s post-filter still runs afterwards as a redundant safety net.

When the allow-set is smaller than ef_search, the HNSW field skips the graph walk entirely and scores the allowed documents directly. With so few candidates there is nothing for the graph to find; the direct scan touches exactly the allowed documents — never more than the walk would — and is exact, so a very selective filter returns the true nearest matches rather than an approximation. Larger allow-sets keep using the filter-aware traversal above.

Deletion-Aware HNSW Traversal

Logically deleted documents stay in the HNSW graph until compaction (see Deletions & Compaction), so the walk must skip them the same way it skips filtered-out documents. The graph traversal applies a single admission rule: a node enters the result set only if it matches the filter (when one is present) and is not deleted, while the frontier still expands through deleted nodes to preserve connectivity.

This is what keeps recall correct as deletions accumulate. If deleted nodes were allowed into the result heap, they would consume the fixed ef_search slots and push out live neighbours — in the worst case an ef_search window made entirely of deleted documents would return nothing. Skipping them during the walk means the same slots fill with live results instead, so a 10-hit page stays full even after the nearest documents are deleted. The exact tiny-allow-set scan above and the Flat/IVF inline paths apply the same deletion check.

The fast path is preserved: when a search has neither a filter nor any deletions, the traversal runs the original loop unchanged, paying nothing for the per-neighbour admission bookkeeping.

Allow-Set Representation

The allow-set is a typed structure chosen by shape: a Roaring bitmap for dense filters and a hash set for sparse ones. For a filtered hybrid search the lexical side already produced the matching document set as a bitmap (see the lexical Filter Result Cache); that bitmap is handed to the vector side as-is, so the set is materialised once for the whole query instead of being rebuilt for each side. This is an internal optimisation — the public filter API is unchanged.

Filter with Numeric Range

#![allow(unused)]
fn main() {
use laurus::lexical::NumericRangeQuery;
use laurus::lexical::core::field::NumericType;

let request = SearchRequestBuilder::new()
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "embedding".to_string(),
                payload: DataValue::Text("type systems".to_string()),
                weight: 1.0,
            },
        ])
    )
    .filter_query(Box::new(NumericRangeQuery::new(
        "year", NumericType::Integer,
        Some(2020.0), Some(2024.0), true, true
    )))
    .limit(10)
    .build();
}

Distance Metrics

The distance metric is configured per field in the schema (see Vector Indexing):

Code Example: Complete Vector Search

use std::sync::Arc;
use laurus::{Document, Engine, Schema, SearchRequestBuilder, PerFieldEmbedder};
use laurus::lexical::TextOption;
use laurus::vector::HnswOption;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_hnsw_field("text_vec", HnswOption {
            dimension: 384,
            ..Default::default()
        })
        .build();

    // Set up per-field embedder
    let embedder = Arc::new(my_embedder);
    let pfe = PerFieldEmbedder::new(embedder.clone());
    pfe.add_embedder("text_vec", embedder.clone());

    let engine = Engine::builder(storage, schema)
        .embedder(Arc::new(pfe))
        .build()
        .await?;

    // Index documents (text in vector field is auto-embedded)
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming")
        .add_text("text_vec", "Rust is a systems programming language.")
        .build()
    ).await?;
    engine.commit().await?;

    // Search by semantic similarity
    let results = engine.search(
        SearchRequestBuilder::new()
            .vector_query(
                VectorSearchQuery::Payloads(vec![
                    QueryPayload {
                        field: "text_vec".to_string(),
                        payload: DataValue::Text("systems language".to_string()),
                        weight: 1.0,
                    },
                ])
            )
            .limit(5)
            .build()
    ).await?;

    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }

    Ok(())
}

Next Steps

• Combine with keyword search: Hybrid Search
• DSL syntax for vector queries: Query DSL

Hybrid Search

Hybrid search combines lexical search (keyword matching) with vector search (semantic similarity) to deliver results that are both precise and semantically relevant. This is Laurus’s most powerful search mode.

Why Hybrid Search?

How It Works

sequenceDiagram
    participant User
    participant Engine
    participant Lexical as LexicalStore
    participant Vector as VectorStore
    participant Fusion

    User->>Engine: SearchRequest\n(lexical + vector)

    par Execute in parallel
        Engine->>Lexical: BM25 keyword search
        Lexical-->>Engine: Ranked hits (by relevance)
    and
        Engine->>Vector: ANN similarity search
        Vector-->>Engine: Ranked hits (by distance)
    end

    Engine->>Fusion: Merge two result sets
    Note over Fusion: RRF or WeightedSum
    Fusion-->>Engine: Unified ranked list
    Engine-->>User: Vec of SearchResult

Basic Usage

Builder API

#![allow(unused)]
fn main() {
use laurus::{SearchRequestBuilder, FusionAlgorithm};
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;

let request = SearchRequestBuilder::new()
    // Lexical component
    .lexical_query(
        LexicalSearchQuery::Obj(
            Box::new(TermQuery::new("body", "rust"))
        )
    )
    // Vector component
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "text_vec".to_string(),
                payload: DataValue::Text("systems programming".to_string()),
                weight: 1.0,
            },
        ])
    )
    // Fusion algorithm
    .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

Query DSL

Mix lexical and vector clauses in a single query string:

#![allow(unused)]
fn main() {
use laurus::UnifiedQueryParser;
use laurus::lexical::QueryParser;
use laurus::vector::VectorQueryParser;

let unified = UnifiedQueryParser::new(
    QueryParser::new(analyzer).with_default_field("body"),
    VectorQueryParser::new(embedder),
);

// Lexical + vector in one query
let request = unified.parse(r#"body:rust text_vec:"systems programming""#).await?;
let results = engine.search(request).await?;
}

The parser uses the schema to identify vector clauses by field type. Fields defined as vector fields (e.g., HNSW) are parsed as vector queries; everything else is parsed as lexical.

Fusion Algorithms

When both lexical and vector results exist, they must be merged into a single ranked list. Laurus supports two fusion algorithms:

RRF (Reciprocal Rank Fusion)

The default algorithm. Combines results based on their rank positions rather than raw scores.

score(doc) = sum( 1 / (k + rank_i) )

Where rank_i is the position of the document in each result list, and k is a smoothing parameter (default 60).

#![allow(unused)]
fn main() {
use laurus::FusionAlgorithm;

let fusion = FusionAlgorithm::RRF { k: 60.0 };
}

Advantages:

• Robust to different score distributions between lexical and vector results
• No need to tune weights
• Works well out of the box

WeightedSum

Linearly combines normalized lexical and vector scores:

score(doc) = lexical_weight * lexical_score + vector_weight * vector_score

#![allow(unused)]
fn main() {
use laurus::FusionAlgorithm;

let fusion = FusionAlgorithm::WeightedSum {
    lexical_weight: 0.3,
    vector_weight: 0.7,
};
}

When to use:

• When you want explicit control over the balance between lexical and vector relevance
• When you know one signal is more important than the other

SearchRequest Fields

SearchResult

Each result contains:

Filtered Hybrid Search

Apply a filter to restrict both lexical and vector results:

#![allow(unused)]
fn main() {
let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust")))
    )
    .vector_query(
        VectorSearchQuery::Payloads(vec![
            QueryPayload {
                field: "text_vec".to_string(),
                payload: DataValue::Text("systems programming".to_string()),
                weight: 1.0,
            },
        ])
    )
    // Only search within "tutorial" category
    .filter_query(Box::new(TermQuery::new("category", "tutorial")))
    .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
    .limit(10)
    .build();
}

How Filtering Works

1. The filter query runs on the lexical index to produce a set of allowed document IDs
2. For lexical search: the filter is combined with the user query as a boolean AND
3. For vector search: the allowed IDs are passed to restrict the ANN search

Pagination

Use offset and limit for pagination:

#![allow(unused)]
fn main() {
// Page 1: results 0-9
let page1 = SearchRequestBuilder::new()
    .lexical_query(/* ... */)
    .vector_query(/* ... */)
    .offset(0)
    .limit(10)
    .build();

// Page 2: results 10-19
let page2 = SearchRequestBuilder::new()
    .lexical_query(/* ... */)
    .vector_query(/* ... */)
    .offset(10)
    .limit(10)
    .build();
}

Complete Example

use std::sync::Arc;
use laurus::{
    Document, Engine, Schema, SearchRequestBuilder,
    FusionAlgorithm, PerFieldEmbedder,
};
use laurus::lexical::{TextOption, TermQuery};
use laurus::lexical::core::field::IntegerOption;
use laurus::lexical::search::searcher::LexicalSearchQuery;
use laurus::vector::HnswOption;
use laurus::vector::search::searcher::VectorSearchQuery;
use laurus::vector::store::request::QueryPayload;
use laurus::data::DataValue;
use laurus::storage::memory::MemoryStorage;

#[tokio::main]
async fn main() -> laurus::Result<()> {
    let storage = Arc::new(MemoryStorage::new(Default::default()));

    // Schema with both lexical and vector fields
    let schema = Schema::builder()
        .add_text_field("title", TextOption::default())
        .add_text_field("body", TextOption::default())
        .add_text_field("category", TextOption::default())
        .add_integer_field("year", IntegerOption::default())
        .add_hnsw_field("body_vec", HnswOption {
            dimension: 384,
            ..Default::default()
        })
        .build();

    // Configure analyzer and embedder (see Text Analysis and Embeddings docs)
    // let analyzer = Arc::new(StandardAnalyzer::new()?);
    // let embedder = Arc::new(CandleBertEmbedder::new("sentence-transformers/all-MiniLM-L6-v2")?);
    let engine = Engine::builder(storage, schema)
        // .analyzer(analyzer)
        // .embedder(embedder)
        .build()
        .await?;

    // Index documents with both text and vector fields
    engine.add_document("doc-1", Document::builder()
        .add_text("title", "Rust Programming Guide")
        .add_text("body", "Rust is a systems programming language.")
        .add_text("category", "programming")
        .add_integer("year", 2024)
        .add_text("body_vec", "Rust is a systems programming language.")
        .build()
    ).await?;
    engine.commit().await?;

    // Hybrid search: keyword "rust" + semantic "systems language"
    let results = engine.search(
        SearchRequestBuilder::new()
            .lexical_query(
                LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust")))
            )
            .vector_query(
                VectorSearchQuery::Payloads(vec![
                    QueryPayload {
                        field: "body_vec".to_string(),
                        payload: DataValue::Text("systems language".to_string()),
                        weight: 1.0,
                    },
                ])
            )
            .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
            .limit(10)
            .build()
    ).await?;

    for r in &results {
        println!("{}: score={:.4}", r.id, r.score);
    }

    Ok(())
}

Next Steps

• Full query syntax reference: Query DSL
• Understand ID resolution: ID Management
• Data durability: Persistence & WAL

Query DSL

Laurus provides a unified query DSL (Domain Specific Language) that allows lexical (keyword) and vector (semantic) search in a single query string. The UnifiedQueryParser splits the input into lexical and vector portions and delegates to the appropriate sub-parser.

Overview

title:hello AND content:"cute kitten"^0.8
|--- lexical --|    |--- vector --------|

The field type in the schema determines whether a clause is lexical or vector. If the field is a vector field (e.g., HNSW), the clause is treated as a vector query. Everything else is treated as a lexical query.

Field validation

Every field:value clause is validated against the schema at parse time. A query that references a field not declared in the schema is rejected with an error rather than returning silently-empty results. This catches typos early (e.g. titl:hello instead of title:hello).

If you want the engine to accept documents with previously-unknown fields, set the schema’s dynamic_field_policy so that the field gets added during ingestion. Once a field is part of the schema, queries referencing it succeed.

Lexical Query Syntax

Lexical queries search the inverted index using exact or approximate keyword matching.

Term Query

Match a single term against a field (or the default field):

hello
title:hello

Boolean Operators

Combine clauses with AND and OR (case-insensitive):

title:hello AND body:world
title:hello OR title:goodbye

AND is symmetric — it makes the clauses on both sides required (Must). For example, title:hello AND body:world returns only documents that match both clauses. The same is true for chains: a AND b AND c requires all three. A clause that was already explicitly marked with + (required) or - (prohibited) keeps that intent — AND does not override an explicit prefix.

Space-separated clauses without an explicit operator use implicit boolean (behaves like OR with scoring), so a b AND c reads as “optionally match a, and require both b and c”.

Required / Prohibited Clauses

Use + (must match) and - (must not match):

+title:hello -title:goodbye

Phrase Query

Match an exact phrase using double quotes. Optional proximity (~N) allows N words between terms:

"hello world"
"hello world"~2

Fuzzy Query

Approximate matching with edit distance. Append ~ and optionally the maximum edit distance:

roam~
roam~2

Wildcard Query

Use ? (single character) and * (zero or more characters):

te?t
test*

Range Query

Inclusive [] or exclusive {} ranges, useful for numeric and date fields:

price:[100 TO 500]
date:{2024-01-01 TO 2024-12-31}
price:[* TO 100]

2D Geographic Queries (geo_*)

Two function-style forms target Geo (2D latitude / longitude) fields. All arguments are signed floats; latitudes / longitudes are in degrees and the distance is in metres:

location:geo_distance(lat, lon, distance_m)
location:geo_bbox(min_lat, min_lon, max_lat, max_lon)

Examples:

# Within 10 km (= 10 000 m) of Tokyo (35.6895, 139.6917)
location:geo_distance(35.6895, 139.6917, 10000)

# Inside an axis-aligned lat/lon bounding box
location:geo_bbox(35.0, 139.0, 36.0, 140.0)

The query field must be declared as a Geo field in the schema. Latitudes must lie in [-90, 90] and longitudes in [-180, 180]; the parser rejects out-of-range values.

3D Geographic Queries (geo3d_*)

Three function-style forms target Geo3d (3D ECEF Cartesian) fields. All arguments are signed floats in metres, except k (an unsigned integer):

position:geo3d_distance(x, y, z, distance_m)
position:geo3d_bbox(min_x, min_y, min_z, max_x, max_y, max_z)
position:geo3d_nearest(x, y, z, k)

Examples:

# Within 5 km of Tokyo Tower (ECEF coordinates)
position:geo3d_distance(-3955182, 3350553, 3700276, 5000)

# Inside an axis-aligned ECEF bounding box
position:geo3d_bbox(-4000000, 3300000, 3650000, -3900000, 3400000, 3750000)

# 10 nearest indexed points
position:geo3d_nearest(-3955182, 3350553, 3700276, 10)

The query field must be declared as a Geo3d field in the schema. See 3D Geographic Search for the underlying coordinate system, WGS84 conversion helpers, and detailed semantics.

Boost

Increase the weight of a clause with ^:

title:hello^2
"important phrase"^1.5

Grouping

Use parentheses for sub-expressions:

(title:hello OR title:hi) AND body:world

Lexical PEG Grammar

The full lexical grammar (parser.pest):

query          = { SOI ~ boolean_query ~ EOI }
boolean_query  = { clause ~ (boolean_op ~ clause | clause)* }
clause         = { required_clause | prohibited_clause | sub_clause }
required_clause   = { "+" ~ sub_clause }
prohibited_clause = { "-" ~ sub_clause }
sub_clause     = { grouped_query | field_query | term_query }
grouped_query  = { "(" ~ boolean_query ~ ")" ~ boost? }
boolean_op     = { ^"AND" | ^"OR" }
field_query    = { field ~ ":" ~ field_value }
field_value    = { geo3d_query | geo_query | range_query | phrase_query
                 | fuzzy_term | wildcard_term | simple_term }
geo3d_query    = { geo3d_distance | geo3d_bbox | geo3d_nearest }
geo3d_distance = { ^"geo3d_distance" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ signed_float ~ ")" }
geo3d_bbox     = { ^"geo3d_bbox" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ signed_float ~ ","
                 ~ signed_float ~ "," ~ signed_float ~ ")" }
geo3d_nearest  = { ^"geo3d_nearest" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ unsigned_int ~ ")" }
geo_query      = { geo_distance | geo_bbox }
geo_distance   = { ^"geo_distance" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ ")" }
geo_bbox       = { ^"geo_bbox" ~ "(" ~ signed_float ~ "," ~ signed_float
                 ~ "," ~ signed_float ~ "," ~ signed_float ~ ")" }
phrase_query   = { "\"" ~ phrase_content ~ "\"" ~ proximity? ~ boost? }
proximity      = { "~" ~ number }
fuzzy_term     = { term ~ "~" ~ fuzziness? ~ boost? }
wildcard_term  = { wildcard_pattern ~ boost? }
simple_term    = { term ~ boost? }
boost          = { "^" ~ boost_value }

Vector Query Syntax

Vector queries embed text into vectors at parse time and perform similarity search.

Basic Syntax

field:"text"
field:text
field:"text"^weight

The field name must refer to a vector field defined in the schema. The parser uses the schema to determine whether a clause is a vector query.

Vector Query Examples

# Single field (quoted text)
content:"cute kitten"

# Unquoted text
content:python

# With boost weight
content:"cute kitten"^0.8

# Multiple clauses
content:"cats" image:"dogs"^0.5

# Nested field name (dot notation)
metadata.embedding:"text"

Multiple Clauses

Multiple vector clauses are space-separated. All clauses are executed and their scores are combined using the score_mode (default: WeightedSum):

content:"cats" image:"dogs"^0.5

This produces:

score = similarity("cats", content) * 1.0
      + similarity("dogs", image)   * 0.5

There are no AND/OR operators in the vector DSL. Vector search is inherently a ranking operation, and the weight (^) controls the contribution of each clause.

Score Modes

Score mode cannot be set from DSL syntax. Use the Rust API to override:

#![allow(unused)]
fn main() {
let mut request = parser.parse(r#"content:"cats" image:"dogs""#).await?;
request.vector_options.score_mode = VectorScoreMode::MaxSim;
}

Vector PEG Grammar

The full vector grammar (parser.pest):

query          = { SOI ~ vector_clause+ ~ EOI }
vector_clause  = { field_prefix ~ (quoted_text | unquoted_text) ~ boost? }
field_prefix   = { field_name ~ ":" }
field_name     = @{ (ASCII_ALPHA | "_") ~ (ASCII_ALPHANUMERIC | "_" | ".")* }
quoted_text    = ${ "\"" ~ inner_text ~ "\"" }
inner_text     = @{ (!("\"") ~ ANY)* }
unquoted_text  = @{ (!(" " | "^" | "\"") ~ ANY)+ }
boost          = { "^" ~ float_value }
float_value    = @{ ASCII_DIGIT+ ~ ("." ~ ASCII_DIGIT+)? }

Unified (Hybrid) Query Syntax

The UnifiedQueryParser allows mixing lexical and vector clauses freely in a single query string:

title:hello content:"cute kitten"^0.8

How It Works

1. Split: The parser checks each field name against the schema. Fields defined as vector fields (e.g., HNSW, Flat, IVF) are routed to the vector parser; all other fields are routed to the lexical parser.
2. Delegate: Vector portion goes to VectorQueryParser, remainder goes to lexical QueryParser.
3. Fuse: If both lexical and vector results exist, they are combined using a fusion algorithm.

Disambiguation

The parser uses the schema’s field type information to distinguish vector clauses from lexical clauses. A clause like content:"cute kitten" is a vector query if content is a vector field, or a phrase query if content is a text field. Lexical ~ syntax (e.g., roam~2 for fuzzy, "hello world"~10 for proximity) is unaffected.

Fusion Algorithms

When a query contains both lexical and vector clauses, results are fused:

Note: The fusion algorithm cannot be specified in the DSL syntax. It is configured when constructing the UnifiedQueryParser via .with_fusion(). The default is RRF (k=60). See Custom Fusion for a code example.

Hybrid AND/OR Semantics (the + Prefix)

By default, hybrid queries use union (OR) — documents appearing in either the lexical results or the vector results are included. You can switch to intersection (AND) by prefixing a vector clause with +, which requires documents to appear in both result sets.

Rules:

• When no vector clause carries the + prefix, the fusion produces a union (OR) of lexical and vector results.
• When at least one vector clause carries the + prefix, the fusion switches to intersection (AND) — only documents present in both the lexical and vector result sets are returned.
• + on a lexical field (e.g., +title:Rust) is interpreted by the lexical query parser as a required clause, which is the existing Tantivy/Lucene-style behaviour. It does not by itself trigger intersection mode for the hybrid fusion.

Unified Query Examples

# Lexical only — no fusion
title:hello AND body:world

# Vector only — no fusion
content:"cute kitten"

# Hybrid — fusion applied automatically (OR / union)
title:hello content:"cute kitten"

# Hybrid with AND / intersection — only docs in both result sets
title:hello +content:"cute kitten"

# Hybrid with boolean operators
title:hello AND category:animal content:"cute kitten"^0.8

# Multiple vector clauses + lexical
category:animal content:"cats" image:"dogs"^0.5

# Unquoted vector text
category:animal content:python

Code Examples

Lexical Search with DSL

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::analysis::analyzer::standard::StandardAnalyzer;
use laurus::lexical::query::QueryParser;

let analyzer = Arc::new(StandardAnalyzer::new()?);
let parser = QueryParser::new(analyzer)
    .with_default_field("title");

let query = parser.parse("title:hello AND body:world")?;
}

Vector Search with DSL

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::vector::query::VectorQueryParser;

let parser = VectorQueryParser::new(embedder)
    .with_default_field("content");

let request = parser.parse(r#"content:"cute kitten"^0.8"#).await?;
}

Hybrid Search with Unified DSL

#![allow(unused)]
fn main() {
use laurus::engine::query::UnifiedQueryParser;

let unified = UnifiedQueryParser::new(lexical_parser, vector_parser);

let request = unified.parse(
    r#"title:hello content:"cute kitten"^0.8"#
).await?;
// request.query              -> SearchQuery::Hybrid { lexical, vector }
// request.fusion_algorithm   -> Some(RRF)  — fusion algorithm
}

Custom Fusion

#![allow(unused)]
fn main() {
use laurus::engine::search::FusionAlgorithm;

let unified = UnifiedQueryParser::new(lexical_parser, vector_parser)
    .with_fusion(FusionAlgorithm::WeightedSum {
        lexical_weight: 0.3,
        vector_weight: 0.7,
    });
}

BKD-Tree

Laurus stores numeric, datetime, and geographic point data in a BKD-Tree (Block KD-Tree) — a disk-resident, multi-dimensional index that supports range, bounding-box, distance, and k-nearest-neighbour queries on the same underlying file format.

The BKD primitive is shared by every “spatial-shaped” field type:

Adding a new spatial field type therefore boils down to picking a dimensionality and writing a query-side IntersectVisitor — the writer, reader, and on-disk layout are reused unchanged.

File Format (Version 2)

A .bkd segment file is a self-contained binary blob made of three regions:

+----------------------------------------+
| File Header                            |   fixed-size, version-tagged
+----------------------------------------+
| Leaf Blocks                            |   row-major points + doc_ids
|   leaf 0                               |
|   leaf 1                               |
|   ...                                  |
+----------------------------------------+
| Index Nodes                            |   internal navigation nodes
|   node N-1                             |
|   ...                                  |
|   node 0  (root, written last)         |
+----------------------------------------+

The header (BKDFileHeader) records magic, version (currently 2), num_dims, bytes_per_dim, the total point count, the number of leaf blocks, the per-axis global min/max for the whole tree, and offsets to the index region and the root node.

Leaf Block Layout

Each leaf block stores the points that fall inside its subtree:

count               u32                       — number of points in the leaf
leaf_min            [f64; num_dims]           — leaf-level AABB minimum
leaf_max            [f64; num_dims]           — leaf-level AABB maximum
point_values        [f64; count * num_dims]   — row-major point coordinates
doc_ids             [u64; count]              — parallel doc-id buffer

The per-leaf AABB lets the reader prune the leaf without scanning any of its points when the query region is fully outside (Outside) or fully inside (Inside) the leaf.

Internal Index Node Layout

Internal nodes carry both the split decision and the per-child AABB:

split_dim           u32                       — axis to split on
split_value         f64                       — split threshold on that axis
left_min            [f64; num_dims]           — left subtree AABB min
left_max            [f64; num_dims]           — left subtree AABB max
right_min           [f64; num_dims]           — right subtree AABB min
right_max           [f64; num_dims]           — right subtree AABB max
left_offset         u64                       — file offset of left child
right_offset        u64                       — file offset of right child

Per-node AABBs (added in v2) replace the v1 layout that only carried the split value. They make Inside / Outside pruning a constant-time rectangle test instead of a recursive descent.

Build Algorithm

BKDWriter::write builds the tree from a flat row-major point buffer plus a parallel doc_ids buffer. Construction is driven by the widest-axis split heuristic:

1. Compute the AABB of the input subset.
2. Pick the axis whose (max - min) range is the widest (ties broken by lower dimension index for determinism).
3. Sort the index permutation by that axis and split at the median.
4. Recurse on the two halves until a subtree fits in block_size points (default 512); emit it as a leaf.
5. Back-patch each parent’s left_offset / right_offset once the children have been flushed.

The builder sorts an index permutation rather than the point/doc-id buffers themselves, so it does not allocate per-point storage no matter how many points are written.

Numeric Robustness

Coordinates must be totally orderable. BKDWriter::write rejects NaN explicitly because NaN has no defined ordering and would corrupt the split decisions and per-node AABB invariants. Both ±INFINITY are accepted and act as natural sentinels for “unbounded” semantics in queries.

Query: The IntersectVisitor Protocol

Queries against a BKD index are expressed as an IntersectVisitor implementation. The reader walks the tree and asks the visitor three things:

#![allow(unused)]
fn main() {
pub enum CellRelation {
    Inside,   // entire subtree is a hit — collect without per-point checks
    Outside,  // entire subtree can be skipped
    Crosses,  // recurse, or filter the leaf per-point
}

pub trait IntersectVisitor {
    fn compare(&self, cell: &AABB) -> CellRelation;
    fn visit_inside(&mut self, doc_id: u64);
    fn visit(&mut self, doc_id: u64, point: &[f64]);
}
}

The reader’s traversal is:

graph TD
    A["compare(node.aabb)"]
    A -->|Inside| B["visit_inside(doc_id) for every doc<br/>under the subtree — no point read"]
    A -->|Outside| C["skip subtree"]
    A -->|Crosses, internal| D["recurse into children"]
    A -->|Crosses, leaf| E["visit(doc_id, point) per point<br/>visitor decides what is a hit"]

This three-valued classification is what unlocks pruning. A visitor that always returns Crosses would still produce correct results — it would just degrade to a full leaf scan.

Range Queries

The legacy BKDTree::range_search API is now a thin wrapper around intersect: it constructs a RangeQueryVisitor from the half-open / closed range parameters and converts unbounded None slots into ±INFINITY. The visitor handles inclusive-vs-exclusive boundary checks itself.

3D Geographic Queries

Three additional visitors live in laurus::lexical::query::geo3d and target Geo3d (3D ECEF) fields:

The same primitive could host any future spatial query — for example, polygon queries or great-circle 2D Geo queries — by writing a new visitor.

Reader Internals

BKDReader::intersect uses a single per-query scratch buffer (IntersectScratch) that grows to the size of the largest leaf encountered and is then reused for every subsequent leaf, so a query touches the allocator at most a handful of times regardless of how many leaves it visits.

Single-leaf trees (very small fields) are handled as a special case: the “root offset” points directly at the only leaf and the reader skips the internal-node descent entirely.

See Also

• 3D Geographic Search — concrete BKD-backed visitors for ECEF distance, bounding-box, and k-NN queries.
• Lexical Indexing — where the .bkd segment file fits within the broader segment layout.
• Lexical Search — NumericRangeQuery, GeoQuery, and Geo3dDistanceQuery programmatic entry points.

3D Geographic Search (ECEF)

Laurus offers two geographic field types that target different problems:

Both share the same BKD-Tree primitive; Geo3d simply adds a third dimension and a different query vocabulary.

Why ECEF?

(latitude, longitude, altitude) triples are convenient for humans but make Euclidean distance unusable: a degree of longitude is ~111 km at the equator and 0 km at the poles, and “altitude” is curved with the Earth’s surface.

ECEF (Earth-Centered Earth-Fixed) flattens this into a Cartesian frame:

• The origin is the Earth’s centre of mass.
• The +X axis points through the equator at 0° longitude.
• The +Y axis points through the equator at 90° E longitude.
• The +Z axis points through the geographic North Pole.
• All three axes are in metres.

In this frame the straight-line distance between two points is just the Euclidean norm — no spherical trigonometry, no pole singularity, no longitude-wrap. That property is what makes Geo3d queries usable from a 3D BKD-Tree without bespoke spatial code.

Defining a Geo3d Field

#![allow(unused)]
fn main() {
use laurus::Schema;
use laurus::lexical::core::field::Geo3dOption;

let schema = Schema::builder()
    .add_geo3d_field("position", Geo3dOption::default())
    .build();
}

Geo3dOption exposes the same indexed / stored switches as the other lexical field options. Indexed values populate the field’s 3D BKD-Tree; stored values are returned by get_documents.

The TOML form (used by laurus-cli):

[fields.position.Geo3d]
indexed = true
stored = true

Indexing 3D Points

Use DocumentBuilder::add_geo_ecef to add a point in raw Cartesian metres:

#![allow(unused)]
fn main() {
use laurus::Document;

// A point at lat=35.6586°, lon=139.7454°, height=250 m
// (already converted to ECEF — see "Coordinate Conversion" below).
let doc = Document::builder()
    .add_geo_ecef("position", -3_955_182.0, 3_350_553.0, 3_700_276.0)
    .build();

engine.put_document("tokyo-tower", doc).await?;
engine.commit().await?;
}

If the value already exists as a GeoEcefPoint, build it through the unified DataValue API instead:

#![allow(unused)]
fn main() {
use laurus::{DataValue, GeoEcefPoint};

let p = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);
let doc = Document::builder()
    .add_field("position", DataValue::GeoEcef(p))
    .build();
}

Coordinate Conversion (WGS84 ↔ ECEF)

Most input arrives as (latitude°, longitude°, height_m). Laurus exposes the canonical pair of conversions in laurus::util::ecef:

#![allow(unused)]
fn main() {
use laurus::util::ecef::{wgs84_to_ecef, ecef_to_wgs84};

// Forward: lat/lon/height → ECEF
let p = wgs84_to_ecef(35.6586, 139.7454, 250.0);

// Reverse: ECEF → lat/lon/height
let (lat, lon, height) = ecef_to_wgs84(&p);
}

The reverse conversion round-trips with sub-µm precision on every axis from sub-surface up to far beyond LEO altitudes. Near the poles the implementation switches from the standard h = p / cos(lat) - N formula (which diverges at cos(lat) → 0) to the meridian form h = z / sin(lat) - N(1 - e²).

The WGS84 ellipsoid constants used internally are also exposed as pub consts (WGS84_A, WGS84_F, WGS84_B, WGS84_E2, WGS84_E_PRIME_SQ) so external code can reference the exact values laurus uses.

Query Types

Three query types target Geo3d fields. All three are IntersectVisitor implementations on top of the shared 3D BKD-Tree.

Sphere (radius) — Geo3dDistanceQuery

Find every document whose stored point lies within distance_m metres of a centre point. Score is 1 - distance / radius, clamped to [0, 1].

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo3d::Geo3dDistanceQuery;
use laurus::GeoEcefPoint;

let centre = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);
let query = Geo3dDistanceQuery::new("position", centre, 5_000.0); // 5 km radius
}

The visitor uses AABB::min_distance_sq_to_point for the Outside test and AABB::max_distance_sq_to_point for the Inside test, so both cell-vs-sphere checks run in squared-distance space and avoid sqrt.

Bounding Box — Geo3dBoundingBoxQuery

Find every document whose stored point lies inside an axis-aligned 3D box defined by (min_x, min_y, min_z) and (max_x, max_y, max_z).

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo3d::Geo3dBoundingBoxQuery;
use laurus::GeoEcefPoint;

let min = GeoEcefPoint::new(-4_000_000.0, 3_300_000.0, 3_650_000.0);
let max = GeoEcefPoint::new(-3_900_000.0, 3_400_000.0, 3_750_000.0);
let query = Geo3dBoundingBoxQuery::new("position", min, max)?;
}

The constructor returns Result because every min[i] must be ≤ max[i]; mismatched bounds are rejected at construction time rather than producing silently-empty results.

Caveat. The query box is axis-aligned in ECEF Cartesian space. A box built from two (lat, lon, height) corners by converting each corner separately is not the same shape as the spherical-shell region a user might intuitively expect. For region queries on the Earth’s surface, Geo3dDistanceQuery (a true 3D sphere) usually matches user intuition better.

k-Nearest Neighbours — Geo3dNearestQuery

Find the k nearest documents to a query point. The query starts from a small probe sphere (default 1 km) and doubles the radius until either:

1. it has collected at least k distinct candidates,
2. doubling makes no progress,
3. the radius reaches max_radius_m (default 1.0e10 m, i.e. 10⁷ km).

#![allow(unused)]
fn main() {
use laurus::lexical::query::geo3d::Geo3dNearestQuery;
use laurus::GeoEcefPoint;

let centre = GeoEcefPoint::new(-3_955_182.0, 3_350_553.0, 3_700_276.0);
let query = Geo3dNearestQuery::new("position", centre, 10) // 10 nearest
    .with_initial_radius(500.0)         // start at 500 m
    .with_max_radius(1_000_000.0);      // cap at 1 000 km
}

A smaller-than-k result simply means the index does not contain k points within max_radius_m. Score is normalised so that the closest hit gets 1.0 and the farthest hit in the returned set gets 0.0.

The k-NN visitor never returns CellRelation::Inside from compare — only Outside or Crosses — so every candidate is delivered to visit with its exact coordinates, ensuring k-NN ordering uses the true distance.

Query DSL

Geo3d queries are also exposed in the unified Query DSL (see Query DSL → Geo3d Functions):

position:geo3d_distance(-3955182, 3350553, 3700276, 5000)
position:geo3d_bbox(-4000000, 3300000, 3650000, -3900000, 3400000, 3750000)
position:geo3d_nearest(-3955182, 3350553, 3700276, 10)

All numeric arguments are signed floats; the k of geo3d_nearest is an unsigned integer.

Wire Formats

gRPC

Protocol Buffers expose 3D geo as a dedicated Value variant alongside a Geo3dPoint message and a Geo3dOption schema option:

message Geo3dPoint {
  double x = 1;
  double y = 2;
  double z = 3;
}

message Value {
  oneof kind {
    // ...
    Geo3dPoint geo3d_value = 12;
  }
}

message Geo3dOption {
  bool indexed = 1;
  bool stored = 2;
}

See common.proto and index.proto for the full definitions.

HTTP gateway / MCP

JSON documents accept Geo3d values as a { "x": …, "y": …, "z": … } object:

{
  "position": { "x": -3955182.0, "y": 3350553.0, "z": 3700276.0 }
}

The HTTP gateway converts this to DataValue::GeoEcef before forwarding to the engine; the MCP server uses the same convention.

When not to Use Geo3d

• Pure 2D map queries (radius from a coordinate on the surface, simple bounding boxes on a tile map) — keep using Geo. The 2D BKD is one dimension lighter and the WGS84 input matches user intuition.
• Approximate semantic similarity of a learned location embedding — that is a vector field (Hnsw, Flat, Ivf), not a Geo3d field.

See Also

• BKD-Tree — the underlying multi-dimensional index that also backs Integer, Float, DateTime, and 2D Geo fields.
• Schema & Fields — Geo3d alongside the rest of the field type table.
• Query DSL — full geo3d DSL grammar.
• Lexical Search — programmatic entry points for every lexical query type.
• laurus-wasm/examples/geo3d/ — browser-side demo: live aircraft positions on a CesiumJS 3D globe with geo3d_bbox and geo3d_nearest queries.

Library Overview

The laurus crate is the core search engine library. It provides lexical search (keyword matching via inverted index), vector search (semantic similarity via embeddings), and hybrid search (combining both) through a unified API.

Module Structure

graph TB
    LIB["laurus (lib.rs)"]

    LIB --> engine["engine\nEngine, EngineBuilder\nSearchRequest, FusionAlgorithm"]
    LIB --> analysis["analysis\nAnalyzer, Tokenizer\nToken Filters, Char Filters"]
    LIB --> lexical["lexical\nInverted Index, BM25\nQuery Types, Faceting, Highlighting"]
    LIB --> vector["vector\nFlat, HNSW, IVF\nDistance Metrics, Quantization"]
    LIB --> embedding["embedding\nCandle BERT, OpenAI\nCLIP, Precomputed"]
    LIB --> storage["storage\nMemory, File, Mmap\nColumnStorage"]
    LIB --> store["store\nDocumentLog (WAL)"]
    LIB --> spelling["spelling\nSpelling Correction\nSuggestion Engine"]
    LIB --> util["util\nShared helpers and macros"]
    LIB --> data["data\nDataValue, Document"]
    LIB --> error["error\nLaurusError, Result"]

Key Types

Feature Flags

The laurus crate has no default features enabled. Enable embedding support as needed:

# Lexical search only (no embedding)
[dependencies]
laurus = "0.9"

# With local BERT embeddings
[dependencies]
laurus = { version = "0.9", features = ["embeddings-candle"] }

# All features
[dependencies]
laurus = { version = "0.9", features = ["embeddings-all"] }

Sections

• Engine – Engine and EngineBuilder internals
• Scoring & Ranking – BM25, TF-IDF, and vector similarity scoring
• Faceting – Hierarchical facet search
• Highlighting – Search result highlighting
• Spelling Correction – Spelling suggestions and auto-correction
• ID Management – Dual-tiered document identity
• Persistence & WAL – Write-ahead logging and durability
• Deletions & Compaction – Logical deletion and space reclamation
• Error Handling – LaurusError and Result types
• Extensibility – Custom analyzers, embedders, and storage backends
• API Reference – Key types and methods at a glance

Engine

The Engine is the central type in Laurus. It coordinates the lexical index, vector index, and document log behind a single async API.

Engine Struct

#![allow(unused)]
fn main() {
pub struct Engine {
    schema: Schema,
    lexical: LexicalStore,
    vector: VectorStore,
    log: Arc<DocumentLog>,
}
}

EngineBuilder

Use EngineBuilder to configure and create an Engine:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use laurus::{Engine, Schema};
use laurus::lexical::TextOption;
use laurus::storage::memory::MemoryStorage;

let storage = Arc::new(MemoryStorage::new(Default::default()));
let schema = Schema::builder()
    .add_text_field("title", TextOption::default())
    .add_text_field("body", TextOption::default())
    .add_default_field("body")
    .build();

let engine = Engine::builder(storage, schema)
    .analyzer(my_analyzer)            // optional: custom text analyzer
    .embedder(my_embedder)            // optional: vector embedder
    .embedding_cache_capacity(1024)   // optional: cache query embeddings
    .build()
    .await?;
}

Builder Methods

Query embedding cache

embedding_cache_capacity(n) enables an LRU cache for query-time embeddings (document-ingestion embedding is unaffected). When a vector or hybrid search embeds a query payload, the result is cached keyed by (field, embedder name, payload hash) and reused on subsequent identical queries — both the DSL path (e.g. content:"cute kitten") and the pre-embedded Payloads path share the same cache.

This avoids repeated model inference for local embedders, or network round trips for remote ones, on repeated-query workloads (autocomplete, dashboard refreshes, A/B evaluation). It is disabled by default; pick a capacity that bounds memory to your working set of distinct queries.

Within a single query, all of its payloads are embedded in one Embedder::embed_batch call (cache misses only, grouped per field for PerFieldEmbedder), so a batch-capable embedder pays one round trip for a multi-vector query instead of one per payload (Issue #671).

Build Lifecycle

When build() is called, the following steps occur:

sequenceDiagram
    participant User
    participant EngineBuilder
    participant Engine

    User->>EngineBuilder: .build().await
    EngineBuilder->>EngineBuilder: split_schema()
    Note over EngineBuilder: Separate fields into<br/>LexicalIndexConfig<br/>+ VectorIndexConfig
    EngineBuilder->>Engine: Create PrefixedStorage (lexical/, vector/, documents/)
    EngineBuilder->>Engine: Create LexicalStore
    EngineBuilder->>Engine: Create VectorStore
    EngineBuilder->>Engine: Create DocumentLog
    EngineBuilder->>Engine: Recover from WAL
    EngineBuilder-->>User: Engine ready

1. Split schema – Lexical fields (Text, Integer, Float, etc.) go to LexicalIndexConfig, vector fields (HNSW, Flat, IVF) go to VectorIndexConfig
2. Create prefixed storage – Each component gets an isolated namespace (lexical/, vector/, documents/)
3. Initialize stores – LexicalStore and VectorStore are created with their respective configs
4. Recover from WAL – Any uncommitted operations from a previous session are replayed

Schema Splitting

The Schema contains both lexical and vector fields. At build time, split_schema() separates them:

graph LR
    S["Schema<br/>title: Text<br/>body: Text<br/>page: Integer<br/>content_vec: HNSW"]

    S --> LC["LexicalIndexConfig<br/>title: TextOption<br/>body: TextOption<br/>page: IntegerOption<br/>_id: KeywordAnalyzer"]

    S --> VC["VectorIndexConfig<br/>content_vec: HnswOption<br/>(dim=384, m=16, ef=200)"]

The reserved _id field is always added to the lexical config with KeywordAnalyzer for exact match lookups.

Per-Field Dispatch

PerFieldAnalyzer

When a PerFieldAnalyzer is provided, text analysis is dispatched to field-specific analyzers:

graph LR
    PFA["PerFieldAnalyzer"]
    PFA -->|"title"| KA["KeywordAnalyzer"]
    PFA -->|"body"| SA["StandardAnalyzer"]
    PFA -->|"description"| JA["JapaneseAnalyzer"]
    PFA -->|"_id"| KA2["KeywordAnalyzer<br/>(always)"]
    PFA -->|other fields| DEF["Default Analyzer"]

PerFieldEmbedder

Similarly, PerFieldEmbedder routes embedding to field-specific embedders:

graph LR
    PFE["PerFieldEmbedder"]
    PFE -->|"text_vec"| BERT["CandleBertEmbedder<br/>(384 dim)"]
    PFE -->|"image_vec"| CLIP["CandleClipEmbedder<br/>(512 dim)"]
    PFE -->|other fields| DEF["Default Embedder"]

Engine Methods

Document Operations

Search

The search() method accepts a SearchRequest which can contain a lexical query, a vector query, or both. When both are present, results are merged using the specified FusionAlgorithm.

#![allow(unused)]
fn main() {
use laurus::{SearchRequestBuilder, FusionAlgorithm};
use laurus::lexical::TermQuery;
use laurus::lexical::search::searcher::LexicalSearchQuery;

// Lexical-only search
let request = SearchRequestBuilder::new()
    .lexical_query(
        LexicalSearchQuery::Obj(Box::new(TermQuery::new("body", "rust")))
    )
    .limit(10)
    .build();

// Hybrid search with RRF fusion
let request = SearchRequestBuilder::new()
    .lexical_query(lexical_query)
    .vector_query(vector_query)
    .fusion_algorithm(FusionAlgorithm::RRF { k: 60.0 })
    .limit(10)
    .build();

let results = engine.search(request).await?;
}

Warmup

engine.warmup() (Issue #677) primes the vector searcher so the first vector / hybrid query does not pay one-time setup costs. Call it once after building the engine, before serving traffic:

let engine = builder.build()?;
engine.warmup()?;   // optional: move first-query latency to startup

It eagerly builds and caches the vector searcher — loading the reader (file → memory for InMemory, the offset table for Mmap) — and, for HNSW indexes in Mmap mode, pre-faults the on-disk vector data into the OS page cache. The HNSW graph is always loaded eagerly, so only the vector data needs warming. warmup() is safe to call multiple times and is a no-op for lexical-only workloads.

SearchRequest

FusionAlgorithm

See also: Architecture for the high-level data flow diagrams.

Scoring & Ranking

Laurus uses BM25 for lexical search, distance-based similarity for vector search, and a configurable fusion algorithm to combine the two for hybrid search. This page describes each scoring path and how to influence it from the public API.

Lexical Scoring

BM25 (Default)

BM25 is the lexical scoring function. It balances term frequency with document length normalization:

score = IDF * (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (doc_len / avg_doc_len)))

Where:

• tf — term frequency in the document.
• IDF — inverse document frequency (rarity of the term across all documents).
• k1 — term-frequency saturation parameter. Laurus uses 1.2.
• b — document-length normalization factor. Laurus uses 0.75.
• doc_len / avg_doc_len — ratio of document length to average document length.

The (k1, b) parameters are fixed at the implementation defaults today. The values match Lucene / Elasticsearch defaults, so BM25 scores from Laurus are directly comparable to those engines for tuning intuition.

Field Boosts

Per-field score multipliers are configured on the search request, not in a separate scoring struct:

#![allow(unused)]
fn main() {
use laurus::SearchRequestBuilder;

let request = SearchRequestBuilder::new()
    .query_dsl("rust programming")
    .add_field_boost("title", 2.0) // title matches score 2x
    .add_field_boost("body", 1.0)  // body matches score 1x (the default)
    .limit(10)
    .build();
}

The boost is multiplied into the BM25 score contribution of matches in that field. A boost of 1.0 is a no-op; boosts apply only to fields named in the query (or in the schema’s default-search fields).

Over gRPC and HTTP, the same setting is exposed as SearchRequest.field_boosts (map<string, float>). See gRPC API → SearchRequest.

Vector Scoring

Vector search ranks results by distance-based similarity. The distance metric is configured per field on the vector index (HNSW / Flat / IVF):

Distances are converted to similarity scores so that “higher is better” holds across both lexical and vector results, which is what the fusion algorithms below assume.

Hybrid Search Fusion

When a search request contains both lexical and vector clauses, the two result lists need to be merged. Laurus exposes two fusion algorithms via FusionAlgorithm.

RRF (Reciprocal Rank Fusion)

RRF avoids score normalisation entirely by combining ranks instead of raw scores:

rrf_score(doc) = Σ 1 / (k + rank_i(doc))

The sum runs over each result list the document appears in. The k parameter (default 60.0) smooths the distribution — higher k flattens the contribution of top-ranked results.

#![allow(unused)]
fn main() {
use laurus::{FusionAlgorithm, SearchRequestBuilder};

let request = SearchRequestBuilder::new()
    .query_dsl("title:rust ~\"systems programming\"")
    .fusion_algorithm(FusionAlgorithm::Rrf { k: 60.0 })
    .build();
}

WeightedSum

WeightedSum first min-max normalises each list of scores independently, then takes a weighted linear combination:

norm(score)  = (score - min) / (max - min)
final(doc)   = lexical_weight * norm(lexical_score(doc))
             + vector_weight  * norm(vector_score(doc))

#![allow(unused)]
fn main() {
use laurus::{FusionAlgorithm, SearchRequestBuilder};

let request = SearchRequestBuilder::new()
    .query_dsl("title:rust ~\"systems programming\"")
    .fusion_algorithm(FusionAlgorithm::WeightedSum {
        lexical_weight: 0.6,
        vector_weight: 0.4,
    })
    .build();
}

Both weights are clamped to [0.0, 1.0]. Use RRF when you do not have a calibrated reason to pick specific weights — it is parameter-light and robust to scale differences between lists.

See Also

• API Reference → FusionAlgorithm — variant signatures
• Hybrid Search — when to pick which fusion algorithm
• Vector Search — distance metric trade-offs

Faceting

Faceting enables counting and categorizing search results by field values. It is commonly used to build navigation filters in search UIs (e.g., “Electronics (42)”, “Books (18)”).

Concepts

FacetPath

A FacetPath represents a hierarchical facet value. For example, a product category “Electronics > Computers > Laptops” is a facet path with three levels.

#![allow(unused)]
fn main() {
use laurus::lexical::search::features::facet::FacetPath;

// Single-level facet
let facet = FacetPath::from_value("category", "Electronics");

// Hierarchical facet from components
let facet = FacetPath::new("category", vec![
    "Electronics".to_string(),
    "Computers".to_string(),
    "Laptops".to_string(),
]);

// From a delimited string
let facet = FacetPath::from_delimited("category", "Electronics/Computers/Laptops", "/");
}

FacetPath Methods

FacetCount

FacetCount represents the result of a facet aggregation:

#![allow(unused)]
fn main() {
pub struct FacetCount {
    pub path: FacetPath,
    pub count: u64,
    pub children: Vec<FacetCount>,
}
}

Example: Hierarchical Facets

Category
├── Electronics (42)
│   ├── Computers (18)
│   │   ├── Laptops (12)
│   │   └── Desktops (6)
│   └── Phones (24)
└── Books (35)
    ├── Fiction (20)
    └── Non-Fiction (15)

Each node in this tree corresponds to a FacetCount with its children populated for drill-down navigation.

Use Cases

• E-commerce: Filter by category, brand, price range, rating
• Document search: Filter by author, department, date range, document type
• Content management: Filter by tags, topics, content status

Performance

Facet counts are read from each field’s DocValues column, not from the stored document. For every collected hit the collector reads only the facet field’s value via the per-field DocValues lookup, so it never decodes or clones the whole stored-fields blob when every faceted field has a DocValues column (which is the default — every stored field is written to DocValues at index time). A field that lacks DocValues transparently falls back to the stored document, so results are identical either way; only the read path changes.

Highlighting

Highlighting marks matching terms in search results, helping users see why a document matched their query. Laurus generates highlighted text fragments with configurable HTML tags.

HighlightConfig

HighlightConfig controls how highlights are generated:

#![allow(unused)]
fn main() {
use laurus::lexical::search::features::highlight::HighlightConfig;

let config = HighlightConfig::default()
    .tag("mark")
    .css_class("highlight")
    .max_fragments(3)
    .fragment_size(200);
}

Configuration Options

Builder Methods

HighlightFragment

Each highlight result is a HighlightFragment:

#![allow(unused)]
fn main() {
pub struct HighlightFragment {
    pub text: String,
}
}

The text field contains the fragment with matching terms wrapped in the configured HTML tags.

Output Example

Given a document with body = "Rust is a systems programming language focused on safety and performance." and a search for “rust programming”:

<mark>Rust</mark> is a systems <mark>programming</mark> language focused on safety and performance.

With css_class("highlight"):

<mark class="highlight">Rust</mark> is a systems <mark class="highlight">programming</mark> language focused on safety and performance.

Fragment Selection

When a field is long, Laurus selects the most relevant fragments:

1. The text is split into overlapping windows of fragment_size characters
2. Each fragment is scored by how many query terms it contains
3. The top max_fragments fragments are returned, joined by fragment_separator

If no fragments contain matches and return_entire_field_if_no_highlight is true, the full field value is returned instead.

Spelling Correction

Laurus includes a built-in spelling correction system that can suggest corrections for misspelled query terms and provide “Did you mean?” functionality.

Overview

The spelling corrector uses edit distance (Levenshtein distance) combined with word frequency data to suggest corrections. It supports:

• Word-level suggestions — correct individual misspelled words
• Auto-correction — automatically apply high-confidence corrections
• “Did you mean?” — suggest alternative queries to the user
• Query learning — improve suggestions by learning from user queries
• Custom dictionaries — use your own word lists

Basic Usage

SpellingCorrector

#![allow(unused)]
fn main() {
use laurus::spelling::corrector::SpellingCorrector;

// Create a corrector with the built-in English dictionary
let mut corrector = SpellingCorrector::new();

// Correct a query
let result = corrector.correct("programing langauge");

// Check if suggestions are available
if result.has_suggestions() {
    for (word, suggestions) in &result.word_suggestions {
        println!("'{}' -> {:?}", word, suggestions);
    }
}

// Get the best corrected query
if let Some(corrected) = result.query() {
    println!("Corrected: {}", corrected);
}
}

“Did You Mean?”

The DidYouMean wrapper provides a higher-level interface for search UIs:

#![allow(unused)]
fn main() {
use laurus::spelling::corrector::{SpellingCorrector, DidYouMean};

let corrector = SpellingCorrector::new();
let mut did_you_mean = DidYouMean::new(corrector);

if let Some(suggestion) = did_you_mean.suggest("programing") {
    println!("Did you mean: {}?", suggestion);
}
}

Configuration

Use CorrectorConfig to customize behavior:

#![allow(unused)]
fn main() {
use laurus::spelling::corrector::{CorrectorConfig, SpellingCorrector};

let config = CorrectorConfig {
    max_distance: 2,              // Maximum edit distance (default: 2)
    max_suggestions: 5,           // Max suggestions per word (default: 5)
    min_frequency: 1,             // Minimum word frequency threshold (default: 1)
    auto_correct: false,          // Enable auto-correction (default: false)
    auto_correct_threshold: 0.8,  // Confidence threshold for auto-correction (default: 0.8)
    use_index_terms: true,        // Use indexed terms as dictionary (default: true)
    learn_from_queries: true,     // Learn from user queries (default: true)
};
}

Configuration Options

CorrectionResult

The correct() method returns a CorrectionResult with detailed information:

Helper Methods

Custom Dictionaries

You can provide your own dictionary instead of using the built-in English one:

#![allow(unused)]
fn main() {
use laurus::spelling::corrector::SpellingCorrector;
use laurus::spelling::dictionary::SpellingDictionary;

// Build a custom dictionary
let mut dictionary = SpellingDictionary::new();
dictionary.add_word("elasticsearch", 100);
dictionary.add_word("lucene", 80);
dictionary.add_word("laurus", 90);

let corrector = SpellingCorrector::with_dictionary(dictionary);
}

Learning from Index Terms

When use_index_terms is enabled, the corrector can learn from terms in your search index:

#![allow(unused)]
fn main() {
let mut corrector = SpellingCorrector::new();

// Feed index terms to the corrector
let index_terms = vec!["rust", "programming", "search", "engine"];
corrector.learn_from_terms(&index_terms);
}

This improves suggestion quality by incorporating domain-specific vocabulary.

Statistics

Monitor the corrector’s state with stats():

#![allow(unused)]
fn main() {
let stats = corrector.stats();
println!("Dictionary words: {}", stats.dictionary_words);
println!("Total frequency: {}", stats.dictionary_total_frequency);
println!("Learned queries: {}", stats.queries_learned);
}

Next Steps

• Lexical Search – full-text search with query types
• Query DSL – human-readable query syntax

ID Management

Laurus uses a dual-tiered ID management strategy to ensure efficient document retrieval, updates, and aggregation in distributed environments.

1. External ID (String)

The External ID is a logical identifier used by users and applications to uniquely identify a document.

• Type: String
• Role: You can use any unique value, such as UUIDs, URLs, or database primary keys.
• Storage: Persisted transparently as a reserved system field name _id within the Lexical Index.
• Uniqueness: Expected to be unique across the entire system.
• Updates: Indexing a document with an existing external_id triggers an automatic “Delete-then-Insert” (Upsert) operation, replacing the old version with the newest.

2. Internal ID (u64 / Stable ID)

The Internal ID is a physical handle used internally by Laurus’s engines (Lexical and Vector) for high-performance operations.

• Type: Unsigned 64-bit Integer (u64)
• Role: Used for bitmap operations, point references, and routing between distributed nodes.
• Immutability (Stable): Once assigned, an Internal ID never changes due to index merges (segment compaction) or restarts. This prevents inconsistencies in deletion logs and caches.

ID Structure (Shard-Prefixed)

Laurus employs a Shard-Prefixed Stable ID scheme designed for multi-node distributed environments.

Why this structure?

1. Zero-Cost Aggregation: Since u64 IDs are globally unique, the aggregator can perform fast sorting and deduplication without worrying about ID collisions between nodes.
2. Fast Routing: The aggregator can immediately identify the physical node responsible for a document just by looking at the upper bits, avoiding expensive hash lookups.
3. High-Performance Fetching: Internal IDs map directly to physical data structures. This allows Laurus to skip the “External-to-Internal ID” conversion step during retrieval, achieving O(1) access speed.

ID Lifecycle

1. Registration (engine.put_document() / engine.add_document()): User provides a document with an External ID.
2. ID Assignment: The Engine combines the current shard_id with a new Local ID to issue a Shard-Prefixed Internal ID.
3. Mapping: The engine maintains the relationship between the External ID and the new Internal ID.
4. Search: Search results return the External ID (String), resolved from the Internal ID.
5. Retrieval/Deletion: While the user-facing API accepts External IDs for convenience, the engine internally converts them to Internal IDs for near-instant processing.

Persistence & WAL

Laurus uses a Write-Ahead Log (WAL) to ensure data durability. Every write operation is persisted to the WAL before modifying in-memory structures, guaranteeing that no data is lost even if the process crashes.

Write Path

sequenceDiagram
    participant App as Application
    participant Engine
    participant WAL as DocumentLog (WAL)
    participant Mem as In-Memory Buffers
    participant Disk as Storage (segments)

    App->>Engine: add_document() / delete_documents()
    Engine->>WAL: 1. Append operation to WAL
    Engine->>Mem: 2. Update in-memory buffers

    Note over Mem: Document is buffered but\nNOT yet searchable

    App->>Engine: commit()
    Engine->>Disk: 3. Flush segments to storage
    Engine->>WAL: 4. Truncate WAL
    Note over Disk: Documents are now\nsearchable and durable

Key Principles

1. WAL-first: Every write (add or delete) is appended to the WAL before updating in-memory structures
2. Buffered writes: In-memory buffers accumulate changes until commit() is called
3. Atomic commit: commit() flushes all buffered changes to segment files and truncates the WAL
4. Crash safety: If the process crashes between writes and commit, the WAL is replayed on the next startup

Write-Ahead Log (WAL)

The WAL is managed by the DocumentLog component and stored at the root level of the storage backend (engine.wal).

WAL Entry Types

WAL File

The WAL file (engine.wal) is an append-only binary log. Each entry is self-contained with:

• Operation type (add/delete)
• Sequence number
• Payload (document data or ID)

Recovery

When an engine is built (Engine::builder(...).build().await), it automatically checks for remaining WAL entries and replays them (the WAL is truncated on commit, so any remaining entries are from a crashed session):

graph TD
    Start["Engine::build()"] --> Check["Check WAL for\nuncommitted entries"]
    Check -->|"Entries found"| Replay["Replay operations\ninto in-memory buffers"]
    Replay --> Ready["Engine ready"]
    Check -->|"No entries"| Ready

Recovery is transparent — you do not need to handle it manually.

The Commit Lifecycle

#![allow(unused)]
fn main() {
// 1. Add documents (buffered, not yet searchable)
engine.add_document("doc-1", doc1).await?;
engine.add_document("doc-2", doc2).await?;

// 2. Commit — flush to persistent storage
engine.commit().await?;
// Documents are now searchable

// 3. Add more documents
engine.add_document("doc-3", doc3).await?;

// 4. If the process crashes here, doc-3 is in the WAL
//    and will be recovered on next startup
}

When to Commit

Tip: Commits are relatively expensive because they flush segments to storage. For bulk indexing, batch many documents before calling commit().

Storage Layout

The engine uses PrefixedStorage to organize data:

<storage root>/
├── lexical/          # Inverted index segments
│   ├── seg-000/
│   │   ├── terms.dict
│   │   ├── postings.post
│   │   └── ...
│   └── metadata.json
├── vector/           # Vector index segments
│   ├── seg-000/
│   │   ├── graph.hnsw
│   │   ├── vectors.vecs
│   │   └── ...
│   └── metadata.json
├── documents/        # Document storage
│   └── ...
└── engine.wal        # Write-ahead log

Next Steps

• How deletions are handled: Deletions & Compaction
• Storage backends: Storage

Deletions & Compaction

Laurus uses a two-phase deletion strategy: fast logical deletion followed by periodic physical compaction.

Deleting Documents

#![allow(unused)]
fn main() {
// Delete a document by its external ID
engine.delete_documents("doc-1").await?;
engine.commit().await?;
}

Logical Deletion

When a document is deleted, it is not immediately removed from the index files. Instead:

graph LR
    Del["delete_documents('doc-1')"] --> Bitmap["Add internal ID\nto Deletion Bitmap"]
    Bitmap --> Search["Search skips\ndeleted IDs"]

1. The document’s internal ID is added to a deletion bitmap
2. The bitmap is checked during every search, filtering out deleted documents from results
3. The original data remains in the segment files

Why Logical Deletion?

Upserts (Update = Delete + Insert)

When you index a document with an existing external ID, Laurus performs an automatic upsert:

1. The old document is logically deleted (its ID is added to the deletion bitmap)
2. A new document is inserted with a new internal ID
3. The external-to-internal ID mapping is updated

#![allow(unused)]
fn main() {
// First insert
engine.put_document("doc-1", doc_v1).await?;
engine.commit().await?;

// Update: old version is logically deleted, new version is inserted
engine.put_document("doc-1", doc_v2).await?;
engine.commit().await?;
}

Physical Compaction

Over time, logically deleted documents accumulate and waste space. Compaction reclaims this space by rewriting segment files without the deleted entries.

graph LR
    subgraph "Before Compaction"
        S1["Segment 0\ndoc-1 (deleted)\ndoc-2\ndoc-3 (deleted)"]
        S2["Segment 1\ndoc-4\ndoc-5"]
    end

    Compact["Compaction"]

    subgraph "After Compaction"
        S3["Segment 0\ndoc-2\ndoc-4\ndoc-5"]
    end

    S1 --> Compact
    S2 --> Compact
    Compact --> S3

What Compaction Does

1. Reads all live (non-deleted) documents from existing segments
2. Rebuilds the inverted index and/or vector index without deleted entries
3. Writes new, clean segment files
4. Removes the old segment files
5. Resets the deletion bitmap

Cost and Frequency

When to Compact

• Low-write workloads: Compact periodically (e.g., daily or weekly)
• High-write workloads: Compact when the deletion ratio exceeds a threshold
• After bulk updates: Compact after a large batch of upserts

Deletion Bitmap

The deletion bitmap tracks which internal IDs have been deleted:

• Storage: a Roaring bitmap of deleted document IDs. For the dense deletion sets that accumulate over a segment’s life this is dramatically smaller than a plain ID list — e.g. a 10M-doc segment at 10% deletion is ~125 KB on disk instead of ~8 MB.
• Lookup: a branch-light bit test, which stays CPU-cache-resident even for large deletion sets — is_deleted is on the per-document (lexical) and per-neighbour (vector) search hot paths.

The bitmap is persisted alongside the index segments (the .delmap file) and is rebuilt from the WAL during recovery. The on-disk format is versioned: the current writer emits v4 (Roaring), and the reader still loads the older v1–v3 (raw ID list) layouts for backward compatibility.

Next Steps

• How data is persisted: Persistence & WAL
• ID management and internal/external ID mapping: ID Management

Error Handling

Laurus uses a unified error type for all operations. Understanding the error system helps you write robust applications that handle failures gracefully.

LaurusError

All Laurus operations return Result<T>, which is an alias for std::result::Result<T, LaurusError>.

LaurusError is an enum with variants for each category of failure:

Basic Error Handling

Using the ? Operator

The simplest approach — propagate errors to the caller:

#![allow(unused)]
fn main() {
use laurus::{Engine, Result};

async fn index_documents(engine: &Engine) -> Result<()> {
    let doc = laurus::Document::builder()
        .add_text("title", "Rust Programming")
        .build();

    engine.put_document("doc1", doc).await?;
    engine.commit().await?;
    Ok(())
}
}

Matching on Error Variants

When you need different behavior for different error types:

#![allow(unused)]
fn main() {
use laurus::{Engine, LaurusError};

async fn safe_search(engine: &Engine, query: &str) {
    match engine.search(/* request */).await {
        Ok(results) => {
            for result in results {
                println!("{}: {}", result.id, result.score);
            }
        }
        Err(LaurusError::Query(msg)) => {
            eprintln!("Invalid query syntax: {}", msg);
        }
        Err(LaurusError::Io(e)) => {
            eprintln!("Storage I/O error: {}", e);
        }
        Err(e) => {
            eprintln!("Unexpected error: {}", e);
        }
    }
}
}

Checking Error Types with downcast

Since LaurusError implements std::error::Error, you can use standard error handling patterns:

#![allow(unused)]
fn main() {
use laurus::LaurusError;

fn is_retriable(error: &LaurusError) -> bool {
    matches!(error, LaurusError::Io(_) | LaurusError::ResourceExhausted(_))
}
}

Common Error Scenarios

Schema Mismatch

Adding a document with fields that don’t match the schema:

#![allow(unused)]
fn main() {
// Schema has "title" (Text) and "year" (Integer)
let doc = Document::builder()
    .add_text("title", "Hello")
    .add_text("unknown_field", "this field is not in schema")
    .build();

// Fields not in the schema are silently ignored during indexing.
// No error is raised — only schema-defined fields are processed.
}

Query Parsing Errors

Invalid Query DSL syntax returns a Query error:

#![allow(unused)]
fn main() {
use laurus::engine::query::UnifiedQueryParser;

let parser = UnifiedQueryParser::new();
match parser.parse("title:\"unclosed phrase") {
    Ok(request) => { /* ... */ }
    Err(LaurusError::Query(msg)) => {
        // msg contains details about the parse failure
        eprintln!("Bad query: {}", msg);
    }
    Err(e) => { /* other errors */ }
}
}

Storage I/O Errors

File-based storage may encounter I/O errors:

#![allow(unused)]
fn main() {
use laurus::storage::{StorageConfig, StorageFactory};

match StorageFactory::open(StorageConfig::File {
    path: "/nonexistent/path".into(),
    loading_mode: Default::default(),
}) {
    Ok(storage) => { /* ... */ }
    Err(LaurusError::Io(e)) => {
        eprintln!("Cannot open storage: {}", e);
    }
    Err(e) => { /* other errors */ }
}
}

Convenience Constructors

LaurusError provides factory methods for creating errors in custom implementations:

These are useful when implementing custom Analyzer, Embedder, or Storage traits:

#![allow(unused)]
fn main() {
use laurus::{LaurusError, Result};

fn validate_dimension(dim: usize) -> Result<()> {
    if dim == 0 {
        return Err(LaurusError::invalid_argument("dimension must be > 0"));
    }
    Ok(())
}
}

Automatic Conversions

LaurusError implements From for common error types, so they convert automatically with ?:

Next Steps

• Extensibility — implement custom traits with proper error handling
• API Reference — full method signatures and return types

Extensibility

Laurus uses trait-based abstractions for its core components. You can implement these traits to provide custom analyzers, embedders, and storage backends.

Custom Analyzer

Implement the Analyzer trait to create a custom text analysis pipeline:

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::analyzer::Analyzer;
use laurus::analysis::token::{Token, TokenStream};
use laurus::Result;

#[derive(Debug)]
struct ReverseAnalyzer;

impl Analyzer for ReverseAnalyzer {
    fn analyze(&self, text: &str) -> Result<TokenStream> {
        let tokens: Vec<Token> = text
            .split_whitespace()
            .enumerate()
            .map(|(i, word)| Token {
                text: word.chars().rev().collect(),
                position: i,
                ..Default::default()
            })
            .collect();
        Ok(Box::new(tokens.into_iter()))
    }

    fn name(&self) -> &str {
        "reverse"
    }

    fn as_any(&self) -> &dyn std::any::Any {
        self
    }
}
}

Required Methods

Using a Custom Analyzer

Pass your analyzer to EngineBuilder:

#![allow(unused)]
fn main() {
use std::sync::Arc;

let analyzer = Arc::new(ReverseAnalyzer);
let engine = Engine::builder(storage, schema)
    .analyzer(analyzer)
    .build()
    .await?;
}

For per-field analyzers, wrap with PerFieldAnalyzer:

#![allow(unused)]
fn main() {
use laurus::analysis::analyzer::per_field::PerFieldAnalyzer;
use laurus::analysis::analyzer::standard::StandardAnalyzer;

let per_field = PerFieldAnalyzer::new(Arc::new(StandardAnalyzer::new()?));
per_field.add_analyzer("custom_field", Arc::new(ReverseAnalyzer));

let engine = Engine::builder(storage, schema)
    .analyzer(Arc::new(per_field))
    .build()
    .await?;
}

Custom Embedder

Implement the Embedder trait to integrate your own vector embedding model:

#![allow(unused)]
fn main() {
use async_trait::async_trait;
use laurus::embedding::embedder::{Embedder, EmbedInput, EmbedInputType};
use laurus::vector::core::vector::Vector;
use laurus::{LaurusError, Result};

#[derive(Debug)]
struct MyEmbedder {
    dimension: usize,
}

#[async_trait]
impl Embedder for MyEmbedder {
    async fn embed(&self, input: &EmbedInput<'_>) -> Result<Vector> {
        match input {
            EmbedInput::Text(text) => {
                // Your embedding logic here
                let vector = vec![0.0f32; self.dimension];
                Ok(Vector::new(vector))
            }
            _ => Err(LaurusError::invalid_argument(
                "this embedder only supports text input",
            )),
        }
    }

    fn supported_input_types(&self) -> Vec<EmbedInputType> {
        vec![EmbedInputType::Text]
    }

    fn name(&self) -> &str {
        "my-embedder"
    }

    fn as_any(&self) -> &dyn std::any::Any {
        self
    }
}
}

Required Methods

Optional Methods

Using a Custom Embedder

#![allow(unused)]
fn main() {
let embedder = Arc::new(MyEmbedder { dimension: 384 });
let engine = Engine::builder(storage, schema)
    .embedder(embedder)
    .build()
    .await?;
}

For per-field embedders, wrap with PerFieldEmbedder:

#![allow(unused)]
fn main() {
use laurus::embedding::per_field::PerFieldEmbedder;

let per_field = PerFieldEmbedder::new(Arc::new(MyEmbedder { dimension: 384 }));
per_field.add_embedder("image_vec", Arc::new(ClipEmbedder::new()?));

let engine = Engine::builder(storage, schema)
    .embedder(Arc::new(per_field))
    .build()
    .await?;
}

Custom Storage

Implement the Storage trait to add a new storage backend:

#![allow(unused)]
fn main() {
use laurus::storage::{Storage, StorageInput, StorageOutput, LoadingMode, FileMetadata};
use laurus::Result;

#[derive(Debug)]
struct S3Storage {
    bucket: String,
    prefix: String,
}

impl Storage for S3Storage {
    fn loading_mode(&self) -> LoadingMode {
        LoadingMode::Eager  // S3 requires full download
    }

    fn open_input(&self, name: &str) -> Result<Box<dyn StorageInput>> {
        // Download from S3 and return a reader
        todo!()
    }

    fn create_output(&self, name: &str) -> Result<Box<dyn StorageOutput>> {
        // Create an upload stream to S3
        todo!()
    }

    fn create_output_append(&self, name: &str) -> Result<Box<dyn StorageOutput>> {
        todo!()
    }

    fn file_exists(&self, name: &str) -> bool {
        todo!()
    }

    fn delete_file(&self, name: &str) -> Result<()> {
        todo!()
    }

    fn list_files(&self) -> Result<Vec<String>> {
        todo!()
    }

    fn file_size(&self, name: &str) -> Result<u64> {
        todo!()
    }

    fn metadata(&self, name: &str) -> Result<FileMetadata> {
        todo!()
    }

    fn rename_file(&self, old_name: &str, new_name: &str) -> Result<()> {
        todo!()
    }

    fn create_temp_output(&self, prefix: &str) -> Result<(String, Box<dyn StorageOutput>)> {
        todo!()
    }

    fn sync(&self) -> Result<()> {
        todo!()
    }

    fn close(&mut self) -> Result<()> {
        todo!()
    }
}
}

Required Methods

Optional Methods

Thread Safety

All three traits require Send + Sync. This means your implementations must be safe to share across threads. Use Arc<Mutex<_>> or lock-free data structures for shared mutable state.

Next Steps

• Error Handling — handle errors in custom implementations
• Text Analysis — built-in analyzers and pipeline components
• Embeddings — built-in embedder options
• Storage — built-in storage backends

API Reference

This page provides a quick reference of the most important types and methods in Laurus. For full details, generate the Rustdoc:

cargo doc --open

Engine

The central coordinator for all indexing and search operations.

put_document vs add_document: put_document performs an upsert — if a document with the same external ID already exists, it is deleted and replaced. add_document always appends, allowing multiple document chunks to share the same external ID. See Schema & Fields — Indexing Documents for details.

EngineBuilder

Schema

Defines document structure.

SchemaBuilder

Document

A collection of named field values.

DocumentBuilder

Search

SearchRequestBuilder

LexicalSearchQuery

VectorSearchQuery

SearchResult

FusionAlgorithm

Query Types (Lexical)

Query Parsers

Analyzers

Embedders

Storage

DataValue

CLI Overview

Laurus provides a command-line tool laurus that lets you create indexes, manage documents, and run search queries without writing code.

Features

• Index management – Create and inspect indexes from TOML schema files, with an interactive schema generator
• Document CRUD – Add, retrieve, and delete documents via JSON
• Search – Execute queries using the Query DSL
• Dual output – Human-readable tables or machine-parseable JSON
• Interactive REPL – Explore your index in a live session
• gRPC server – Start a gRPC server with laurus serve

Getting Started

# Install
cargo install laurus-cli

# Generate a schema interactively
laurus create schema

# Create an index from the schema
laurus --index-dir ./my_index create index --schema schema.toml

# Add a document
laurus --index-dir ./my_index add doc --id doc1 --data '{"title":"Hello","body":"World"}'

# Commit changes
laurus --index-dir ./my_index commit

# Search
laurus --index-dir ./my_index search "body:world"

See the sub-sections for detailed documentation:

• Installation – How to install the CLI
• Commands – Full command reference
• Schema Format – Schema TOML format reference
• REPL – Interactive mode

Installation

From crates.io

cargo install laurus-cli

This installs the laurus binary to ~/.cargo/bin/.

From source

git clone https://github.com/mosuka/laurus.git
cd laurus
cargo install --path laurus-cli

Verify

laurus --version

Hands-on Tutorial

This tutorial walks you through a complete workflow using the laurus CLI: creating a schema, building an index, adding documents, searching, updating, deleting, and using the interactive REPL.

Prerequisites

• laurus CLI installed (see Installation)

Step 1: Create a Schema

First, create a schema file that defines your index structure. You can generate one interactively:

laurus create schema

The interactive wizard guides you through defining fields, their types, and options. For this tutorial, create a schema file manually instead:

cat > schema.toml << 'EOF'
default_fields = ["title", "body"]

[fields.title.Text]
indexed = true
stored = true
term_vectors = false

[fields.body.Text]
indexed = true
stored = true
term_vectors = false

[fields.category.Text]
indexed = true
stored = true
term_vectors = false
EOF

This defines three text fields. The default_fields setting means queries without a field prefix will search title and body.

Step 2: Create an Index

Create an index using the schema:

laurus --index-dir ./tutorial_data create index --schema schema.toml

Verify the index was created:

laurus --index-dir ./tutorial_data get stats

The output shows the document count is 0.

Step 3: Add Documents

Add documents to the index. Each document needs an ID and a JSON object with field values:

laurus --index-dir ./tutorial_data add doc \
  --id doc001 \
  --data '{"title":"Introduction to Rust Programming","body":"Rust is a modern systems programming language that focuses on safety, speed, and concurrency.","category":"programming"}'

laurus --index-dir ./tutorial_data add doc \
  --id doc002 \
  --data '{"title":"Web Development with Rust","body":"Building web applications with Rust has become increasingly popular. Frameworks like Actix and Rocket make it easy to create fast and secure web services.","category":"web-development"}'

laurus --index-dir ./tutorial_data add doc \
  --id doc003 \
  --data '{"title":"Python for Data Science","body":"Python is the most popular language for data science and machine learning. Libraries like NumPy and Pandas provide powerful tools for data analysis.","category":"data-science"}'

Step 4: Commit Changes

Documents are not searchable until committed:

laurus --index-dir ./tutorial_data commit

Step 5: Search Documents

Basic Search

Search for documents containing “rust”:

laurus --index-dir ./tutorial_data search "rust"

This searches the default fields (title and body). Results show doc001 and doc002.

Field-Specific Search

Search only in the title field:

laurus --index-dir ./tutorial_data search "title:python"

Only doc003 is returned.

Category Search

laurus --index-dir ./tutorial_data search "category:programming"

Only doc001 is returned.

Boolean Queries

Combine conditions with + (must) and - (must not):

laurus --index-dir ./tutorial_data search "+body:rust -body:web"

Only doc001 is returned (contains “rust” but not “web”).

Phrase Search

Search for an exact phrase:

laurus --index-dir ./tutorial_data search 'body:"data science"'

Only doc003 is returned.

Fuzzy Search

Search with typo tolerance using ~:

laurus --index-dir ./tutorial_data search "body:programing~1"

Matches “programming” despite the typo.

JSON Output

Get results in JSON format for programmatic use:

laurus --index-dir ./tutorial_data --format json search "rust"

Step 6: Retrieve a Document

Fetch a specific document by ID:

laurus --index-dir ./tutorial_data get docs --id doc001

Step 7: Delete a Document

Delete a document and commit the change:

laurus --index-dir ./tutorial_data delete docs --id doc003
laurus --index-dir ./tutorial_data commit

Verify it was deleted:

laurus --index-dir ./tutorial_data search "python"

No results are returned.

Step 8: Use the REPL

The REPL provides an interactive session for exploring your index:

laurus --index-dir ./tutorial_data repl

Try these commands in the REPL:

> get stats
> search rust
> add doc doc004 {"title":"Go Programming","body":"Go is a statically typed language designed for simplicity and efficiency.","category":"programming"}
> commit
> search programming
> get docs doc004
> delete docs doc004
> commit
> quit

The REPL supports command history (Up/Down arrows) and line editing.

Step 9: Clean Up

Remove the tutorial data:

rm -rf ./tutorial_data schema.toml

Next Steps

• Learn about the Schema Format for advanced field configurations
• See the full Commands reference
• Explore the REPL for interactive usage
• Try the server tutorial for gRPC/HTTP access

Command Reference

Global Options

Every command accepts these options:

# Example: use JSON output with a custom data directory
laurus --index-dir /var/data/my_index --format json search "title:rust"

create — Create a Resource

create index

Create a new index. If --schema is given, uses that TOML file; otherwise launches the interactive schema wizard.

laurus create index [--schema <FILE>]

Arguments:

Schema file format:

The schema file follows the same structure as the Schema type in the Laurus library. See Schema Format Reference for full details. Example:

default_fields = ["title", "body"]

[fields.title.Text]
stored = true
indexed = true

[fields.body.Text]
stored = true
indexed = true

[fields.category.Text]
stored = true
indexed = true

Examples:

# From a schema file
laurus --index-dir ./my_index create index --schema schema.toml
# Index created at ./my_index.

# Interactive wizard (no --schema flag)
laurus --index-dir ./my_index create index
# === Laurus Schema Generator ===
# Field name: title
# ...
# Index created at ./my_index.

Note: If both schema.toml and store/ already exist, an error is returned. Delete the index directory to recreate. If only schema.toml exists (e.g. after an interrupted creation), running create index without --schema recovers the index by creating the missing storage from the existing schema.

create schema

Interactively generate a schema TOML file through a guided wizard.

laurus create schema [--output <FILE>]

Arguments:

The wizard guides you through:

1. Field definition — Enter a field name, select the type, and configure type-specific options
2. Repeat — Add as many fields as needed
3. Default fields — Select which lexical fields to use as default search fields
4. Preview — Review the generated TOML before saving
5. Save — Write the schema file

Supported field types:

Example:

# Generate schema.toml interactively
laurus create schema

# Specify output path
laurus create schema --output my_schema.toml

# Then create an index from the generated schema
laurus create index --schema schema.toml

get — Get a Resource

get stats

Display statistics about the index.

laurus get stats

Table output example:

Document count: 42

Vector fields:
╭──────────┬─────────┬───────────╮
│ Field    │ Vectors │ Dimension │
├──────────┼─────────┼───────────┤
│ text_vec │ 42      │ 384       │
╰──────────┴─────────┴───────────╯

JSON output example:

laurus --format json get stats

{
  "document_count": 42,
  "fields": {
    "text_vec": {
      "vector_count": 42,
      "dimension": 384
    }
  }
}

get schema

Display the current index schema as JSON.

laurus get schema

Example:

laurus get schema
# {
#   "fields": { ... },
#   "default_fields": ["title", "body"],
#   ...
# }

get docs

Retrieve all documents (including chunks) by external ID.

laurus get docs --id <ID>

Table output example:

╭──────┬─────────────────────────────────────────╮
│ ID   │ Fields                                  │
├──────┼─────────────────────────────────────────┤
│ doc1 │ body: This is a test, title: Hello World │
╰──────┴─────────────────────────────────────────╯

JSON output example:

laurus --format json get docs --id doc1

[
  {
    "id": "doc1",
    "document": {
      "title": "Hello World",
      "body": "This is a test document."
    }
  }
]

add — Add a Resource

add doc

Add a document to the index. Documents are not searchable until commit is called.

laurus add doc --id <ID> --data <JSON>

Arguments:

The JSON format is a flat object mapping field names to values:

{
  "title": "Introduction to Rust",
  "body": "Rust is a systems programming language.",
  "category": "programming"
}

Example:

laurus add doc --id doc1 --data '{"title":"Hello World","body":"This is a test document."}'
# Document 'doc1' added. Run 'commit' to persist changes.

Tip: Multiple documents can share the same external ID (chunking pattern). Use add doc for each chunk.

put — Put (Upsert) a Resource

put doc

Put (upsert) a document into the index. If a document with the same ID already exists, all its chunks are deleted before the new document is indexed. Documents are not searchable until commit is called.

laurus put doc --id <ID> --data <JSON>

Arguments:

Example:

laurus put doc --id doc1 --data '{"title":"Updated Title","body":"This replaces the existing document."}'
# Document 'doc1' put (upserted). Run 'commit' to persist changes.

Note: Unlike add doc, put doc replaces all existing chunks for the given ID. Use add doc when you want to append chunks, and put doc when you want to replace the entire document.

add field

Dynamically add a new field to an existing index.

laurus add field --index-dir ./data \
    --name category \
    --field-option '{"Text": {"indexed": true, "stored": true}}'

The --field-option argument accepts a JSON string using the same externally-tagged format as the schema file. The schema is automatically persisted after the field is added.

delete — Delete a Resource

delete docs

Delete all documents (including chunks) by external ID.

laurus delete docs --id <ID>

Example: