Schema & Fields

The Schema defines the structure of your documents — what fields exist and how each field is indexed. It is the single source of truth for the Engine.

For the TOML file format used by the CLI, see Schema Format Reference.

Schema

A Schema is a collection of named fields. Each field is either a lexical field (for keyword search) or a vector field (for similarity search).

#![allow(unused)]
fn main() {
use laurus::Schema;
use laurus::lexical::TextOption;
use laurus::lexical::core::field::IntegerOption;
use laurus::vector::HnswOption;

let schema = Schema::builder()
    .add_text_field("title", TextOption::default())
    .add_text_field("body", TextOption::default())
    .add_integer_field("year", IntegerOption::default())
    .add_hnsw_field("embedding", HnswOption::default())
    .add_default_field("body")
    .build();
}

Default Fields

add_default_field() specifies which field(s) are searched when a query does not explicitly name a field. This is used by the Query DSL parser.

Field Types

graph TB
    FO["FieldOption"]

    FO --> T["Text"]
    FO --> I["Integer"]
    FO --> FL["Float"]
    FO --> B["Boolean"]
    FO --> DT["DateTime"]
    FO --> G["Geo"]
    FO --> G3["Geo3d"]
    FO --> BY["Bytes"]

    FO --> FLAT["Flat"]
    FO --> HNSW["HNSW"]
    FO --> IVF["IVF"]

Lexical Fields

Lexical fields are indexed using an inverted index and support keyword-based queries.

Text Field Options

TextOption controls how text is indexed:

#![allow(unused)]
fn main() {
use laurus::lexical::TextOption;

// Default: indexed + stored + term vectors + doc values (all true)
let opt = TextOption::default();

// Customize
let opt = TextOption::default()
    .indexed(true)
    .stored(true)
    .multi_valued(false)
    .position_increment_gap(100)
    .term_vectors(true)
    .doc_values(true);
}

doc_values is not unique to TextOption — every lexical field option except BytesOption carries it (IntegerOption, FloatOption, BooleanOption, DateTimeOption, GeoOption, Geo3dOption). BytesOption has no such setting: a Bytes value is never written to DocValues regardless, since neither sorting nor faceting can do anything with it. The effective rule is: a DocValues column is written only when stored and doc_values are both true (and the value’s type isn’t Bytes); setting doc_values: false on a field that is never sorted or faceted on shrinks its segment footprint by skipping the second copy.

Vector Fields

Vector fields are indexed using vector indexes for approximate nearest neighbor (ANN) search.

HNSW Field Options (most common)

#![allow(unused)]
fn main() {
use laurus::vector::HnswOption;
use laurus::vector::core::distance::DistanceMetric;

use laurus::vector::core::quantization::QuantizationMethod;

let opt = HnswOption {
    dimension: 384,                                  // vector dimensions
    distance: DistanceMetric::Cosine,                // distance metric
    m: 16,                                           // max connections per layer
    ef_construction: 200,                            // construction search width
    default_ef_search: Some(100),                    // schema-level ef_search default (issue #644)
    base_weight: 1.0,                                // relative priority vs. other vector fields (issue #1084)
    quantizer: QuantizationMethod::Scalar8Bit,       // mandatory; default Scalar8Bit
    embedder: None,                                  // optional named embedder
};
}

default_ef_search: the search-time recall knob

ef_search controls the dynamic candidate list size during query time (distinct from ef_construction, which only affects index build). Higher values explore more graph neighbours and yield higher recall at the cost of latency.

• Schema-level default: set HnswOption.default_ef_search = Some(ef) to raise the per-field default. When None, the searcher falls back to its built-in 50.
• Per-query override: search requests honour SearchRequestBuilder::vector_ef_search. The per-query value takes precedence over the schema default.
• Auto-lifting: regardless of which source provides ef_search, the searcher lifts the effective value to at least top_k (and top_k * rerank_factor when both are set) so the candidate heap is never undersized for the requested top_k.
• Tracked under Issue #644.

See Vector Indexing for detailed parameter guidance.

Multi-Vector Fields

A MultiVector field holds all the token vectors of a document, for example the per-token embeddings of a ColBERT-style late-interaction model (typically 100–300 vectors of 128 dimensions per passage). It has no ANN index: it is never a vector-search target, and its vectors are read only to rescore candidates that another query (lexical, vector, or hybrid) already found.

#![allow(unused)]
fn main() {
use laurus::{MultiVectorOption, Schema};
use laurus::vector::core::distance::DistanceMetric;

let schema = Schema::builder()
    .add_multi_vector_field(
        "body_colbert",
        MultiVectorOption::new(128).distance(DistanceMetric::Cosine), // Cosine (default) or DotProduct
    )
    .build();
}

• Options: dimension (every token vector’s length), distance, storage, pool_factor and embedder. Only Cosine (vectors are L2-normalized when written, so the similarity is their dot product; a zero vector is kept as-is) and DotProduct are accepted for distance; other metrics are rejected when the index is created. storage picks the on-disk element kind of the token vectors — F32 (default, exact), F16, Int8, or Binary (1 bit per element; requires Cosine) — trading some precision for a smaller index; see MultiVector Storage for the size and quality figures. pool_factor (default 1, no pooling) keeps about 1 / pool_factor of each document’s token vectors by averaging similar ones at index time; see Token pooling. embedder names a token-level embedder, such as a candle_colbert one (MultiVectorOption::new(96).embedder("colbert")).
• Values: a DataValue::VectorArray(Vec<Vec<f32>>) holding at least one and at most 8,192 vectors, each of the field’s dimension and with finite values. In JSON it is an array of equal-length numeric arrays ([[0.1, 0.2, ...], [0.3, 0.4, ...]]). When a token-level embedder serves the field (named in embedder, or registered through EngineBuilder::embedder), a text value works too: it is embedded into token vectors when the document is indexed (Issue #1349), and a document is rejected if the field has no such embedder.
• Not stored: the token vectors are kept only in the vector index (about 150 KB per document for 300 × 128 f32), so get_documents and search results never include the field.
• Not queryable: a field-less vector query and a prefix field selector skip it, naming it in a vector query is an error, and so is naming it in a DSL query.
• Schema changes: changing its dimension, distance or embedder, or changing it to or from another vector field type, is a destructive change. Changing storage is a Reindex change instead — the stored values are decoded and re-encoded, so Engine::update_field(.., UpdateFieldOptions { reindex: true, .. }) rebuilds the field’s existing segments under the new on-disk kind (precision a lossy kind already dropped does not come back). Turning pooling on (pool_factor from 1 to more) is a Reindex change too: the stored token vectors are pooled. Changing or turning off a pool_factor already in use is destructive, because pooled token vectors cannot be restored. An option the field’s index would reject, such as Binary with DotProduct or a pool_factor of 0, is rejected before any data is touched.

Document

A Document is a collection of named field values. Use DocumentBuilder to construct documents:

#![allow(unused)]
fn main() {
use laurus::Document;

let doc = Document::builder()
    .add_text("title", "Introduction to Rust")
    .add_text("body", "Rust is a systems programming language.")
    .add_integer("year", 2024)
    .add_float("rating", 4.8)
    .add_boolean("published", true)
    .build();
}

Indexing Documents

The Engine provides two methods for adding documents, each with different semantics:

#![allow(unused)]
fn main() {
// Upsert: replaces any existing document with id "doc1"
engine.put_document("doc1", doc).await?;

// Append: adds another chunk under the same id "doc1"
engine.add_document("doc1", chunk2).await?;

// Always commit after indexing
engine.commit().await?;
}

Retrieving Documents

Use get_documents to retrieve all documents (including chunks) by external ID:

#![allow(unused)]
fn main() {
let docs = engine.get_documents("doc1").await?;
for doc in &docs {
    if let Some(title) = doc.get("title") {
        println!("Title: {:?}", title);
    }
}
}

Deleting Documents

Delete all documents and chunks sharing an external ID:

#![allow(unused)]
fn main() {
engine.delete_documents("doc1").await?;
engine.commit().await?;
}

Document Lifecycle

graph LR
    A["Build Document"] --> B["put/add_document()"]
    B --> C["WAL"]
    C --> D["commit()"]
    D --> E["Searchable"]
    E --> F["get_documents()"]
    E --> G["delete_documents()"]

Important: Documents are not searchable until commit() is called.

DocumentBuilder Methods

DataValue

DataValue is the unified value enum that represents any field value in Laurus:

#![allow(unused)]
fn main() {
pub enum DataValue {
    Null,
    Bool(bool),
    Int64(i64),
    Float64(f64),
    Text(String),
    Bytes(Vec<u8>, Option<String>),  // (data, optional MIME type)
    Vector(Vec<f32>),
    DateTime(DateTime<Utc>),
    Geo(GeoPoint),                   // 2D WGS84 point (latitude, longitude)
    GeoEcef(GeoEcefPoint),           // 3D ECEF Cartesian point (x, y, z) in metres
    Int64Array(Vec<i64>),            // multi-valued integer field
    Float64Array(Vec<f64>),          // multi-valued float field
    GeoArray(Vec<GeoPoint>),         // multi-valued 2D geo field
    GeoEcefArray(Vec<GeoEcefPoint>), // multi-valued 3D ECEF field
    DateTimeArray(Vec<DateTime<Utc>>), // multi-valued datetime field
    BoolArray(Vec<bool>),            // multi-valued boolean field
    TextArray(Vec<String>),          // multi-valued text field
    BytesArray(Vec<(Vec<u8>, Option<String>)>), // multi-valued bytes field, per-element MIME
    VectorArray(Vec<Vec<f32>>),      // token vectors of a MultiVector field
}
}

DataValue implements From<T> for common types, so you can use .into() conversions:

#![allow(unused)]
fn main() {
use laurus::DataValue;

let v: DataValue = "hello".into();       // Text
let v: DataValue = 42i64.into();         // Int64
let v: DataValue = 3.14f64.into();       // Float64
let v: DataValue = true.into();          // Bool
let v: DataValue = vec![0.1f32, 0.2].into(); // Vector
}

Reserved Fields

Any field name starting with an underscore (_) is reserved for the engine. Declaring such a field is rejected when the index is created, and documents that carry user-supplied _-prefixed keys are rejected at ingest time — so the field can never hold a value either way. The language bindings check even earlier: their field-adding methods (add_*_field / add*Field) reject such a name as soon as it is passed. See Field Naming for the exact error and how an index created before this check is handled.

The only _-prefixed name that is accepted is the allow-listed _id system field described below.

_id — external document identifier

Stores the external document ID supplied to put_document / add_document. It is injected automatically and indexed with KeywordAnalyzer (exact match). You do not need to add it to your schema.

Dynamic Schema

Laurus can accept documents even when some of their fields have not been declared in the schema. The behaviour is controlled by the DynamicFieldPolicy attached to the schema:

Set the policy on the builder:

#![allow(unused)]
fn main() {
use laurus::{DynamicFieldPolicy, Schema};

let schema = Schema::builder()
    .dynamic_field_policy(DynamicFieldPolicy::Dynamic)
    .build();
}

Type inference rules (Dynamic policy)

Vector fields (Hnsw, Flat, Ivf, MultiVector) are never inferred: they must be declared in the schema explicitly, since dimension, distance metric, and embedder configuration cannot be recovered from a value alone. An undeclared field carrying token vectors is rejected. Bytes values can be parsed from the {data, mime} object shape above, but an undeclared field carrying a Bytes value is still rejected rather than auto-registered — bytes fields must always be declared explicitly, same as vector fields. Mixing markers from more than one shape (2D lat/lon, 3D x/y/z, or bytes data) in a single object is rejected as ambiguous; use exactly one shape per object.

Multi-valued fields

Integer and Float fields can be declared with multi_valued = true to hold multiple values per document. A range query matches a document if any of its values satisfies the predicate (Lucene-style “any match” semantics with constant scoring — there is no per-match BM25 weighting).

Geo and Geo3d fields accept multi_valued = true as well. Every point of a multi-valued geo field becomes its own entry in the field’s BKD tree (2 dimensions for Geo, 3 for Geo3d), so distance and bounding-box queries — and, for Geo3d, nearest queries — match a document if any of its points satisfies the predicate. A document is reported once, scored by its closest matching point (for the 2D bounding-box query, the point closest to the box centre). Under the Dynamic policy an array that mixes 2D and 3D objects, or numbers and objects, is rejected; each element gets the same range validation as a single point.

DateTime fields accept multi_valued = true as well (Issue #1184). Every instant of a multi-valued datetime field becomes its own 1-dimensional entry in the field’s BKD tree, so DateTimeRangeQuery, a NumericRangeQuery on the field, and DSL date ranges such as seen_at:[2024-06-01 TO 2024-12-31] match a document if any of its instants is in range (Lucene-style “any match” with constant scoring — there is no per-match BM25 weighting). A document is reported once even when several of its instants match, and sub-second instants are honored (the bounds are encoded as fractional seconds). Under the Dynamic policy an array whose elements are all RFC 3339 strings infers a multi-valued DateTime; only RFC 3339 is recognised here (not the naive or date-only forms the query DSL accepts), and an array of strings where any element is not RFC 3339 infers a multi-valued Text instead (Issue #1175) — it is not an error. Note the asymmetry: a single RFC 3339 string still infers Text (unchanged, so existing text fields keep their behaviour), while an array of RFC 3339 strings infers a multi-valued DateTime — declare the field in the schema if you want a single-valued DateTime.

Boolean fields accept multi_valued = true as well (Issue #1180), but the mechanism differs from every other multi-valued type: a Boolean field has no BKD points. Every element of a multi-valued boolean field is indexed as its own "true" / "false" term posting, so TermQuery and DSL term queries such as flags:true match a document if any element equals the queried value (Lucene-style “any match”). A document is reported once for a given term even when several of its elements equal it (postings are aggregated per document and term), and Boolean fields remain not range-queryable. Because these are ordinary term postings, repeated elements affect BM25 scoring the way they do in Lucene: [true, true] yields one posting with term frequency 2 and field length 2, so it scores higher than [true] for flags:true, while [true, false] scores slightly lower than [true] because of length normalization. There is no dedupe and no constant scoring for term queries (constant-scored term queries are tracked in Issue #580). Under the Dynamic policy a JSON array whose elements are all booleans (e.g. [true, false]) infers a multi-valued Boolean; a mixed array such as [true, 1] is rejected with the existing error that array fields must contain only numeric values. Note the asymmetry with scalars: a single "true" string coerces to a Bool on a declared Boolean field, and a string array such as ["true", "false"] sent to a declared multi-valued Boolean field is parsed element-wise under the same rule; on an undeclared field, however, that array infers a multi-valued Text (Issue #1175), not a Boolean — use JSON booleans for arrays you want inferred as booleans. Turning multi_valued on for an existing Boolean field is a metadata-only change; turning it off requires a reindex when the field is stored, and is Destructive when the field is stored: false — unlike the BKD-backed types there is no point tree to rebuild from, only the stored value.

Text fields accept multi_valued = true as well (Issue #1175), together with a second option, position_increment_gap (default 100, matching Lucene and Elasticsearch). Every element of a multi-valued text field is analyzed on its own by the field’s analyzer, and the tokens of all elements are appended to one ascending position sequence: the first token of element n + 1 is placed position_increment_gap positions after the last token of element n. The gap is charged per element, even for an element that analyzes to no tokens. Everything else follows from that numbering:

• A TermQuery (or a DSL term such as tags:rust) matches a document if any element contains the term (Lucene-style “any match”).
• A PhraseQuery cannot span two elements unless its slop is at least position_increment_gap — the threshold is exactly slop == gap. With the default gap, ["hello world", "foo bar"] does not match the phrase "world foo" at slop 0 through 99 and does match at slop 100. Span queries (SpanNearQuery) inherit the same protection.
• A gap of 0 numbers the elements as if they had been concatenated, so phrases then match across element boundaries. The gap is never a “restart at 0”.
• Repeated terms across elements raise the term frequency (and hence the BM25 score) but not the hit count, like the other multi-valued types. The field length is the number of positions, not the number of tokens — a synonym stacked on another token (position_increment = 0) shares its position and is not counted again — so the gap itself does not inflate BM25 length normalization.
• Positions must be stored for phrase queries of two or more terms to work at all (term_vectors: true, the default); without them the search is rejected with a query error, for multi-valued and single-valued fields alike. A quoted value that analyzes to one token, such as tags:"lang/rust" on a keyword field, is a term match and does not need positions.

A multi-valued text field is highlighted element by element — only the elements that match contribute fragments, and a fragment never straddles two elements; see Highlighting. Under the Dynamic policy a JSON array whose elements are all strings infers a multi-valued DateTime when every element parses as RFC 3339 and a multi-valued Text otherwise (e.g. ["rust", "search"]); a single string still infers Text, never a datetime. Turning multi_valued on for an existing Text field is a metadata-only change; turning it off requires a reindex when the field is stored and is Destructive when it is stored: false — like Boolean, there is no BKD tree to rebuild from. Changing position_increment_gap requires a reindex when the field stores positions (term_vectors: true) — Reindex when stored, Destructive otherwise — because the existing postings were numbered under the old gap; when term_vectors is false the gap is unobservable and the change is metadata-only.

Bytes fields accept multi_valued = true as well (Issue #1176), but the mechanism differs from every other multi-valued type, Boolean included: BytesOption has no indexed flag at all — a Bytes value is never lexically indexed, so there are no BKD points and no term postings, and multi_valued has no “any match” query semantics to speak of. It only governs the stored shape and the ingestion arity. DataValue::BytesArray is Vec<(Vec<u8>, Option<String>)>: unlike the bare Vec<T> every other *Array variant uses, each element mirrors the scalar Bytes(Vec<u8>, Option<String>) shape exactly, carrying its own MIME type. A Bytes value — scalar or array — can never be inferred for an undeclared field under the Dynamic policy (see Type inference rules above); a multi-valued bytes field must always be declared explicitly. Once declared, a single Bytes value or a base64 Text string is auto-wrapped into a one-element array (a Text element is decoded as base64, the same rule the scalar field uses), and an empty numeric array ([]) is accepted as an empty byte-value list. Turning multi_valued on for an existing Bytes field is a metadata-only change; turning it off requires a reindex when the field is stored, and is Destructive when the field is stored: false — like Boolean and Text, there is no point tree or postings to rebuild from, only the stored value.

Faceting expands every multi-valued type element by element (Issue #1187): each integer, float, boolean, datetime or text element becomes its own facet value (a text element containing / becomes a hierarchical path), counted once per document even when an element repeats; geo and bytes arrays are not facetable and contribute nothing. See Faceting for the element formats.

Single values sent to a multi-valued field are auto-wrapped into a one-element array; arrays sent to a single-valued field are rejected rather than silently truncating (the error tells you to declare the field with multi_valued = true). An empty array sent to a multi-valued geo, datetime, boolean or text field is accepted and simply has no points, instants or terms, so it matches no spatial, range, term or phrase query; an empty array on a multi-valued bytes field is accepted the same way and is simply stored as an empty list — a Bytes field is never queried, so there is no query for it to not match.

Segments that contain multi-valued geo, datetime, boolean, text or bytes values use new stored-field type tags and cannot be read by builds that predate these features; there is no format version bump, so an older reader fails loudly instead of misreading the data. Stored multi-valued datetimes are kept at microsecond precision (one i64 Unix microsecond per instant, so sub-microsecond digits are truncated), whereas a single-valued DateTime keeps its full precision. Stored multi-valued booleans are written one byte per element (not bit-packed). Stored multi-valued text values are written as an element count followed by each string with its length prefix — the same body as a single-valued text value. Stored multi-valued bytes values are written as an element count followed by, for each element, the same (MIME, then data) length-prefixed layout the scalar Bytes value uses (an empty MIME string means None).

Type conflicts

When a value arrives for a field that is already declared, Laurus attempts to coerce the value to the declared type. The coercion rules are:

Coercion errors interact with the policy:

• Strict: error is returned immediately.
• Dynamic: error is returned — the coercion layer already applied every conversion that is considered safe.
• Ignore: the offending field is dropped; the rest of the document is indexed.

⚠️ Warning: silent information loss is possible.

Several coercions throw away information without reporting an error:

• An Integer field truncates incoming Float values (3.14 → 3, -3.9 → -3). Ingest does not fail.
• A Float field may lose precision for very large integers that do not fit in an f64 mantissa.
• A Text field accepts any scalar by stringifying it, losing the original type.
• Ignore drops incompatible fields quietly.

If the correctness of your data matters more than the convenience of schema-less ingestion, use DynamicFieldPolicy::Strict (or declare every field up-front). The Dynamic policy prioritises keeping the document ingestable over preserving every bit of incoming data.

Query DSL and undeclared fields

Once the schema is settled, the query parser validates that every field:value clause references a declared field. Typos such as titl:hello (for title:hello) produce a clear parse error instead of returning silently-empty results.

Dynamic Field Management

Fields can be added to, removed from, or changed on a running engine at runtime.

Adding a Field

Use Engine::add_field() to add a new field to the schema.

Adding a Lexical Field

let updated_schema = engine.add_field(
    "category",
    FieldOption::Text(TextOption::default()),
).await?;

Adding a Vector Field

let updated_schema = engine.add_field(
    "embedding",
    FieldOption::Flat(FlatOption::default().dimension(384)),
).await?;

Existing documents are unaffected—they simply have no value for the new field.

If a schema-persist hook was configured via Engine::builder().persist_schema_with(hook), add_field/delete_field/ update_field call it themselves before returning, persisting the schema on the caller’s behalf (both laurus-cli and laurus-server configure this hook to write schema.toml). Otherwise, as before, the returned Schema should be persisted (e.g., to schema.toml) by the caller.

Removing a Field

Use Engine::delete_field() to remove a field from the schema.

let updated_schema = engine.delete_field("category").await?;

When a field is deleted:

• The field definition is removed from the schema.
• Existing indexed data for the field remains in the index but becomes inaccessible through queries.
• If the field was listed in default_fields, it is automatically removed.
• Any per-field analyzer or embedder registered for the field is unregistered.

Changing a Field

Use Engine::update_field() to change an existing field’s type/options on a running engine.

let outcome = engine.update_field(
    "title",
    FieldOption::Text(TextOption::default().analyzer("english")),
    UpdateFieldOptions { reindex: true, ..Default::default() },
).await?;

The requested change is classified into one of three kinds, reported as outcome.classification:

• MetadataOnly — no existing data is affected; always applied (e.g. an HNSW field’s default_ef_search).
• Reindex — can be rebuilt from the field’s already-stored values (e.g. a text field’s analyzer or — while term_vectors is on — its position_increment_gap, indexed: false → true, or an HNSW field’s m/ef_construction).
• Destructive — cannot be rebuilt from existing data; applying it discards the field’s data (e.g. a vector field’s dimension/embedder/ distance, a type change on a stored: false field, or turning multi_valued off on a stored: false Boolean, Text or Bytes field).

Reindex and Destructive changes are rejected unless you explicitly pass UpdateFieldOptions { reindex: true, .. } — an opt-in gate against accidentally triggering an expensive rebuild or a data-losing change.

Applying a Destructive change records the field’s name in the schema’s pending_reindex set. This is a visibility mechanism against the known Solr failure mode where the schema and the index silently drift apart — check it via GetSchema / laurus get schema to find fields whose data no longer matches the schema, and re-ingest their documents as needed.

A vector field’s registered embedder follows its new embedder option (Issue #1354): naming one registers it, and removing the name unregisters the previous one, so the field falls back to the default embedder (text sent to it is rejected when that default is PrecomputedEmbedder). A change to a field that names no embedder, before or after, leaves an embedder registered directly through EngineBuilder::embedder in place.

Pass UpdateFieldOptions { dry_run: true, .. } to see how a change would be classified without applying anything.

Schema Design Tips

1. Separate lexical and vector fields — a field is either lexical or vector, never both. For hybrid search, create separate fields (e.g., body for text, body_vec for vector).

2. Use KeywordAnalyzer for exact-match fields — category, status, and tag fields should use KeywordAnalyzer via PerFieldAnalyzer to avoid tokenization.

3. Choose the right vector index — use HNSW for most cases, Flat for small datasets, IVF for very large datasets. See Vector Indexing.

4. Set default fields — if you use the Query DSL, set default fields so users can write hello instead of body:hello.

5. Use the schema generator — run laurus create schema to interactively build a schema TOML file instead of writing it by hand. See CLI Commands.