API Reference

Index

The primary entry point. Wraps the Laurus search engine.

Laurus::Index.new(path: nil, schema: nil, wal_sync_policy: nil, commit_policy: nil)

Constructor

Creating vs. reopening a file-backed index (path: given): if <path>/schema.toml does not yet exist, this call creates a new index and persists schema: (or an empty schema, if omitted) to it. If <path>/schema.toml already exists, this call reopens the index – schema: must be omitted, or an ArgumentError is raised (it would be ambiguous which schema should win). An ArgumentError is also raised if path: contains an index in the layout used before this convention was introduced (segment files directly under path:, no schema.toml).

Methods

search query argument

The query parameter accepts any of the following:

• A DSL string (e.g. "title:hello", "embedding:\"memory safety\"")
• A lexical query object (TermQuery, PhraseQuery, BooleanQuery, …)
• A vector query object (VectorQuery, VectorTextQuery)
• A SearchRequest for full control

When query is a SearchRequest, its own limit:, offset:, highlight: and rescore: are used and the keyword arguments of search are ignored.

The same value kinds are accepted as the elements of search_batch’s queries Array — DSL strings, query objects, and SearchRequest instances may be mixed within a single batch.

Highlighting

search/search_batch’s highlight: keyword (Issue #1134) requests highlighted fragments per field on each hit’s SearchResult#highlights. It accepts:

• An Array of field names: highlight: ["body"]
• A Hash with a required fields key (String or Symbol) plus any of the optional HighlightConfig knobs — max_fragments, fragment_size, tag, css_class, require_field_match, max_analyzed_chars, return_entire_field_if_no_highlight (String or Symbol keys both work) — e.g. highlight: {fields: ["body"], tag: "em", max_fragments: 2}

results = index.search("body:rust", highlight: ["body"])
results[0].highlights # => {"body" => ["<mark>Rust</mark> is a systems programming language."]}

Highlighting follows the query passed to search/search_batch (or SearchRequest’s query/lexical_query — see below), and only stored: true text fields can be highlighted; a field that is absent, not stored, or not text is silently skipped. Omitting highlight: leaves every result’s highlights empty. The same highlight: keyword is also accepted by SearchRequest.new.

WAL sync policy & durability

The write-ahead log (WAL) protects committed data against crashes. By default the WAL is fully durable: every record is fsynced before the write returns. You can trade some durability for higher write throughput by opting into group commit, which batches fsync calls.

WalSyncPolicy

Laurus::WalSyncPolicy is an immutable value object describing how the WAL is flushed. Pass it to Index.new(wal_sync_policy:).

# Default: durable per write (each record is fsynced individually).
Laurus::WalSyncPolicy.per_record

# Group commit: batch fsyncs to amortise their cost.
Laurus::WalSyncPolicy.group(
  max_records: nil,      # flush after this many records (default 1024)
  max_bytes: nil,        # flush after this many bytes (default 1 MiB)
  max_interval_ms: nil,  # also flush periodically after this many ms
)

Group commit is analogous to SQLite’s synchronous = NORMAL: a crash can lose at most the last unsynced batch of records, but the index never corrupts. Records are always made durable at commit, so a successful commit is a durability barrier regardless of policy.

Forcing a flush

Call flush_wal to force a durable barrier between commits — for example before signalling that a batch has been safely persisted. It synchronously fsyncs any unsynced records and returns nil. Under the default per-record policy it is effectively a no-op.

# Opt into group commit, then force durability on demand.
policy = Laurus::WalSyncPolicy.group(max_records: 4096, max_bytes: 4 * 1024 * 1024)
index = Laurus::Index.new(path: "./myindex", wal_sync_policy: policy)

index.put_document("doc1", { "title" => "Hello" })
index.flush_wal  # records persisted even though the group batch is not full

Commit policy & auto-commit

By default the caller drives every commit: buffered writes only become searchable once you call commit explicitly. You can hand that responsibility to the engine with an auto-commit policy, which commits automatically after a fixed number of applied documents or on a periodic timer.

CommitPolicy

Laurus::CommitPolicy is an immutable value object describing when the engine materialises buffered writes into the stores. Pass it to Index.new(commit_policy:).

# Default: manual — the caller drives every commit.
Laurus::CommitPolicy.manual

# Auto-commit: commit after every N applied documents.
Laurus::CommitPolicy.every_docs(1000)

# Auto-commit: commit at least every N milliseconds (native only).
Laurus::CommitPolicy.interval_ms(5000)

every_docs(0) is valid and disables auto-commit, making it equivalent to manual.

Commit policy is orthogonal to WalSyncPolicy: WalSyncPolicy governs WAL fsync durability, whereas CommitPolicy governs when the stores materialise buffered writes. Set them independently.

# Auto-commit after every 1000 applied documents.
policy = Laurus::CommitPolicy.every_docs(1000)
index = Laurus::Index.new(path: "./myindex", commit_policy: policy)

Schema

Defines the fields and index types for an Index.

Laurus::Schema.new

Field methods

Vector quantization & rerank storage (HNSW fields):

• quantizer — "scalar_8bit" (default, 4× compression) or "product_quantization" for higher compression. Product quantization requires subvector_count (must divide dimension).
• rerank_storage — set to "f32" to write a full-precision *.hnsw.f32 sidecar enabling exact Stage-2 rerank; omit to keep the int8-only segment.
• pq_codebook_path — storage-relative file name of a shared PQ codebook (Issue #631), trained once via the laurus train pq-codebook CLI command. Only meaningful with quantizer: "product_quantization"; commits then encode against the pre-trained codebook instead of re-training k-means per segment. Omit to keep per-segment training.

Every add_*_field method above raises ArgumentError when name starts with _ (other than _id), and adds nothing. A schema loaded with from_toml / from_toml_file keeps accepting such a field, so a persisted schema still loads; creating a new Index from it raises ArgumentError. See Field Naming.

Other methods

Dynamic field policy

Controls what happens when a document is ingested with field names that are not declared in the schema:

• "strict" — Reject the document.
• "dynamic" (default) — Infer a type for each undeclared field and add it to the schema. Warning: integer fields silently truncate incoming float values (3.14 → 3). Use "strict" if you need to reject such type mismatches.
• "ignore" — Silently drop the undeclared fields.

See Schema & Fields for the full behaviour matrix.

Embedder types

See Schema Format Reference → Embedders for the canonical description of each type.

"candle_colbert" also takes the optional keys "revision", "query_maxlen" and "doc_maxlen". It produces token vectors, so only a multi-vector field (add_multi_vector_field) can use it. Keys may be Strings or Symbols:

schema = Laurus::Schema.new
schema.add_embedder(
  "colbert",
  { type: "candle_colbert", model: "colbert-ir/colbertv2.0", query_maxlen: 32 },
)
schema.add_multi_vector_field("body_colbert", 128, embedder: "colbert")

Analyzer components

Used by add_analyzer(name, tokenizer, char_filters: nil, token_filters: nil) and by the [analyzers.<name>] TOML section. tokenizer is a single Hash; char_filters:/token_filters: are Arrays of Hashes, applied in Array order.

See Schema Format Reference → Analyzers for the canonical description of each component.

Tokenizers (tokenizer, exactly one):

Char filters (char_filters, applied to raw text before tokenization):

Token filters (token_filters, applied to the token stream after tokenization):

schema = Laurus::Schema.new
schema.add_analyzer(
  "ja_ipadic",
  { type: "lindera", mode: "normal", dict: "/var/lib/lindera/ipadic" },
  char_filters: [
    { type: "unicode_normalization", form: "nfkc" },
    { type: "japanese_iteration_mark" },
  ],
  token_filters: [{ type: "lowercase" }],
)
schema.add_text_field("title", analyzer: "ja_ipadic")

Distance metrics

Query classes

TermQuery

Laurus::TermQuery.new(field, term)

Matches documents containing the exact term in the given field.

PhraseQuery

Laurus::PhraseQuery.new(field, terms)

Matches documents containing the terms in order. terms is an Array of Strings.

FuzzyQuery

Laurus::FuzzyQuery.new(field, term, max_edits: 2)

Approximate match allowing up to max_edits edit-distance errors.

WildcardQuery

Laurus::WildcardQuery.new(field, pattern)

Pattern match. * matches any sequence of characters, ? matches any single character.

NumericRangeQuery

Laurus::NumericRangeQuery.new(field, min: nil, max: nil)

Matches numeric values in the range [min, max]. Pass nil for an open bound. The type (integer or float) is inferred from the Ruby type of min/max.

DateTimeRangeQuery

Laurus::DateTimeRangeQuery.new(field, min: nil, max: nil)

Matches DateTime values in the range [min, max] (both bounds inclusive). Pass nil (or omit the keyword) for an open bound. A bound is a String literal in any form the query DSL accepts — RFC 3339 ("2024-01-01T09:00:00+09:00", normalized to UTC), naive "YYYY-MM-DDTHH:MM:SS[.fff]" (UTC), or "YYYY-MM-DD" (midnight UTC) — or any object responding to iso8601 (Time, DateTime). A malformed bound raises ArgumentError at construction. Usable wherever a query object is accepted (Index#search, BooleanQuery, SearchRequest).

GeoDistanceQuery

Laurus::GeoDistanceQuery.within_radius(field, lat, lon, distance_m)

Geo-distance (radius) search. Returns documents whose (lat, lon) coordinate is within distance_m metres of the given point.

GeoBoundingBoxQuery

Laurus::GeoBoundingBoxQuery.within_bounding_box(
  field, min_lat, min_lon, max_lat, max_lon,
)

Geo bounding-box search. Returns documents whose (lat, lon) coordinate lies inside the axis-aligned [min_lat, max_lat] × [min_lon, max_lon] rectangle.

Geo3dDistanceQuery

Laurus::Geo3dDistanceQuery.within_sphere(field, x, y, z, distance_m)

Sphere search over a 3D ECEF point field. Returns documents whose (x, y, z) coordinate is within distance_m metres of the centre. See Geo3d concepts for ECEF theory.

Geo3dBoundingBoxQuery

Laurus::Geo3dBoundingBoxQuery.within_box(
  field,
  min_x, min_y, min_z,
  max_x, max_y, max_z,
)

Axis-aligned 3D bounding-box search.

Geo3dNearestQuery

Laurus::Geo3dNearestQuery.k_nearest(
  field, x, y, z, k,
  initial_radius_m: nil,
  max_radius_m: nil,
)

k-nearest-neighbour search over a 3D ECEF point field. The optional initial_radius_m: and max_radius_m: keyword arguments tune the iterative-expansion search cone.

BooleanQuery

bq = Laurus::BooleanQuery.new
bq.must(query)
bq.should(query)
bq.must_not(query)

Compound boolean query. must clauses all have to match; must_not clauses must not match. should clauses contribute to scoring; at least one of them must match if there are no must clauses.

SpanQuery

# Single term
Laurus::SpanQuery.term(field, term)

# Near: terms within slop positions
Laurus::SpanQuery.near(field, terms, slop: 0, ordered: true)

# Near with nested SpanQuery clauses
Laurus::SpanQuery.near_spans(field, clauses, slop: 0, ordered: true)

# Containing: big span contains little span
Laurus::SpanQuery.containing(field, big, little)

# Within: include span within exclude span at max distance
Laurus::SpanQuery.within(field, include_span, exclude_span, distance)

Positional / proximity span queries. near takes an Array of term Strings, while near_spans takes an Array of SpanQuery objects for nested expressions.

VectorQuery

Laurus::VectorQuery.new(field, vector)

Approximate nearest-neighbor search using a pre-computed embedding vector. vector is an Array of Floats.

VectorTextQuery

Laurus::VectorTextQuery.new(field, text)

Converts text to an embedding at query time and runs vector search. Requires an embedder configured on the index.

SearchRequest

Full-featured search request for advanced control.

Laurus::SearchRequest.new(
  query: nil,
  lexical_query: nil,
  vector_query: nil,
  filter_query: nil,
  fusion: nil,
  limit: 10,
  offset: 0,
  highlight: nil,
  rescore: nil,
)

LateInteractionRescore

Late-interaction (ColBERT MaxSim) rescore of the top search results (Issue #1351). Pass it as rescore: to Index#search or SearchRequest.new.

Laurus::LateInteractionRescore.new(field, query, window_size: nil)

The top window_size first-stage results are reordered by their MaxSim against the field, and a rescored result’s score is its MaxSim. Results beyond the window keep their first-stage order and score, after the rescored ones. See Vector Search → Late-Interaction Rescore for the ordering rules and the similarity.

Errors: the constructor raises TypeError when query is neither a String nor an Array of numeric Arrays. The other values are checked by the engine when searching, which raises ArgumentError with a message containing rescore: ... when the field is unknown or not a multi-vector field, a text query is blank or the field has no token-level embedder, the query does not hold between 1 and 1,024 token vectors of the field’s dimension, or window_size is not between 1 and 10,000.

schema = Laurus::Schema.new
schema.add_text_field("title")
schema.add_multi_vector_field("tokens", 2, distance: "dot_product")

index = Laurus::Index.new(schema: schema)
index.put_document("a", { "title" => "rust", "tokens" => [[0.1, 0.0]] })
index.put_document("b", { "title" => "rust language", "tokens" => [[0.9, 0.2]] })
index.commit

rescore = Laurus::LateInteractionRescore.new("tokens", [[1.0, 0.0], [0.0, 1.0]], window_size: 50)
results = index.search("title:rust", rescore: rescore)
results.map(&:id) # => ["b", "a"] -- "b" has MaxSim 1.1, "a" 0.1

# With a "candle_colbert" embedder on the field, the query can be text.
rescore = Laurus::LateInteractionRescore.new("body_colbert", "how do lifetimes work")

SearchResult

Returned by Index#search.

result.id         # => String   -- External document identifier
result.score      # => Float    -- Relevance score
result.document   # => Hash|nil -- Retrieved field values, or nil if deleted
result.highlights # => Hash     -- Highlighted fragments per requested field

highlights maps each field named in highlight: to its highlighted fragments (best first); a field that did not highlight is absent from the Hash, and highlights is {} when highlight: was not requested. See Highlighting.

Fusion algorithms

RRF

Laurus::RRF.new(k: 60.0)

Reciprocal Rank Fusion. Merges lexical and vector result lists by rank position. k is a smoothing constant; higher values reduce the influence of top-ranked results.

WeightedSum

Laurus::WeightedSum.new(lexical_weight: 0.5, vector_weight: 0.5)

Normalises both score lists independently, then combines them as lexical_weight * lexical_score + vector_weight * vector_score.

Text analysis

SynonymDictionary

dict = Laurus::SynonymDictionary.new
dict.add_synonym_group(["fast", "quick", "rapid"])

A dictionary of synonym groups. All terms in a group are treated as synonyms of each other.

WhitespaceTokenizer

tokenizer = Laurus::WhitespaceTokenizer.new
tokens = tokenizer.tokenize("hello world")

Splits text on whitespace boundaries and returns an Array of Token objects.

SynonymGraphFilter

filter = Laurus::SynonymGraphFilter.new(dictionary, keep_original: true, boost: 1.0)
expanded = filter.apply(tokens)

Token filter that expands tokens with their synonyms from a SynonymDictionary.

Token

token.text                # => String  -- The token text
token.position            # => Integer -- Position in the token stream
token.start_offset        # => Integer -- UTF-8 byte start offset in the original text
token.end_offset          # => Integer -- UTF-8 byte end offset in the original text
token.boost               # => Float   -- Score boost factor (1.0 = no adjustment)
token.stopped             # => Boolean -- Whether removed by a stop filter
token.position_increment  # => Integer -- Difference from the previous token's position
token.position_length     # => Integer -- Number of positions spanned
token.token_type          # => String or nil -- Token type, e.g. "alphanum"

The offsets count bytes, not characters; for non-ASCII text slice the bytes: text.byteslice(token.start_offset...token.end_offset).

token_type is one of "alphanum", "num", "cjk", "katakana", "hiragana", "hangul", "punctuation", "whitespace", "synonym", "email", "url" and "other", or nil. SynonymGraphFilter#apply keeps each token’s offsets and type, and gives each synonym it inserts the type "synonym" and the offsets of the words it replaces.

Field value types

Ruby values are automatically converted to Laurus DataValue types: