API Reference

Index

The primary entry point. Wraps the Laurus search engine.

new \Laurus\Index(?string $path = null, ?Schema $schema = null, ?WalSyncPolicy $wal_sync_policy = null, ?CommitPolicy $commit_policy = null)

Constructor

Creating vs. reopening a file-backed index ($path given): if <path>/schema.toml does not yet exist, this call creates a new index and persists $schema (or an empty schema, if omitted) to it. If <path>/schema.toml already exists, this call reopens the index – $schema must be omitted, or a ValueError is thrown (it would be ambiguous which schema should win). A ValueError is also thrown if $path contains an index in the layout used before this convention was introduced (segment files directly under $path, no schema.toml).

Methods

search query argument

The $query parameter accepts any of the following:

• A DSL string (e.g. "title:hello", "embedding:\"memory safety\"")
• A lexical query object (TermQuery, PhraseQuery, BooleanQuery, …)
• A vector query object (VectorQuery, VectorTextQuery)
• A SearchRequest for full control

The same value kinds are accepted as the elements of searchBatch’s $queries array — DSL strings, query objects, and SearchRequest instances may be mixed within a single batch.

Highlighting

search/searchBatch’s $highlight parameter (Issue #1134) requests highlighted fragments per field on each hit’s SearchResult::getHighlights(). It accepts:

• A list of field names: ["body"]
• An associative array with a required "fields" key plus any of the optional HighlightConfig knobs — max_fragments, fragment_size, tag, css_class, require_field_match, max_analyzed_chars, return_entire_field_if_no_highlight — e.g. ["fields" => ["body"], "tag" => "em", "max_fragments" => 2]

$results = $index->search("body:rust", 10, 0, ["body"]);
$results[0]->getHighlights(); // ["body" => ["<mark>Rust</mark> is a systems programming language."]]

Highlighting follows the query passed to search/searchBatch (or SearchRequest’s $query/$lexicalQuery — see below), and only stored: true text fields can be highlighted; a field that is absent, not stored, or not text is silently skipped. Omitting $highlight leaves every result’s highlights empty. The same highlight argument is also accepted by SearchRequest’s constructor.

WAL sync policy & durability

The write-ahead log (WAL) protects committed data against crashes. By default the WAL is fully durable: every record is fsynced before the write returns. You can trade some durability for higher write throughput by opting into group commit, which batches fsync calls.

WalSyncPolicy

Laurus\WalSyncPolicy is an immutable value object describing how the WAL is flushed. Pass it to the Index constructor’s $wal_sync_policy argument.

// Default: durable per write (each record is fsynced individually).
\Laurus\WalSyncPolicy::perRecord(): WalSyncPolicy

// Group commit: batch fsyncs to amortise their cost.
\Laurus\WalSyncPolicy::group(
    ?int $max_records = null,      // flush after this many records (default 1024)
    ?int $max_bytes = null,        // flush after this many bytes (default 1 MiB)
    ?int $max_interval_ms = null,  // also flush periodically after this many ms
): WalSyncPolicy

Group commit is analogous to SQLite’s synchronous = NORMAL: a crash can lose at most the last unsynced batch of records, but the index never corrupts. Records are always made durable at commit(), so a successful commit() is a durability barrier regardless of policy.

Forcing a flush

Call flushWal() to force a durable barrier between commits — for example before signalling that a batch has been safely persisted. It synchronously fsyncs any unsynced records. Under the default per-record policy it is effectively a no-op.

// Opt into group commit, then force durability on demand.
$policy = \Laurus\WalSyncPolicy::group(4096, 4 * 1024 * 1024);
$index = new \Laurus\Index("./myindex", null, $policy);

$index->putDocument("doc1", ["title" => "Hello"]);
$index->flushWal(); // records persisted even though the group batch is not full

Commit policy & auto-commit

A commit materialises buffered writes into the lexical and vector stores and makes pending changes searchable. By default Laurus never commits on your behalf — the caller drives every commit(). You can instead let the engine auto-commit after a fixed number of applied documents, or at least once per fixed time interval.

CommitPolicy

Laurus\CommitPolicy is an immutable value object describing when the engine commits. Pass it to the Index constructor’s $commit_policy argument.

// Default: no auto-commit — the caller drives every commit().
\Laurus\CommitPolicy::manual(): CommitPolicy

// Auto-commit after every N applied documents.
\Laurus\CommitPolicy::everyDocs(
    int $n,   // commit after this many applied documents
): CommitPolicy

// Auto-commit at least every N milliseconds (native only; no-op on wasm).
\Laurus\CommitPolicy::intervalMs(
    int $ms,   // commit at least this often, in milliseconds
): CommitPolicy

CommitPolicy::everyDocs(0) is valid and disables auto-commit — it is equivalent to CommitPolicy::manual().

The commit policy is orthogonal to the WAL sync policy: WalSyncPolicy governs when the WAL is fsynced for durability, whereas CommitPolicy governs when the stores materialise and pending changes become searchable. The two are configured independently.

// Auto-commit every 1000 applied documents; keep the default WAL policy.
$index = new \Laurus\Index(null, $schema, null, \Laurus\CommitPolicy::everyDocs(1000));

foreach ($docs as $id => $doc) {
    $index->putDocument($id, $doc); // engine commits automatically every 1000 docs
}

Schema

Defines the fields and index types for an Index.

new \Laurus\Schema()

Field methods

Vector quantization & rerank storage (HNSW fields; quantizer also on Flat and IVF fields):

• quantizer — "scalar_8bit" (default, 4× compression), "float16" (IEEE 754 half precision, 2× compression, nothing to train; Issue #623) or "product_quantization" for higher compression. Product quantization requires subvectorCount (must divide dimension). addFlatField and addIvfField take $quantizer as their last argument and accept "scalar_8bit" and "float16"; "product_quantization" is HNSW-only and throws \Exception there.
• rerankStorage — set to "f32" to write a full-precision *.hnsw.f32 sidecar enabling exact Stage-2 rerank; omit to keep the int8-only segment.
• pqCodebookPath — storage-relative file name of a shared PQ codebook (Issue #631), trained once via the laurus train pq-codebook CLI command. Only meaningful with $quantizer = "product_quantization"; commits then encode against the pre-trained codebook instead of re-training k-means per segment. Omit to keep per-segment training.

Every add*Field method above throws \ValueError when name starts with _ (other than _id), and adds nothing. A schema loaded with fromToml / fromTomlFile keeps accepting such a field, so a persisted schema still loads; creating a new Index from it throws \ValueError. See Field Naming.

Other methods

Dynamic field policy

Controls what happens when a document is ingested with field names that are not declared in the schema:

• "strict" — Reject the document.
• "dynamic" (default) — Infer a type for each undeclared field and add it to the schema. Warning: integer fields silently truncate incoming float values (3.14 → 3). Use "strict" if you need to reject such type mismatches.
• "ignore" — Silently drop the undeclared fields.

See Schema & Fields for the full behaviour matrix.

Embedder types

See Schema Format Reference → Embedders for the canonical description of each type.

"candle_colbert" produces one vector per token, so it serves only a multi-vector field (addMultiVectorField):

$schema->addEmbedder("colbert", ["type" => "candle_colbert", "model" => "colbert-ir/colbertv2.0"]);
$schema->addMultiVectorField("body_colbert", 128, null, "colbert");

Analyzer components

Used by addAnalyzer(string $name, array $tokenizer, ?array $charFilters = null, ?array $tokenFilters = null) and by the [analyzers.<name>] TOML section. $tokenizer is a single associative array; $charFilters/$tokenFilters are arrays of associative arrays, applied in array order.

See Schema Format Reference → Analyzers for the canonical description of each component.

Tokenizers ($tokenizer, exactly one):

Char filters ($charFilters, applied to raw text before tokenization):

Token filters ($tokenFilters, applied to the token stream after tokenization):

$schema = new Laurus\Schema();
$schema->addAnalyzer(
    "ja_ipadic",
    ["type" => "lindera", "mode" => "normal", "dict" => "/var/lib/lindera/ipadic"],
    [
        ["type" => "unicode_normalization", "form" => "nfkc"],
        ["type" => "japanese_iteration_mark"],
    ],
    [["type" => "lowercase"]],
);
$schema->addTextField("title", analyzer: "ja_ipadic");

Distance metrics

addMultiVectorField accepts only "cosine" and "dot_product".

Query classes

TermQuery

new \Laurus\TermQuery(string $field, string $term)

Matches documents containing the exact term in the given field.

PhraseQuery

new \Laurus\PhraseQuery(string $field, array $terms)

Matches documents containing the terms in order. $terms is an array of strings.

FuzzyQuery

new \Laurus\FuzzyQuery(string $field, string $term, int $maxEdits = 2)

Approximate match allowing up to $maxEdits edit-distance errors.

WildcardQuery

new \Laurus\WildcardQuery(string $field, string $pattern)

Pattern match. * matches any sequence of characters, ? matches any single character.

NumericRangeQuery

new \Laurus\NumericRangeQuery(string $field, mixed $min, mixed $max, ?string $numericType = "integer")

Matches numeric values in the range [$min, $max]. Pass null for an open bound. Set $numericType to "integer" or "float".

DateTimeRangeQuery

new \Laurus\DateTimeRangeQuery(string $field, ?string $min = null, ?string $max = null)

Matches DateTime values in the range [$min, $max] (both bounds inclusive). Pass null for an open bound. Bounds are string literals in any form the query DSL accepts: RFC 3339 ("2024-01-01T09:00:00+09:00", normalized to UTC), naive "YYYY-MM-DDTHH:MM:SS[.fff]" (UTC), or "YYYY-MM-DD" (midnight UTC); pass $dt->format(DATE_RFC3339) for a DateTimeInterface. Because ext-php-rs constructors cannot fail, a malformed bound is reported as a \Throwable when the query is used (Index::search, BooleanQuery, SearchRequest).

GeoDistanceQuery

\Laurus\GeoDistanceQuery::withinRadius(
    string $field, float $lat, float $lon, float $distanceM,
): GeoDistanceQuery

Geo-distance (radius) search. Returns documents whose (lat, lon) coordinate is within $distanceM metres of the given point.

GeoBoundingBoxQuery

\Laurus\GeoBoundingBoxQuery::withinBoundingBox(
    string $field,
    float $minLat, float $minLon,
    float $maxLat, float $maxLon,
): GeoBoundingBoxQuery

Geo bounding-box search. Returns documents whose (lat, lon) coordinate lies inside the axis-aligned [$minLat, $maxLat] × [$minLon, $maxLon] rectangle.

Geo3dDistanceQuery

\Laurus\Geo3dDistanceQuery::withinSphere(
    string $field,
    float $x, float $y, float $z,
    float $distanceM,
): Geo3dDistanceQuery

Sphere search over a 3D ECEF point field. Returns documents whose (x, y, z) coordinate is within $distanceM metres of the centre. See Geo3d concepts for ECEF theory.

Geo3dBoundingBoxQuery

\Laurus\Geo3dBoundingBoxQuery::withinBox(
    string $field,
    float $minX, float $minY, float $minZ,
    float $maxX, float $maxY, float $maxZ,
): Geo3dBoundingBoxQuery

Axis-aligned 3D bounding-box search.

Geo3dNearestQuery

\Laurus\Geo3dNearestQuery::kNearest(
    string $field,
    float $x, float $y, float $z,
    int $k,
    ?float $initialRadiusM = null,
    ?float $maxRadiusM = null,
): Geo3dNearestQuery

k-nearest-neighbour search over a 3D ECEF point field. The optional $initialRadiusM and $maxRadiusM parameters tune the iterative-expansion search cone.

BooleanQuery

$bq = new \Laurus\BooleanQuery();
$bq->must($query);
$bq->should($query);
$bq->mustNot($query);

Compound boolean query. must clauses all have to match; mustNot clauses must not match. should clauses contribute to scoring; at least one of them must match if there are no must clauses.

SpanQuery

// Single term
\Laurus\SpanQuery::term(string $field, string $term): SpanQuery

// Near: terms within slop positions
\Laurus\SpanQuery::near(string $field, array $terms, int $slop = 0, bool $ordered = true): SpanQuery

// NearSpans: nested SpanQuery clauses within slop positions
\Laurus\SpanQuery::nearSpans(string $field, array $clauses, int $slop = 0, bool $ordered = true): SpanQuery

// Containing: big span contains little span
\Laurus\SpanQuery::containing(string $field, SpanQuery $big, SpanQuery $little): SpanQuery

// Within: include span within exclude span at max distance
\Laurus\SpanQuery::within(string $field, SpanQuery $include, SpanQuery $exclude, int $distance): SpanQuery

Positional / proximity span queries. near takes an array of term strings, while nearSpans takes an array of SpanQuery objects for nested expressions (each clause’s field is re-rooted to the outer $field).

VectorQuery

new \Laurus\VectorQuery(string $field, array $vector)

Approximate nearest-neighbor search using a pre-computed embedding vector. $vector is an array of floats, each finite and within the range of a 32-bit float; the search throws \ValueError otherwise (Issue #1573).

VectorTextQuery

new \Laurus\VectorTextQuery(string $field, string $text)

Converts $text to an embedding at query time and runs vector search. Requires an embedder configured on the index.

SearchRequest

Full-featured search request for advanced control.

new \Laurus\SearchRequest(
    mixed $query = null,
    mixed $lexicalQuery = null,
    mixed $vectorQuery = null,
    mixed $filterQuery = null,
    mixed $fusion = null,
    int $limit = 10,
    int $offset = 0,
    ?array $highlight = null,
    ?LateInteractionRescore $rescore = null,
    ?GroupBy $groupBy = null,
)

When a SearchRequest is passed to Index->search(), its own $limit, $offset, $highlight, $rescore and $groupBy are used; search’s other arguments are ignored.

LateInteractionRescore

Late-interaction (ColBERT MaxSim) rescore of the top search results (Issue #1351). Pass it as Index->search()’s 5th argument or as SearchRequest’s $rescore. See Vector Search → Late-Interaction Rescore for how it works.

new \Laurus\LateInteractionRescore(string $field, string|array $query, ?int $windowSize = null)

Methods

Ordering and scores

The top $windowSize first-stage results – lexical, vector, or hybrid – are reordered by their MaxSim against the field, and a rescored result’s getScore() is its MaxSim. Results beyond the window keep their first-stage order and score, after the rescored ones (window results without token vectors in the field come between the two, in first-stage order). The two kinds of score are not comparable.

Errors

• The constructor throws \TypeError (query must be a string or a list of numeric lists) when $query is neither, and \Exception (token vector N must hold only numbers) when a token vector holds a non-number such as a bool or string.
• Index->search() and SearchRequest throw \TypeError (rescore must be a Laurus\LateInteractionRescore) for any other $rescore object.
• Everything else is checked by the engine when searching, before any search runs, and throws \ValueError with a message containing rescore: ...: the field is unknown or not a multi-vector field, the query does not hold between 1 and 1,024 finite vectors of the field’s dimension, a text query is blank or the field has no token-level embedder, or $windowSize is outside 1..=10,000.

Example

$schema = new Laurus\Schema();
$schema->addTextField("title");
$schema->addMultiVectorField("tokens", 2, "dot_product");

$index = new Laurus\Index(null, $schema);
$index->putDocument("a", ["title" => "rust", "tokens" => [[0.1, 0.0]]]);
$index->putDocument("b", ["title" => "rust language", "tokens" => [[0.9, 0.2]]]);
$index->commit();

$rescore = new Laurus\LateInteractionRescore("tokens", [[1.0, 0.0], [0.0, 1.0]], 50);
$results = $index->search("title:rust", 10, 0, null, $rescore);
$results[0]->getId();    // "b"
$results[0]->getScore(); // ≈ 1.1 = 0.9 + 0.2 (MaxSim)

// The same rescore through a SearchRequest (9th argument)
$results = $index->search(new Laurus\SearchRequest("title:rust", null, null, null, null, 10, 0, null, $rescore));

// A text query needs a token-level embedder on the field (see "Embedder types")
$rescore = new Laurus\LateInteractionRescore("body_colbert", "how do lifetimes work");

GroupBy

Groups the search results by a field’s value (Issue #1297), so that the chunks of one document do not fill the page. Pass it as the $groupBy argument of Index->search() or Index->searchBatch(), or as SearchRequest’s $groupBy.

new \Laurus\GroupBy(string $field, ?int $groupSize = null)

Methods

Results and errors

Each returned SearchResult is then the best hit of a group: its getGroupKey() is the value the group shares and its getInnerHits() hold up to $groupSize - 1 further hits of the group, in ranking order. $limit and $offset count groups. Grouping runs on the final ranking, after fusion and rescore, for lexical, vector and hybrid searches alike. See Result Grouping for the ordering, the candidate pool and why a group can hold fewer than $groupSize hits.

• Index->search(), Index->searchBatch() and SearchRequest throw \TypeError (groupBy must be a Laurus\GroupBy) for any other $groupBy value.
• Everything else is checked by the engine when searching, before any search runs, and throws \ValueError with a message containing group_by: ...: the field is unknown, of another type, multi-valued, not stored or without docValues, $groupSize is outside 1..=100 (a negative size is reported as 0), or $offset + $limit exceeds 10,000.

Example

$schema = new Laurus\Schema();
$schema->addTextField("body");

$index = new Laurus\Index(null, $schema);
$index->addDocument("manual", ["body" => "Lifetimes tie a reference to its owner."]);
$index->addDocument("manual", ["body" => "Elided lifetimes follow three rules."]);
$index->addDocument("faq", ["body" => "Why do lifetimes appear in signatures?"]);
$index->commit();

$results = $index->search("body:lifetimes", groupBy: new Laurus\GroupBy("_id", groupSize: 2));
$results[0]->getGroupKey();                            // "faq"
$results[1]->getGroupKey();                            // "manual"
$results[1]->getInnerHits()[0]->getDocument()["body"]; // "Elided lifetimes follow three rules."

// The same grouping through a SearchRequest (10th argument)
$request = new Laurus\SearchRequest("body:lifetimes", null, null, null, null, 10, 0, null, null, new Laurus\GroupBy("_id", groupSize: 2));
$results = $index->search($request);

SearchResult

Returned by Index->search().

$result->getId()          // string   -- External document identifier
$result->getScore()       // float    -- Relevance score
$result->getDocument()    // array|null -- Retrieved field values, or null if not stored
$result->getHighlights()  // array    -- Highlighted fragments per requested field
$result->getGroupKey()    // mixed    -- With $groupBy, the group's value
$result->getInnerHits()   // array    -- With $groupBy, the group's next best hits

getHighlights() maps each field named in $highlight to its highlighted fragments (best first); a field that did not highlight is absent from the array, and it returns [] when $highlight was not requested. See Highlighting.

With $groupBy, getGroupKey() returns the value the result’s group shares (a string, int, float or bool, or a datetime as an RFC 3339 string; null for a hit without a value) and getInnerHits() returns the group’s next best SearchResults in ranking order, each with the same group key and no inner hits of its own. An ungrouped search returns null and []. See GroupBy.

Fusion algorithms

RRF

new \Laurus\RRF(float $k = 60.0)

Reciprocal Rank Fusion. Merges lexical and vector result lists by rank position. $k is a smoothing constant; higher values reduce the influence of top-ranked results. $k must be finite and at least 0; the search throws \ValueError otherwise (Issue #1540).

WeightedSum

new \Laurus\WeightedSum(float $lexicalWeight = 0.5, float $vectorWeight = 0.5)

Normalises both score lists independently, then combines them as $lexicalWeight * lexical_score + $vectorWeight * vector_score. Each weight must be in [0.0, 1.0], and at least one greater than 0; the search throws \ValueError otherwise instead of clamping them (Issue #1540).

Text analysis

SynonymDictionary

$dict = new \Laurus\SynonymDictionary();
$dict->addSynonymGroup(["fast", "quick", "rapid"]);

A dictionary of synonym groups. All terms in a group are treated as synonyms of each other.

WhitespaceTokenizer

$tokenizer = new \Laurus\WhitespaceTokenizer();
$tokens = $tokenizer->tokenize("hello world");

Splits text on whitespace boundaries and returns an array of Token objects.

SynonymGraphFilter

new \Laurus\SynonymGraphFilter(SynonymDictionary $dictionary, bool $keepOriginal = true, float $boost = 1.0)

$filter = new \Laurus\SynonymGraphFilter($dictionary, true, 1.0);
$expanded = $filter->apply($tokens);

Token filter that expands tokens with their synonyms from a SynonymDictionary.

Token

$token->getText()               // string  -- The token text
$token->getPosition()           // int     -- Position in the token stream
$token->getStartOffset()        // int     -- UTF-8 byte start offset in the original text
$token->getEndOffset()          // int     -- UTF-8 byte end offset in the original text
$token->getBoost()              // float   -- Score boost factor (1.0 = no adjustment)
$token->isStopped()             // bool    -- Whether removed by a stop filter
$token->getPositionIncrement()  // int     -- Difference from the previous token's position
$token->getPositionLength()     // int     -- Number of positions spanned
$token->getTokenType()          // ?string -- Token type, e.g. "alphanum"

The offsets count bytes, as PHP strings do, so substr($text, $start, $end - $start) gives the token’s text.

getTokenType() returns one of "alphanum", "num", "cjk", "katakana", "hiragana", "hangul", "punctuation", "whitespace", "synonym", "email", "url" and "other", or null. SynonymGraphFilter::apply() keeps each token’s offsets and type, and gives each synonym it inserts the type "synonym" and the offsets of the words it replaces.

Field value types

PHP values are automatically converted to Laurus DataValue types: