yggvault ratatoskr-space connected via regular web
Color theme
also available via yggdrasil mesh http://[203:b338:2a84:a18f:986:47ae:1a4:d8d3]/pkg/meilisearch/v1.37.0
vault / meilisearch / v1.37.0

meilisearch @ v1.37.0

integrity

size
20.2 MiB
downloaded
last checked
source https://github.com/meilisearch/meilisearch · available · github

release notes

[!IMPORTANT]
This release contains breaking changes for users of the network experimental feature.

Meilisearch v1.37 introduces replicated sharding, removes the vectorStoreSetting experimental feature, stabilizes our new vector store for best performance, adds a security fix and miscellaneous improvements.

✨ Improvements

§ Replicated sharding

[!NOTE] Replicated sharding requires Meilisearch Enterprise Edition (EE).

  • Users of Meilisearch Cloud, please contact support if you need replicated sharding.
  • Users of the Community Edition, please contact the sales if you want to use replicated sharding in production.

§ Breaking changes

Existing databases will be migrated automatically when upgraded with --experimental-dumpless-upgrade when leader is not null, such that for each remote:

  1. A shard with the same name as the remote is created
  2. This shard has exactly one remote in its remotes list: the remote with the same name as the shard.

This change will not cause any document to be resharded.

To be able to upgrade without resharding, the migration uses the same name for remotes and for shards. However, in new configurations, we recommend using different names for shards and remotes.

For instance, the following network object:

{
  "leader": "ms-00",
  "self": "ms-01",
  "remotes": {
    "ms-00": { /* .. */ },
    "ms-01": { /* .. */ }
  }
}

is converted to:

{
  "leader": "ms-00",
  "self": "ms-01",
  "remotes": {
    "ms-00": { /* .. */ },
    "ms-01": { /* .. */ }
  },
  "shards": {  // ✨ NEW
    "ms-00": {  // shard named like the remote
      "remotes": ["ms-00"] // is owned by the remote
    },
    "ms-01": {
      "remotes": ["ms-01"]
    }
  }
}

Addition of network.shards

The network object for routes PATCH /network and GET /network now contains the new field shards, which is an object whose values are shard objects, and keys the name of each shard.

Each shard object contains a single field remotes, which is an array of strings, each string representing the name of an existing remote.

Convenience fields

The shard objects in PATCH /network contain the additional fields addRemotes and removeRemotes meant for convenience:

// PATCH /network
{
  // assuming that remotes `ms-0`, `ms-1`, `ms-2` where sent in a previous call to PATCH /network
  "shards": {
    "s-a": { // new shard
      "remotes": ["ms-0", "ms-1"]
    }
  }
}

Remotes ms-0 and ms-1 own the new shard s-a.

// PATCH /network
{
  // assuming remotes `ms-0`, `ms-1`, `ms-2`
  // assuming shard `s-a`, owned by `ms-0` and `ms-1`
  "shards": {
    "s-a": {
      "remotes": ["ms-2"]
    }
  }
}

ms-2 is now the sole owner of s-a, replacing ms-0 and ms-1.

// PATCH /network
{
  // assuming remotes `ms-0`, `ms-1`, `ms-2`
  // assuming shard `s-a`, owned by `ms-2`
  "shards": {
    "s-a": {
      "addRemotes": ["ms-0"]
    }
  }
}

ms-0 and ms-2 are now the owners of s-a.

// PATCH /network
{
  // assuming remotes `ms-0`, `ms-1`, `ms-2`
  // assuming shard `s-a`, owned by `ms-0` and `ms-2`
  "shards": {
    "s-a": {
      "removeRemotes": ["ms-2"]
    }
  }
}

ms-0 is now the sole owner of s-a.

Set the shard to null:

// PATCH /network
{
  "shards": {
    "s-a": null
  }
}

Or set its remotes list to the empty list:

// PATCH /network
{
  "shards": {
    "s-a": {
      "remotes": []
    }
  }
}

network.shards validity

When network.leader is not null, each shard object in network.shards must:

  1. Only contain remotes that exist in the list of remotes.
  2. Contain at least one remote.

Additionally, network.shards must contain at least one shard.

Failure to meet any of these conditions will cause the PATCH /network route to respond with 400 invalid_network_shards.

Change in sharding logic

Documents are now sharded according to the list of shards declared in the network rather than the list of remotes. All remotes owning a shard will process the documents that belong to this shard, allowing for replication.

The following configuration defines 3 remotes 0, 1 and 2, and 3 shards A, B, C, such that each remote owns two shards, achieving replication (losing one remote does not lose any document).

{
  "leader": "0",
  "self": "0",
  "remotes": {
    "0": { /* .. */ },
    "1": { /* .. */ },
    "2": { /* .. */ }
  },
  "shards": {
    "A": {
      "remotes": ["0", "1"]
    },
    "B": {
      "remotes": ["1", "2"]
    },
    "C": {
      "remotes": ["2", "0"]
    }
  }
}

useNetwork takes network.shards into account

When useNetwork: true is passed to a search query, it is expanded to multiple queries such that each shard declared in network.shards appears exactly once, associated with a remote that owns that shard.

This ensures that there is no missing or duplicate documents in the results.

_shard filters

When the network experimental feature is enabled, then it becomes possible to filter documents depending on the shard they belong to.

Given s-a and s-b the names of two shards declared in network.shards, then:

You can use these new filters in manual remote federated search to create a partitioning over all shards in the network.

[!IMPORTANT] To avoid duplicate or missing documents in results, for manually crafted remote federated search requests, all shards should appear in exactly one query.

[!TIP] Search requests built with useNetwork: true already build a correct partitioning over shards. They should be preferred to manually crafted remote federated search requests in replicated sharding scenarios.

Update instructions

When updating your Meilisearch network using dumpless upgrade, please observe the following guidelines:

  1. Do not call the PATCH /network route until all remotes of the network are finished updating
  2. If using the search routes with useNetwork: true, call them on un-updated remotes. Calling it on already updated remotes will cause un-updated remotes to fail the search as they don't know about the _shard filters.

By @dureuill in https://github.com/meilisearch/meilisearch/pull/6128

§ Remove vectorStoreSetting experimental feature

The new HNSW vector store (hannoy) has been stabilized and is now the only supported vector store in Meilisearch.

As a result, updating to v1.37.0 will migrate all remaining legacy vector store indexes (using arroy) to hannoy, and the vectorStoreSetting experimental feature is no longer available.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6176

Improve indexing performance for embeddings

We removed a computationally expensive step from vector indexing.

On a DB with 20M documents, this removes 300s per indexing batch of 1100s.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6175

§ 🔒 Security

By @Strift and @curquiza in https://github.com/meilisearch/meilisearch/pull/6186 and https://github.com/meilisearch/meilisearch/pull/6172

§ 🔩 Miscellaneous

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.36.0...v1.37.0

download

unix · zip
curl -fL -o v1.37.0.zip https://ratatoskr.space/pkg/meilisearch/v1.37.0.zip
                    printf '%s  %s\n' 'c3fb5d4438da6a8ac96e8eaecd96010ef601f2aa17ec376b91ecfe486e60346c' 'v1.37.0.zip' | sha256sum -c -
windows · zip
$url = "https://ratatoskr.space/pkg/meilisearch/v1.37.0.zip"
$out = "v1.37.0.zip"
Invoke-WebRequest -Uri $url -OutFile $out
if ((Get-FileHash $out -Algorithm SHA256).Hash.ToLowerInvariant() -ne "c3fb5d4438da6a8ac96e8eaecd96010ef601f2aa17ec376b91ecfe486e60346c") { throw "sha256 mismatch" }
unix · tar.gz
curl -fL -o v1.37.0.tar.gz https://ratatoskr.space/pkg/meilisearch/v1.37.0.tar.gz
                    printf '%s  %s\n' '6dc44513f3c597a46dd17945e124350567201be8665a6203d3042d458d4b41cd' 'v1.37.0.tar.gz' | sha256sum -c -
windows · tar.gz
$url = "https://ratatoskr.space/pkg/meilisearch/v1.37.0.tar.gz"
$out = "v1.37.0.tar.gz"
Invoke-WebRequest -Uri $url -OutFile $out
if ((Get-FileHash $out -Algorithm SHA256).Hash.ToLowerInvariant() -ne "6dc44513f3c597a46dd17945e124350567201be8665a6203d3042d458d4b41cd") { throw "sha256 mismatch" }
download via yggdrasil mesh
unix · zip
curl -fL -o v1.37.0.zip http://14cc7d57b5e70f679b851fe5b272ce17c70632ff4beb5b35ab64bc706b2485af.pk.ygg/pkg/meilisearch/v1.37.0.zip
                    printf '%s  %s\n' 'c3fb5d4438da6a8ac96e8eaecd96010ef601f2aa17ec376b91ecfe486e60346c' 'v1.37.0.zip' | sha256sum -c -
windows · zip
$url = "http://14cc7d57b5e70f679b851fe5b272ce17c70632ff4beb5b35ab64bc706b2485af.pk.ygg/pkg/meilisearch/v1.37.0.zip"
$out = "v1.37.0.zip"
Invoke-WebRequest -Uri $url -OutFile $out
if ((Get-FileHash $out -Algorithm SHA256).Hash.ToLowerInvariant() -ne "c3fb5d4438da6a8ac96e8eaecd96010ef601f2aa17ec376b91ecfe486e60346c") { throw "sha256 mismatch" }
unix · tar.gz
curl -fL -o v1.37.0.tar.gz http://14cc7d57b5e70f679b851fe5b272ce17c70632ff4beb5b35ab64bc706b2485af.pk.ygg/pkg/meilisearch/v1.37.0.tar.gz
                    printf '%s  %s\n' '6dc44513f3c597a46dd17945e124350567201be8665a6203d3042d458d4b41cd' 'v1.37.0.tar.gz' | sha256sum -c -
windows · tar.gz
$url = "http://14cc7d57b5e70f679b851fe5b272ce17c70632ff4beb5b35ab64bc706b2485af.pk.ygg/pkg/meilisearch/v1.37.0.tar.gz"
$out = "v1.37.0.tar.gz"
Invoke-WebRequest -Uri $url -OutFile $out
if ((Get-FileHash $out -Algorithm SHA256).Hash.ToLowerInvariant() -ne "6dc44513f3c597a46dd17945e124350567201be8665a6203d3042d458d4b41cd") { throw "sha256 mismatch" }
artifact format size hashes
v1.37.0.zip zip 20.2 MiB
blake3-24 0d5124af1e18c617efa5804ca7a7ee0770569dc300fb1d19
sha256 c3fb5d4438da6a8ac96e8eaecd96010ef601f2aa17ec376b91ecfe486e60346c
sha1 17d243978066ef04178054ee6756a740d13dca66
v1.37.0.tar.gz tar.gz 19.2 MiB
blake3-24 8b18bac8f1800ff205cb6fffe8d3fc26ffac49a9695e2828
sha256 6dc44513f3c597a46dd17945e124350567201be8665a6203d3042d458d4b41cd
sha1 29bc0f074d0a8933423eb6715d7c28955a26642d

install

bazel
http_archive(
    name = "meilisearch",
    urls = ["https://ratatoskr.space/pkg/meilisearch/v1.37.0.tar.gz"],
    integrity = "sha256-bcRFE/PFl6Rt0XlF4SQ1BWcgG+hmWmID0wQtRY1LQc0=",
    strip_prefix = "meilisearch-v1.37.0",
)
zig
.url = "https://ratatoskr.space/pkg/meilisearch/v1.37.0.tar.gz",
install via yggdrasil mesh
bazel
http_archive(
    name = "meilisearch",
    urls = ["http://14cc7d57b5e70f679b851fe5b272ce17c70632ff4beb5b35ab64bc706b2485af.pk.ygg/pkg/meilisearch/v1.37.0.tar.gz"],
    integrity = "sha256-bcRFE/PFl6Rt0XlF4SQ1BWcgG+hmWmID0wQtRY1LQc0=",
    strip_prefix = "meilisearch-v1.37.0",
)
zig
.url = "http://14cc7d57b5e70f679b851fe5b272ce17c70632ff4beb5b35ab64bc706b2485af.pk.ygg/pkg/meilisearch/v1.37.0.tar.gz",
← v1.38.0v1.36.0 →