✨ Enhancement

Support search fallback on remote unavailability

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6306

We introduce a new fallback system for the sharding and replication enterprise edition feature, along with a way to determine which remote is available. The engine can avoid machines that are unavailable for a period and resume querying them once they're back online.

The following snippet shows what the /network route looks like now that this PR exposes the remote statuses/availabilities.

"remotes": {
	"prod2": {
	  "url": "http://localhost:7702",
	  "searchApiKey": "mykey",
	  "writeApiKey": "mykey",
	  "status": "available"
	},
	"prod3": {
	  "url": "http://localhost:7703",
	  "searchApiKey": "mykey",
	  "writeApiKey": "mykey",
	  "status": "unavailable"
	}
}

🔬 Experimental: Document join Filtering

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6314

This enhancement extends the Cross-index document hydration introduced in v1.39.0 by allowing the user to filter on the foreign indexes to retrieve the documents.

📓 Note: This implementation doesn't support a remote sharding environment

foreignKeys experimental feature

TheforeignKeys experimental feature must be activated to use the foreign filters:

curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
  -H 'Content-Type: application/json' \
  --data-binary '{"foreignKeys": true}'

foreignKeys + filter index setting

To be able to use the foreign filters, the related field must be set as a foreignKey and as a filterableAttribute in /indexes/{index_uid}/settings:

{
	// new setting, an array of foreign keys that allows multiple foreign relationships between indexes
	"foreignKeys":  [
		{
			// the path in the JSON document containing foreign document ids
			"fieldName": "actors",
			
			// the UID of the foreign index containing the documents to fetch during hydration
	        "foreignIndexUid": "actors"
		}
	],
	// the actors field must be filterable on equality
	"filterableAttributes": [
		{
	    	"attributePatterns": ["actors"],
	     	"features": {
				"facetSearch": false,
				"filter": {
					"equality": true,
					"comparison": false
				}
			}
		}
	]
}

filtering using the _foreign filter

On the search route, a new _foreign verb has been introduced and should be used as follows:

{
	"q": "<query>",

	// filters on the movie index:
	// genres = action
    // AND
    // the foreign documents from the actor index match:  birthday STARTS WITH \"1958-\" AND popularity >= 3.5
	"filter": "genres = action AND _foreign(actors, birthday STARTS WITH \"1958-\" AND popularity >= 3.5)"
}

Note: nesting foreign filters is not supported and will return an error

Example of usage

Prerequisites

• Meilisearch running on 127.0.0.1:7700 on the document-join-hydration branch.

Step 1: Enable Foreign Keys Feature

curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
  -H 'Content-Type: application/json' \
  --data-binary '{"foreignKeys": true}'

Step 2: Create Indexes

Create the actors index

curl -X POST 'http://127.0.0.1:7700/indexes' \
  -H 'Content-Type: application/json' \
  --data-binary '{"uid": "actors", "primaryKey": "id"}'

Create the movies index

curl -X POST 'http://127.0.0.1:7700/indexes' \
  -H 'Content-Type: application/json' \
  --data-binary '{"uid": "movies", "primaryKey": "id"}'

Step 3: Add Documents to the actors Index

curl -X POST 'http://127.0.0.1:7700/indexes/actors/documents' \
  -H 'Content-Type: application/json' \
  --data-binary '[
  {"id": 1, "name": "Tom", "familyName": "Hanks", "birthDate": "1956-07-09"},
  {"id": 2, "name": "Meryl", "familyName": "Streep", "birthDate": "1949-06-22"},
  {"id": 3, "name": "Leonardo", "familyName": "DiCaprio", "birthDate": "1974-11-11"},
  {"id": 4, "name": "Emma", "familyName": "Watson", "birthDate": "1990-04-15"}
]'

Step 4: Add Documents to the movies Index

curl -X POST 'http://127.0.0.1:7700/indexes/movies/documents' \
  -H 'Content-Type: application/json' \
  --data-binary '[
  {"id": 1, "title": "Forrest Gump", "description": "The presidencies of Kennedy and Johnson, the Vietnam War, the Watergate scandal and other historical events unfold from the perspective of an Alabama man with an IQ of 75.", "actors": [1]},
  {"id": 2, "title": "The Devil Wears Prada", "description": "A smart but sensible new graduate lands a job as an assistant to Miranda Priestly, the demanding editor-in-chief of a high fashion magazine.", "actors": [2, 4]},
  {"id": 3, "title": "Inception", "description": "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", "actors": [3]},
  {"id": 4, "title": "Cast Away", "description": "A FedEx executive undergoes a physical and emotional transformation after crash landing on a deserted island.", "actors": [1]}
]'

Step 5: Configure Foreign Keys on the movies Index

curl -X PATCH 'http://127.0.0.1:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{"foreignKeys": [{"fieldName": "actors", "foreignIndexUid": "actors"}], "filterableAttributes": [{"attributePatterns": ["actors"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}'

Step 6: Configure filterable on the actors Index

curl -X PATCH 'http://127.0.0.1:7700/indexes/actors/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{"filterableAttributes": [{"attributePatterns": ["birthDate"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}'

Step 7: Perform a Federated Search

curl -X POST 'http://127.0.0.1:7700/multi-search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
  "queries": [
    {
      "indexUid": "movies",
      "q": "Forrest",
      "filter": "_foreign(actors, birthDate = \"1956-07-09\")"
    }
  ],
  "federation": {
    "limit": 20,
    "offset": 0
  }
}'

Expected Result

The federated search should return movie documents with the actors array automatically hydrated with full actor objects instead of just IDs:

{
  "hits": [
    {
      "id": 1,
      "title": "Forrest Gump",
      "description": "...",
      "actors": [
        {
          "id": 1,
          "name": "Tom",
          "familyName": "Hanks",
          "birthDate": "1956-07-09"
        }
      ],
      "_federation": {
        "indexUid": "movies",
        "queriesPosition": 0,
        "weightedRankingScore": 0.9848484848484849
      }
    }
  ],
  "processingTimeMs": 208,
  "limit": 20,
  "offset": 0,
  "estimatedTotalHits": 1
}

🪲 Bug fixes

• Fix a race condition when writing network by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6300

  We fixed a race condition in network topology changes that could cause errors and prevent documents from being correctly indexed. Additionally, we fixed a bug in the networkTopologyChange task batching that was causing it to batch too many task types. We made sure it only batches import tasks, and only those, to avoid out-of-order task processing.

• Throw document template errors when updating the chat settings by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6321

  We fixed an issue that prevented the engine from explicitly showing the possible document template errors users could encounter when updating the template in the chat settings. The engine now correctly checks for and throws template errors when they are detected.

• Fix: Update Index tasks will be properly forwarded to remote nodes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6299

• Fix action mistake on the chat completions route by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6290

🔩 Miscellaneous

• Use the latest version of heed with nested rtxns support by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6316

  This PR bumps the versions of crates that use heed to the latest version, v0.22.1. This version finally stabilized a long-standing piece of work we were doing with Howard Chu: nested read transactions. We no longer have to rely on unstable pre-releases, but rather on a clean, stable version of LMDB (still a fork, but a better one).

• Add section to CONTRIBUTING.md to bump mini-dashboard version and testing section to right place by @curquiza in https://github.com/meilisearch/meilisearch/pull/6195

• Make the no-agent AGENTS.ms more permissive by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6260

• Remove deleted test commands by @Strift in https://github.com/meilisearch/meilisearch/pull/6283

• Fix OpenAPI schema generation for chat completions route by @qdequele in https://github.com/meilisearch/meilisearch/pull/6274

• Rename OpenAPI route names for search rules and compact by @qdequele in https://github.com/meilisearch/meilisearch/pull/6298

• Update README with new features and demos by @qdequele in https://github.com/meilisearch/meilisearch/pull/6297

• Prevent shell injection in benchmark workflows by @curquiza & @Kerollmopsin https://github.com/meilisearch/meilisearch/pull/6308 & https://github.com/meilisearch/meilisearch/pull/6318

• Rename some of the search performance traces by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6323