Meilisearch v1.4 improves the search result relevancy by introducing three new settings: nonSeparatorTokens, separatorTokens and dictionary.

🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available.

Some SDKs might not include all new features—consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️).

New features and improvements 🔥

Customize text separators

Meilisearch word segmentation uses a list of characters to separate one word from another. The nonSeparatorTokens and separatorTokens index settings allow you to configure this to better fit your dataset.

Add a character to the separator-tokens list to use it as a word separator:

curl \
  -X PUT 'http://localhost:7700/indexes/articles/settings/separator-tokens' \
  -H 'Content-Type: application/json'  \
  --data-binary '["§", "&sep"]'

Add a character to the non-separator-tokens list when you don't want Meilisearch to use it to separate words:

curl \
  -X PUT 'http://localhost:7700/indexes/articles/settings/non-separator-tokens' \
  -H 'Content-Type: application/json'  \
  --data-binary '["@", "#", "&"]'

Done by @ManyTheFish in #3946

Load user-defined dictionaries

Meilisearch word segmentation also relies on language-based dictionaries to segment words. Use the dictionary index setting to expand the default dictionaries and improve accuracy when working with datasets using domain-specific terms:

curl \
  -X PUT 'http://localhost:7700/indexes/articles/settings/dictionary' \
  -H 'Content-Type: application/json'  \
  --data-binary '["J. R. R.", "J.R.R."]'

This feature can be used together with the stopWords and synonyms index settings:

{
    "dictionary": ["J. R. R.", "J.R.R."],
    "synonyms": {
            "J.R.R.": ["jrr", "J. R. R."],
            "J. R. R.": ["jrr", "J.R.R."],
            "jrr": ["J.R.R.", "J. R. R."],
    }
}

Done by @ManyTheFish in #3946

Other improvements

Meilisearch better hides your data and no longer shows hidden document's fields in error messages. In case you are trying to sort by using a field that is not sortable but there are some of them that are not displayed either, you will see a message like the following:

Available sortable attributes are: price, stock, <..hidden-attributes>.

Done by @vivek-26 in #3945

Fixes 🐞

• Fix synonyms display (#3946) @ManyTheFish
• Fix thai synonyms (#4033) @Kerollmops
• Fix synonyms with separators (#3994) @ManyTheFish
• Fix the swap index tasks (#4041) @irevoire
• Fix highlighting bug when searching for a phrase with cropping (#4028) @vivek-26
• Empty arrays/objects now return empty instead of null (#3997) @dogukanakkaya
• Fix the stats of the documents deletion by filter (#4053) @irevoire
• \ can now be used at the end of the filter search parameter values (#4038 and #4043) @Kerollmops

🔴 Usage breaking change following filter bug fix

⚠️ This section only concerns users using the filter search parameter with \ characters in the filter expression.

In this version, we fixed a bug regarding \ in the filter search parameter expression. Before v1.4.0, the users were not able to express all the filter expression they wanted.

For example, if you add the following documents:

[
  {
    "id": 1,
    "path": "my\\test\\path"
  },
  {
    "id": 2,
    "path": "my\\test\\path\\"
  }
]

(\\ escaping here is about JSON escaping)

Before v1.4.0, if you wanted to filter on document 2, the filters path = "my\\test\\path\\" and path = "my\\test\\path\\\" led to an error.

Now, in v1.4.0, you can express all the possible filter expressions, no matter if the filter contains \ or not. But to reach this, you have to escape all the \ characters in your filter.

Following our example, to finally succeed in filtering on document 2, the filter should be: path = "my\\\\test\\\\path\\\\".

👉 If you come from v1.3.X or before, and you used \ in filter, here is the change to apply: following our example, to filter on document 1, before v1.4.0, you could apply the filter path = "my\\test\\path". Now, from v1.4.0, you have to escape the \ and apply the following filter: path = "my\\\\test\\\\path". Two escapings are applied: the JSON escaping and the Meilisearch filter escaping. The JSON unescaping transforms the \\\\ into \\ and the Meilisearch filter unescaping ends up with a single \.

If you need any help with this change, please feel free to reach us on our Discord.

Misc

• Upgrade Meilisearch dependencies (#3987) @ManyTheFish
• Upgrade CI dependencies (#3871, #3969 and #3968)
• Fix CI benchmarks (#3963) @irevoire
• Removed borrow call causing failed nightly tests (#3990) @JannisK89
• Expose Puffin server to profile the indexing process (#3913) @Kerollmops
• Update and fix the CI Test Suite (#3918) @Kerollmops
• Improve CI test suite for manual trigger events (#3989) @curquiza
• Update README.md (#3895 and #3932) @ferdi05 & @Strift
• Fix Homebrew CI (#4016) @Kerollmops

❤️ Thanks again to our external contributors:

• Meilisearch: @dogukanakkaya, @JannisK89, and @vivek-26.