# Laravel-Elasticsearch
> An Elasticsearch implementation of Laravel's Eloquent ORM. Extends Laravel's Eloquent model and query builder with seamless Elasticsearch integration — search queries, aggregations, geo-spatial queries, nested objects, and more — using familiar Eloquent syntax.
- Package: pdphilip/elasticsearch v5 for Laravel 10, 11 & 12 with Elasticsearch 8.x
- Docs: https://elasticsearch.pdphilip.com
- GitHub: https://github.com/pdphilip/laravel-elasticsearch
- Last updated: 2026-02-12
---
## Getting Started
### An Elasticsearch Implementation of Laravel's Eloquent ORM
This package seamlessly integrates Elasticsearch functionalities into
Laravel's Eloquent model and query builder, making it feel native to Laravel.
This enables you to utilize Eloquent models while leveraging the powerful
search and analytics capabilities of Elasticsearch.
##### Interested in the [OpenSearch](https://opensearch.pdphilip.com/) version of this package? Visit [Github](https://github.com/pdphilip/laravel-opensearch).
---
> **Caution: Upgrading from v4 to v5?**
> Check out the [upgrade guide](https://elasticsearch.pdphilip.com/upgrade-guide/) for a smooth transition.
## Installation - 5
### Laravel 10/11/12
- Elasticsearch 8.x
```bash
composer require pdphilip/elasticsearch
```
## Configuration Guide
1. Set up your `.env` with the following Elasticsearch settings:
```ini
ES_AUTH_TYPE=http
ES_HOSTS="http://localhost:9200"
ES_USERNAME=
ES_PASSWORD=
ES_CLOUD_ID=
ES_API_ID=
ES_API_KEY=
ES_SSL_CA=
ES_INDEX_PREFIX=my_app_
# prefix will be added to all indexes created by the package with an underscore
# ex: my_app_user_logs for UserLog.php model
ES_SSL_CERT=
ES_SSL_CERT_PASSWORD=
ES_SSL_KEY=
ES_SSL_KEY_PASSWORD=
# Options
ES_OPT_ID_SORTABLE=false
ES_OPT_VERIFY_SSL=true
ES_OPT_RETRIES=
ES_OPT_META_HEADERS=true
ES_ERROR_INDEX=
ES_OPT_BYPASS_MAP_VALIDATION=false
ES_OPT_DEFAULT_LIMIT=1000
```
For multiple nodes, pass in as comma-separated:
```ini
ES_HOSTS="http://es01:9200,http://es02:9200,http://es03:9200"
```
**Example cloud config .env: (Click to expand)**
```ini
ES_AUTH_TYPE=cloud
ES_HOSTS="https://xxxxx-xxxxxx.es.europe-west1.gcp.cloud.es.io:9243"
ES_USERNAME=elastic
ES_PASSWORD=XXXXXXXXXXXXXXXXXXXX
ES_CLOUD_ID=XXXXX:ZXVyb3BlLXdl.........SQwYzM1YzU5ODI5MTE0NjQ3YmEyNDZlYWUzOGNkN2Q1Yg==
ES_API_ID=
ES_API_KEY=
ES_SSL_CA=
ES_INDEX_PREFIX=my_app_
ES_ERROR_INDEX=
```
2. In `config/database.php`, add the elasticsearch connection:
```php
'elasticsearch' => [
'driver' => 'elasticsearch',
'auth_type' => env('ES_AUTH_TYPE', 'http'), //http or cloud
'hosts' => explode(',', env('ES_HOSTS', 'http://localhost:9200')),
'username' => env('ES_USERNAME', ''),
'password' => env('ES_PASSWORD', ''),
'cloud_id' => env('ES_CLOUD_ID', ''),
'api_id' => env('ES_API_ID', ''),
'api_key' => env('ES_API_KEY', ''),
'ssl_cert' => env('ES_SSL_CA', ''),
'ssl' => [
'cert' => env('ES_SSL_CERT', ''),
'cert_password' => env('ES_SSL_CERT_PASSWORD', ''),
'key' => env('ES_SSL_KEY', ''),
'key_password' => env('ES_SSL_KEY_PASSWORD', ''),
],
'index_prefix' => env('ES_INDEX_PREFIX', false),
'options' => [
'track_total_hits' => env('ES_TRACK_TOTAL_HITS', null),
'bypass_map_validation' => env('ES_OPT_BYPASS_MAP_VALIDATION', false),
'logging' => env('ES_OPT_LOGGING', false),
'ssl_verification' => env('ES_OPT_VERIFY_SSL', true),
'retires' => env('ES_OPT_RETRIES', null),
'meta_header' => env('ES_OPT_META_HEADERS', true),
'default_limit' => env('ES_OPT_DEFAULT_LIMIT', 1000),
'allow_id_sort' => env('ES_OPT_ID_SORTABLE', false),
],
],
```
3. If packages are not autoloaded, add the service provider:
For **Laravel 10 and below**:
```php
//config/app.php
'providers' => [
...
...
PDPhilip\Elasticsearch\ElasticServiceProvider::class,
...
```
For **Laravel 11 & 12**:
```php
//bootstrap/providers.php
=', Carbon::now()->subDays(30))
->with('user')
->orderByDesc('_count')
->select('user_id')
->distinct(true);
```
Why: You can now treat distinct aggregations like real Eloquent results, including relationships.
### Bulk Distinct Queries
New query method `bulkDistinct(array $fields, $includeDocCount = false)` - [Docs](https://elasticsearch.pdphilip.com/eloquent/distinct/#bulk-distinct)
Run multiple distinct aggregations **in parallel** within a single Elasticsearch query.
```php
$top3 = UserSession::where('created_at', '>=', Carbon::now()->subDays(30))
->limit(3)
->bulkDistinct(['country', 'device', 'browser_name'], true);
```
Why: Massive performance gains vs running sequential distinct queries.
### Group By Ranges
`groupByRanges()` performs a [range aggregation](https://www.elastic.co/docs/reference/aggregations/search-aggregations-bucket-range-aggregation) on the specified field. - [Docs](https://elasticsearch.pdphilip.com/eloquent/distinct/#groupby-ranges)
`groupByRanges()->get()` — return bucketed results - [Docs](https://elasticsearch.pdphilip.com/eloquent/distinct/#groupby-ranges)
`groupByRanges()->agg()` - apply metric aggregations per bucket -[Docs](https://elasticsearch.pdphilip.com/eloquent/distinct/#groupby-ranges-with-aggregations)
### Group By Date Ranges
`groupByDateRanges()` performs a [date range aggregation](https://www.elastic.co/docs/reference/aggregations/search-aggregations-bucket-daterange-aggregation) on the specified field. - [Docs](https://elasticsearch.pdphilip.com/eloquent/distinct/#groupby-date-ranges)
`groupByDateRanges()->get()` — bucketed date ranges
`groupByDateRanges()->agg()` — metrics per date bucket
### Model Meta Accessor
New model method `getMetaValue($key)` - [Docs](https://elasticsearch.pdphilip.com/eloquent/the-base-model/#get-model-meta-value)
Convenience method to get a specific meta value from the model instance.
```php
$product = Product::where('color', 'green')->first();
$score = $product->getMetaValue('score');
```
### Bucket Values in Meta
When a bucketed query is executed, the raw bucket data is now stored in model meta. -[Docs](https://elasticsearch.pdphilip.com/eloquent/distinct/#raw-bucket-values-from-meta)
```php
$products = Product::distinct('price');
$buckets = $products->map(function ($product) {
return $product->getMetaValue('bucket');
});
```
## 5.2 New feature
### Query String Queries [Docs](https://elasticsearch.pdphilip.com/eloquent/query-string-queries)
- New query methods for advanced full-text and structured searches using Elasticsearch's `query_string` syntax.
- Method: `searchQueryString($query, $fields = null, $options = [])` - [Docs](https://elasticsearch.pdphilip.com/eloquent/query-string-queries#search-query-string)
## 5.1 New features
### Track Total Hits
New query method `withTrackTotalHits(bool|int|null $val = true)` - [Docs](https://elasticsearch.pdphilip.com/eloquent/the-base-model/#track-total-hits)
Appends the `track_total_hits` parameter to the DSL query, setting value to true will count all the hits embedded in the query meta not capping to Elasticsearch default of 10k hits
### Create or Fail
New query model `method createOrFail(array $attributes)` - [Docs](https://elasticsearch.pdphilip.com/eloquent/saving-models/#create-or-fail)
- By default, when using create($attributes) where $attributes has an id that exists, the operation will upsert. createOrFail will throw a BulkInsertQueryException with status code 409 if the id exists
### Set refresh flag
- New query method `withRefresh(bool|string $refresh)` - [Docs](https://elasticsearch.pdphilip.com/eloquent/saving-models/#with-refresh)
By default, inserting documents will wait for the shards to refresh, ie: `withRefresh(true)`, you can set the refresh flag with the following (as per ES docs):
1. `true` (default): Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately.
2. `wait_for`: Wait for the changes made by the request to be made visible by a refresh before replying. This doesn’t force an immediate refresh, rather, it waits for a refresh to happen.
3. `false`: Take no refresh-related actions. The changes made by this request will be made visible at some point after the request returns.
## 5.0 New features
### Laravel generated IDs:
- Ids can be generated on the Laravel side by using the `GeneratesUuids` or the `GeneratesElasticIds` trait. [Docs](https://elasticsearch.pdphilip.com/eloquent/the-base-model/#laravel-vs-elasticsearch-generated-ids)
### Query field map
- Field Maps can be defined in the model to map text to keyword fields, boosting performance [Docs](https://elasticsearch.pdphilip.com/eloquent/the-base-model/#query-field-map)
### Fluent query options as a callback
- All clauses in the query builder now accept an optional callback of Elasticsearch options to be applied to the clause. [Docs](https://elasticsearch.pdphilip.com/eloquent/eloquent-queries/#query-options)
### BelongsToMany Relationships
- `BelongsToMany` relationships are now supported. [Docs](https://elasticsearch.pdphilip.com/relationships/es-es/#many-to-many-belongstomany)
### New queries
Three powerful new query methods have been added to simplify expressive search:
- whereMatch() [Docs](https://elasticsearch.pdphilip.com/eloquent/es-queries/#where-match)
- whereFuzzy() [Docs](https://elasticsearch.pdphilip.com/eloquent/es-queries/#where-fuzzy)
- whereScript() [Docs](https://elasticsearch.pdphilip.com/eloquent/es-queries/#where-script)
### New aggregations
- Boxplot Aggregations [Docs](https://elasticsearch.pdphilip.com/eloquent/aggregation/#boxplot-aggregations)
- Stats Aggregations [Docs](https://elasticsearch.pdphilip.com/eloquent/aggregation/#stats-aggregations)
- Extended Stats Aggregations - [Docs](https://elasticsearch.pdphilip.com/eloquent/aggregation/#extended-stats-aggregations)
- Cardinality Aggregations - [Docs](https://elasticsearch.pdphilip.com/eloquent/aggregation/#cardinality-aggregations)
- Median Absolute Deviation Aggregations - [Docs](https://elasticsearch.pdphilip.com/eloquent/aggregation/#median-absolute-deviation-aggregations)
- Percentiles Aggregations - [Docs](https://elasticsearch.pdphilip.com/eloquent/aggregation/#percentiles-aggregations)
- String Stats Aggregations - [Docs](https://elasticsearch.pdphilip.com/eloquent/aggregation/#string-stats-aggregations)
### Migrations: Add Normalizer
- Normalizers can now be defined in migrations. [Docs](https://elasticsearch.pdphilip.com/schema/index-blueprint/#addnormalizer)
### Direct Access to Elasticsearch PHP client
For low-level operations, you can access the native Elasticsearch PHP client directly:
```php
Connection::on('elasticsearch')->elastic()->{clientMethod}();
```
## V5 Under the Hood
This release is a complete architectural overhaul designed to mirror Laravel’s Eloquent ORM more closely.
The internals have been refactored for better testability, performance, and future extensibility.
[See diagram in the online documentation]
---
## Upgrading From v4
## Breaking Changes
### Connection
**Update Index Prefix**
Index Prefix **no longer auto concatenates with `_`**
```ini
- ES_INDEX_PREFIX=my_prefix
ES_INDEX_PREFIX=my_prefix_
```
### Models
**Model id Field**
`$model->_id` is now `$model->id` and is an alias for Elasticsearch's `_id` field. This is to maintain consistency with Laravel's Eloquent ORM.
If you were using a distinct `id` field alongside the default `_id` field then **this will no longer work.** You will need to update your code to use a different field name for your previous `id` field.
> **Note:**
> `$model->_id` will still work but it is recommended to use `$model->id` for consistency.
**MAX_SIZE → $defaultLimit**
`const MAX_SIZE` has been replaced with `protected $defaultLimit` for defining the default limit for search results.
```php
use PDPhilip\Elasticsearch\Eloquent\Model;
class Product extends Model
{
- const MAX_SIZE = 10000;
+ protected $defaultLimit = 10000;
protected $connection = 'elasticsearch';
```
### Queries
**where() clause now runs a term query**
The `where()` clause now runs a term query, not a match as before.
> **Caution: May return unexpected results**
> - In v4 `where('name', 'John')` would run a **match query**, and match all documents where the `name` field contains `John` as a substring.
> - In v5 `where('name', 'John')` will run a **term query**, and match all documents where the `name` field is exactly `John`.
> If you still want to run a match query, then replace `where()` with `whereMatch()`
```php
- where('name', 'John')
whereMatch('name', 'John')
```
> If exact matching is a better fit, then ensure the field has a `keyword` mapping.
```php
$index->keyword('name'); //or
$index->text('name', hasKeyword: true);
```
**orderByRandom() → functionScore()**
`orderByRandom()` method has been removed. Use `functionScore()` instead.
```php
- Product::where('orders', '>', 2)->orderByRandom('created_at',rand(1, 100)')->get();
Product::functionScore('random_score', ['seed' => rand(1, 100), 'field' => 'created_at'], function ($query) {
$query->where('orders', '>', 2);
})->get();
```
**Full text search options replaced**
`asFuzzy()`, `setMinShouldMatch()`, `setBoost()` search methods been removed. Use `$options` instead.
```php
- Product::searchTerm('espreso tme')
- ->asFuzzy()->setMinShouldMatch(2)->setBoost(2)
- ->get();
Product::searchTerm('espreso tme', function (SearchOptions $options) {
$options->searchFuzzy();
$options->boost(2);
$options->minimumShouldMatch(2);
})->get();
```
**Legacy search query builder methods removed**
All `{xx}->search()` methods have been removed. Use `{multi_match}->get()` instead.
```php
- Book::term('Eric')->orTerm('Lean')->andTerm('Startup')->search();
Book::searchTerm('Eric')->orSearchTerm('Lean')->searchTerm('Startup')->get();
```
```php
- Book::phrase('United States')->orPhrase('United Kingdom')->search();
Book::searchPhrase('United States')->orSearchPhrase('United Kingdom')->get();
```
```php
- $results = Book::term('Eric')->fields(['title', 'author', 'description'])->search();
Book::searchTerm('Eric',['title', 'author', 'description'])->get();
```
```php
- $results = Book::fuzzyTerm('quikc')->orFuzzyTerm('brwn')->andFuzzyTerm('foks')->search();
Book::searchTerm('quikc',['fuzziness' => 'AUTO'])
->orSearchTerm('brwn',['fuzziness' => 'AUTO'])
->searchTerm('foks',['fuzziness' => 'AUTO'])
->get();
```
### Distinct & GroupBy
**distinct() and groupBy() operate differently**
The distinct() and groupBy() methods have been rebuilt and work differently.
- `distinct()` uses: Nested Term Aggs
- Returns full results, and cannot paginate
- Sort on column names and by doc_count (useful for ordering by most aggregated values)
- `groupBy()` uses: Composite Aggregation
- Can paginate
- Sort on column names only
Important: `distinct()` now executes the aggregation:
```php
- UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
- ->distinct()->get('user_id');
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->distinct('user_id');
```
Please review the [Distinct & GroupBy](https://elasticsearch.pdphilip.com/eloquent/distinct/) documentation carefully for more information.
### Schema: Migration methods
**IndexBlueprint/AnalyzerBlueprint → Blueprint**
`IndexBlueprint` and `AnalyzerBlueprint` have been removed and replaced with a single `Blueprint` class
```php
- use PDPhilip\Elasticsearch\Schema\IndexBlueprint;
- use PDPhilip\Elasticsearch\Schema\AnalyzerBlueprint;
use PDPhilip\Elasticsearch\Schema\Blueprint;
```
### Schema: Builder methods
**Schema::hasIndex → Schema::indexExists**
`Schema::hasIndex` has been removed. Use `Schema::hasTable` or `Schema::indexExists` instead.
```php
- Schema::hasIndex('index_name');
Schema::hasTable('index_name'); //or
Schema::indexExists('index_name');
```
**Schema::dsl → Schema::indices()**
`Schema::dsl($method,$dsl)` has been removed. Use `Schema::indices()->{method}`
```php
- Schema::dsl('close', ['index' => 'my_index']);
Schema::indices()->close(['index' => 'my_index']);
```
> **Note:**
> `Schema::indices()` is an instance the Elasticsearch PHP client's `indices()` endpoint.
### Schema: Blueprint index methods
> **Note: Reminder: IndexBlueprint is now Blueprint**
> ```php
> - use PDPhilip\Elasticsearch\Schema\IndexBlueprint;
> use PDPhilip\Elasticsearch\Schema\Blueprint;
> ```
1. `geo($field)` field property has been replaced with `geoPoint($field)`;
```php
- $index->geo('last_login');
$index->geoPoint('last_login');
```
2. `->index($bool)` field property has been replaced with `->indexField($bool)`;
```php
// Remove from index, can't search by this field but can still use for aggregations:
- $index->integer('age')->index(false);
$index->integer('age')->indexField(false);
```
3. `alias()` field type has been removed. Use `aliasField()` instead.
```php
- $index->alias('comments', 'notes');
$index->aliasField('comments', 'notes');
```
4. `settings()` method has been replaced with `withSetting()`
```php
- $index->settings('number_of_shards', 3);
$index->withSetting('number_of_shards', 3);
```
5. `map()` method has been replaced with `withMapping()`
```php
- $index->map('date_detection', false);
$index->withMapping('date_detection', false);
```
### Schema: Blueprint analyser methods
> **Note: Reminder: AnalyzerBlueprint is now Blueprint**
> ```php
> - use PDPhilip\Elasticsearch\Schema\AnalyzerBlueprint;
> use PDPhilip\Elasticsearch\Schema\Blueprint;
> ```
1. `analyzer()` method has been replaced with `addAnalyzer()`
```php
- $settings->analyzer('my_custom_analyzer')
$settings->addAnalyzer('my_custom_analyzer')
```
2. `tokenizer()` method has been replaced with `addTokenizer()`
```php
- $settings->tokenizer('punctuation')
$settings->addTokenizer('punctuation')
```
3. `charFilter()` method has been replaced with `addCharFilter()`
```php
- $settings->charFilter('emoticons')
$settings->addCharFilter('emoticons')
```
4. `filter()` method has been replaced with `addFilter()`
```php
//Custom Field Mapping
- $settings->filter('english_stop')
$index->addFilter('english_stop')
```
### Dynamic Indices
1. Dynamic indices are now managed by the `DynamicIndex` trait.
```php
namespace App\Models;
+use PDPhilip\Elasticsearch\Eloquent\DynamicIndex;
use PDPhilip\Elasticsearch\Eloquent\Model
class PageHit extends Model
{
+ use DynamicIndex;
protected $connection = 'elasticsearch';
- protected $index = 'page_hits_*'; // Dynamic index pattern
```
2. Replace `setIndex()` with `setSuffix()` in the context of saving a record.
```php
$pageHit = new PageHit;
$pageHit->page_id = 4;
$pageHit->ip = $someIp;
// Set the specific index for the new record
- $pageHit->setIndex('page_hits_' . date('Y-m-d'));
+ $pageHit->setSuffix('_' . date('Y-m-d'));
$pageHit->save();
```
3. `getRecordIndex()` method has been replaced with `getRecordSuffix()`.
```php
$pageHit = PageHit::first();
- $index = $pageHit->getRecordIndex();
+ $suffix = $pageHit->getRecordSuffix();
2. Replace `setIndex()` with `withSuffix()` in the context of querying indexes (within the given suffix).
```php
- $model = new PageHit;
- $model->setIndex('page_hits_2023-01-01');
- $pageHits = $model->where('page_id', 3)->get();
+ $pageHits = PageHit::withSuffix('_2023-01-01')->where('page_id', 3)->get();
```
## Medium impact changes
### Result structure changes
**Multiple Aggs are now returned as a flat array**
```php
Product::agg(['avg','sum'],'orders');
```
**v4:**
```json
{
"avg_orders": {
"value": 2.5
},
"sum_orders": {
"value": 5
}
}
```
**v5:**
```json
{
"avg_orders": 2.5,
"sum_orders": 5
}
```
**Matix aggs results differ**
```php
Product::matrix(['orders','status']);
```
**v4:**
```json
{
"doc_count": 100,
"fields": [
{
"name": "orders",
"count": 100,
"mean": 134.69999999999996,
"variance": 5427.323232323232,
"skewness": -0.0660090381451759,
"kurtosis": 1.7977192917815517,
"covariance": {
"orders": 5427.323232323232,
"status": -2.039393939393942
},
"correlation": {
"orders": 1,
"status": -0.010736843748959933
}
},
{
"name": "status",
"count": 100,
"mean": 5.169999999999999,
"variance": 6.647575757575759,
"skewness": -0.1108382470107292,
"kurtosis": 1.829112534153634,
"covariance": {
"orders": -2.039393939393942,
"status": 6.647575757575759
},
"correlation": {
"orders": -0.010736843748959933,
"status": 1
}
}
]
}
```
**v5:**
```json
{
"matrix_stats_orders": {
"name": "orders",
"count": 100,
"mean": 118.60000000000001,
"variance": 5258.121212121213,
"skewness": -0.016424308077811454,
"kurtosis": 1.797729283639266,
"covariance": {
"orders": 5258.121212121213,
"status": -22.759595959595952
},
"correlation": {
"orders": 1,
"status": -0.12227092184902529
}
},
"matrix_stats_status": {
"name": "status",
"count": 100,
"mean": 4.42,
"variance": 6.589494949494949,
"skewness": 0.11800255330047148,
"kurtosis": 1.806059741732609,
"covariance": {
"orders": -22.759595959595952,
"status": 6.589494949494949
},
"correlation": {
"orders": -0.12227092184902529,
"status": 1
}
}
}
```
## Upgrading
**withoutRefresh()**
```php
- $product->saveWithoutRefresh()
$product->withoutRefresh()->save()
- Product::createWithoutRefresh([data])
Product::withoutRefresh()->create([data])
```
## Low impact changes
### Connection
##### 1. **Database connection schema**: Update as following:
```php
'elasticsearch' => [
'driver' => 'elasticsearch',
'auth_type' => env('ES_AUTH_TYPE', 'http'), //http or cloud
'hosts' => explode(',', env('ES_HOSTS', 'http://localhost:9200')),
'username' => env('ES_USERNAME', ''),
'password' => env('ES_PASSWORD', ''),
'cloud_id' => env('ES_CLOUD_ID', ''),
'api_id' => env('ES_API_ID', ''),
'api_key' => env('ES_API_KEY', ''),
'ssl_cert' => env('ES_SSL_CA', ''),
'ssl' => [
'cert' => env('ES_SSL_CERT', ''),
'cert_password' => env('ES_SSL_CERT_PASSWORD', ''),
'key' => env('ES_SSL_KEY', ''),
'key_password' => env('ES_SSL_KEY_PASSWORD', ''),
],
'index_prefix' => env('ES_INDEX_PREFIX', false),
'options' => [
+ 'track_total_hits' => env('ES_TRACK_TOTAL_HITS', null),
'bypass_map_validation' => env('ES_OPT_BYPASS_MAP_VALIDATION', false),
'logging' => env('ES_OPT_LOGGING', false),
'ssl_verification' => env('ES_OPT_VERIFY_SSL', true),
'retires' => env('ES_OPT_RETRIES', null),
'meta_header' => env('ES_OPT_META_HEADERS', true),
+ 'default_limit' => env('ES_OPT_DEFAULT_LIMIT', 1000),
'allow_id_sort' => env('ES_OPT_ID_SORTABLE', false),
],
],
```
## Deprecated
---
## Migrations
Elasticsearch index management differs significantly from traditional SQL schema operations. This package provides a fully custom schema layer tailored to Elasticsearch’s structure and capabilities.
---
## Migration Class Creation
To use migrations for index management, you can create a standard Laravel migration class. However, it's crucial to note that the `up()` and `down()` methods encapsulate the specific Elasticsearch-related operations, namely:
- **Schema Management**: Utilizes the `PDPhilip\Elasticsearch\Schema\Schema` class.
- **Index/Analyzer Blueprint Definition**: Leverages the `PDPhilip\Elasticsearch\Schema\Blueprint` class for defining index & analyser structures.
### Full example
```php
text('first_name')->copyTo('full_name');
$index->text('last_name')->copyTo('full_name');
$index->text('full_name');
//Multiple types => Order matters ::
// Top-level `email` will be indexed as a text field
// Subfield `email.keyword` can be used for sorting in queries (e.g., ->orderBy('email.keyword'))
$index->text('email');
$index->keyword('email');
//Dates have an optional formatting as second parameter
$index->date('first_contact', 'epoch_second');
//Nested properties (optional properties callback to define nested properties)
$index->nested('comments')->properties(function (Blueprint $nested) {
$nested->keyword('name');
$nested->text('comment');
$nested->keyword('country');
$nested->integer('likes');
$nested->date('created_at');
});
// To define object fields without full nesting, use dot notation.
$index->text('products.name');
$index->float('products.price')->coerce(false);
//Disk space considerations ::
//Not indexed and not searchable:
$index->keyword('internal_notes')->docValues(false);
//Remove scoring for search:
$index->text('tags')->norms(false);
//Remove from index, can't search by this field but can still use for aggregations:
$index->integer('score')->indexField(false);
//If null is passed as value, then it will be saved as 'NA' which is searchable
$index->keyword('favorite_color')->nullValue('NA');
//Numeric Types
$index->integer('some_int');
$index->float('some_float');
$index->double('some_double');
$index->long('some_long');
$index->short('some_short');
$index->byte('some_byte');
$index->halfFloat('some_half_float');
$index->scaledFloat('some_scaled_float',140);
$index->unsignedLong('some_unsigned_long');
//Alias Example
$index->text('notes');
$index->aliasField('comments', 'notes');
$index->geoPoint('last_login');
$index->date('created_at');
$index->date('updated_at');
//Settings
$index->withSetting('number_of_shards', 3);
$index->withSetting('number_of_replicas', 2);
//Other Mappings
$index->withMapping('dynamic', false);
$index->withMapping('date_detection', false);
//Custom Mapping
$index->property('flattened','purchase_history');
//Custom Analyzer Setup:
//Analyzer Setup
$index->addAnalyzer('my_custom_analyzer')
->type('custom')
->tokenizer('punctuation')
->filter(['lowercase', 'english_stop'])
->charFilter(['emoticons']);
//Tokenizer Setup
$index->addTokenizer('punctuation')
->type('pattern')
->pattern('[ .,!?]');
//CharFilter Setup
$index->addCharFilter('emoticons')
->type('mapping')
->mappings([":) => _happy_", ":( => _sad_"]);
//Filter Setup
$index->addFilter('english_stop')
->type('stop')
->stopwords('_english_');
//Normalizer Setup
$index->addNormalizer('my_normalizer')
->type('custom')
->charFilter([])
->filter(['lowercase', 'asciifolding']);
});
}
public function down()
{
Schema::deleteIfExists('contacts');
}
}
```
> **Note:**
> If the built-in helpers for mapping fields do not meet your needs, you can use the:
>
> `$index->property($type, $field, $params)` method to define custom mappings.
Example:
```php
$index->property('date', 'last_seen', [
'format' => 'epoch_second||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd',
'ignore_malformed' => true,
]);
```
---
## Index Creation
### Schema::create
Creates a new index with the specified structure and settings.
```php
Schema::create('my_index', function (Blueprint $index) {
// Define fields, settings, and mappings
});
```
### Schema::createIfNotExists
Creates a new index only if it does not already exist.
```php
Schema::createIfNotExists('my_index', function (Blueprint $index) {
// Define fields, settings, and mappings
});
```
---
## Index Deletion
### Schema::delete
Deletes the specified index. Will throw an exception if the index does not exist.
```php
// Boolean
Schema::delete('my_index');
```
### Schema::deleteIfExists
Deletes the specified index if it exists.
```php
// Boolean
Schema::deleteIfExists('my_index');
```
---
## Index Lookup and Information Retrieval
### Schema::getIndex
Retrieves detailed information about a specific index or indices matching a pattern.
```php
Schema::getIndex('my_index');
Schema::getIndex('page_hits_*');
//or
Schema::getTable('my_index');
```
### Schema::getIndices
Equivalent to `Schema::getIndex('*')`, retrieves full information about all indices on the Elasticsearch cluster.
```php
Schema::getIndices();
```
### Schema::getIndicesSummary
Alternative `Schema::getTables()`
Retrieves information about all indices on the Elasticsearch cluster.
```php
Schema::getIndicesSummary();
```
```json
...
{
"name": "user_logs",
"status": "open",
"health": "yellow",
"uuid": "UfLaQLcWSHSP_rVaWZuibA",
"docs_count": "12554",
"docs_deleted": "0",
"store_size": "2.3mb"
},
...
```
### Schema::getMappings
Retrieves the mappings for a specified index.
```php
Schema::getMappings('my_index');
```
> ```json
> {
> "color": {
> "type": "text"
> },
> "color.keyword": {
> "type": "keyword",
> "ignore_above": 256
> },
> "created_at": {
> "type": "date"
> },
> "description": {
> "type": "text"
> },
> "description.keyword": {
> "type": "keyword",
> "ignore_above": 256
> },
> "is_active": {
> "type": "boolean"
> },
> "last_order_datetime": {
> "type": "date",
> "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd"
> },
> "last_order_ts": {
> "type": "date",
> "format": "epoch_millis||epoch_second"
> },
> "manufacturer": [],
> "manufacturer.country": {
> "type": "text"
> },
> "manufacturer.country.keyword": {
> "type": "keyword",
> "ignore_above": 256
> },
> "manufacturer.location": {
> "type": "geo_point"
> },
> "manufacturer.name": {
> "type": "text"
> },
> "manufacturer.name.keyword": {
> "type": "keyword",
> "ignore_above": 256
> },
> "name": {
> "type": "text"
> },
> "name.keyword": {
> "type": "keyword",
> "ignore_above": 256
> },
> "updated_at": {
> "type": "date"
> }
> }
> ```
Raw mapping
```php
Schema::getMappings('my_index',true);
```
> ```json
> {
> "my_index": {
> "mappings": {
> "properties": {
> "color": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "created_at": {
> "type": "date"
> },
> "description": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "is_active": {
> "type": "boolean"
> },
> "last_order_datetime": {
> "type": "date",
> "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd"
> },
> "last_order_ts": {
> "type": "date",
> "format": "epoch_millis||epoch_second"
> },
> "manufacturer": {
> "properties": {
> "country": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "location": {
> "type": "geo_point"
> },
> "name": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> }
> }
> },
> "name": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "updated_at": {
> "type": "date"
> }
> }
> }
> }
> }
> ```
### Schema::getSettings
Retrieves the settings for a specified index.
```php
Schema::getSettings('my_index');
```
> ```json
> {
> "my_index": {
> "settings": {
> "index": {
> "routing": {
> "allocation": {
> "include": {
> "_tier_preference": "data_content"
> }
> }
> },
> "number_of_shards": "1",
> "provided_name": "my_index",
> "creation_date": "1742830787691",
> "number_of_replicas": "1",
> "uuid": "g1Jzm6_ORA6dGftm59asKQ",
> "version": {
> "created": "8521000"
> }
> }
> }
> }
> }
>
> ```
### Schema::hasField
Checks if a specific field exists in the index's mappings.
```php
// Boolean
Schema::hasField('my_index', 'my_field');
```
### Schema::hasFields
Checks if multiple fields exist in the index's mappings.
```php
// Boolean, true if all fields exist
Schema::hasFields('my_index', ['field1', 'field2']);
```
### Schema::indexExists
Checks if a specific index exists.
```php
// Boolean
Schema::indexExists('my_index');
```
### Schema::getFieldMapping
Returns the mapping for a specific field
Schema method that can be called from your model:
```php
Product::getFieldMapping('color'); //Returns a key/value array of field/types for color
Product::getFieldMapping('color',true); //Returns the mapping for color field as is from Elasticsearch
Product::getFieldMapping(['color','name']); //Returns mappings for color and name
Product::getFieldMapping(); //returns all field mappings, same as getFieldMapping('*')
```
> `Product::getFieldMapping()`;
> ```json
> {
> "color": "text",
> "color.keyword": "keyword",
> "created_at": "date",
> "datetime": "text",
> "datetime.keyword": "keyword",
> "description": "text",
> "description.keyword": "keyword",
> "is_active": "boolean",
> "last_order_datetime": "date",
> "last_order_ts": "date",
> "manufacturer.country": "text",
> "manufacturer.country.keyword": "keyword",
> "manufacturer.location": "geo_point",
> "manufacturer.name": "text",
> "manufacturer.name.keyword": "keyword",
> "name": "text",
> "name.keyword": "keyword",
> "updated_at": "date"
> }
>
> ```
or via Schema: `Schema::getFieldMapping($index, $field, $raw)`
```php
Schema::getFieldMapping('products','color',true);
```
---
## Prefix Management
### Schema::overridePrefix
Temporarily overrides the default index prefix for subsequent operations. This can be useful in multi-tenant applications or when accessing indices across different environments.
```php
Schema::overridePrefix('some_other_prefix')->getIndex('my_index');
```
---
## Direct DSL Access
Provides direct access to the Elasticsearch DSL, allowing for custom queries and operations not covered by other methods.
```php
Schema::indices()->close(['index' => 'my_index']);
```
> **Note:**
> `Schema::indices()` is an instance the Elasticsearch PHP client's `indices()` endpoint.
---
---
## Index Blueprint
This guide walks you through defining Elasticsearch index schemas using Laravel, including field types, parameters, and advanced settings.
## Eloquent Column Types
Elasticsearch integration in Laravel maps Eloquent column types to Elasticsearch fields, allowing for seamless schema definitions with custom parameters for specific Elasticsearch functionalities.
```php
Schema::create('new_index', function (Blueprint $index) {
$index->longText('description', parameters: ['similarity' => 'boolean']);
});
```
> This example sets the scoring algorithm of a text field to `boolean`, customizing how Elasticsearch handles the field's relevance scoring.
Note that certain Eloquent types are not supported due to the nature of Elasticsearch:
* Increment types (`bigIncrements`, `increments`, etc.)—Elasticsearch does not support auto-incrementing fields.
* Set types (`set`)—No direct equivalent in Elasticsearch.
* JSON types (`json`, `jsonb`)—Redundant as Elasticsearch inherently supports JSON.
## Elasticsearch Column Types
Elasticsearch-specific field types are also available to cater to specialized data structures and queries:
```php
//Basic types
$index->text('first_name')->copyTo('full_name');
$index->text('last_name')->copyTo('full_name');
$index->text('full_name');
$index->text('text', hasKeyword: true); // Adds a text field with a keyword subfield for exact matching.
$index->keyword('my_keyword_field'); // For exact text matches.
$index->boolean('is_active');
// Facilitates nested object queries
$index->nested('my_nested_field');
// Define nested object fields with their own properties
$index->nested('comments')->properties(function (Blueprint $nested) {
$nested->keyword('name');
$nested->text('comment');
$nested->keyword('country');
$nested->integer('likes');
$nested->date('created_at');
});
//flattened array
$index->flattened('purchase_history');
// Define parent-child relationships
$index->join('question', 'answer'); // 'question' is the parent, 'answer' is the child
//Alias Example
$index->text('notes');
$index->aliasField('comments', 'notes'); // 'comments' is an alias for 'notes'
//Date
$index->date('first_contact', 'epoch_second');
$index->dateRange('date_range');
//Geo
$index->geoPoint('geo_point'); // Stores and queries geographical locations.
$index->geoShape('geo_shape'); // For complex geographical shapes.
//Ips
$index->ip('ip'); // Dedicated for IP addresses.
$index->ipRange('ip_range');
//Numbers
$index->integer('some_int');
$index->integerRange('integer_range');
$index->float('some_float');
$index->floatRange('float_range');
$index->double('some_double');
$index->doubleRange('double_range');
$index->long('long');
$index->longRange('long_range');
$index->short('some_short');
$index->byte('some_byte');
$index->halfFloat('some_half_float');
$index->scaledFloat('some_scaled_float',140);
$index->unsignedLong('some_unsigned_long');
$index->range('integer_range', 'integer_range_2');
// Specialized for percolate queries.
// Used for percolate queries (match stored queries against new docs)
$index->percolator('percolator');
//Custom types
$index->property('text', 'custom_property');
//Multiple types => Order matters ::
//Top level `email` will be a searchable text field
//Sub Property will be a keyword type which can be sorted using orderBy('email.keyword')
$index->text('email');
$index->keyword('email');
```
## Column Parameters
Modify field behaviors with parameters to enhance indexing and querying capabilities:
```php
Schema::create('contacts', function (IndexBlueprint $index) {
$index->text('first_name', parameters: ['copy_to' => 'full_name']);
$index->text('full_name');
});
```
### addColumn() / property()
If the built-in helpers for mapping fields do not meet your needs, you can use the `$table->addColumn($type, $field, array $parameters = [])` (this method is also aliased with `property()`) method to define custom mappings.
```php
Schema::create('contacts', function (Blueprint $index) {
$table->addColumn('date', 'last_seen', [
'format' => 'epoch_second||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd',
'ignore_malformed' => true,
]);
});
```
`addColumn()` is aliased as `property()`.
### properties()
Define nested properties within an index to facilitate complex queries and data structures.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->nested('comments')->properties(function (Blueprint $nested) {
$nested->keyword('name');
$nested->text('comment');
$nested->keyword('country');
$nested->integer('likes');
$nested->date('created_at');
});
});
```
### boost()
Boost specific fields at query time to influence relevance scores more significantly.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->text('test')->boost(2);
});
```
### nullValue()
Allows indexing of explicit null values by replacing them with a specified non-null value.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->keyword('token')->nullValue('NULL');
});
```
### format()
Customize the format of date fields using standard date formatting syntax.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->date('date')->format('yyyy');
});
```
## Other Index Modifications
Enhance your Elasticsearch indices with additional settings and functionalities:
### withSetting()
Adjust core index settings such as the number of shards.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->withSetting('number_of_shards', 3);
});
```
### withMapping()
Adjust core index settings such as the number of shards.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->withMapping('dynamic', false);
});
```
### meta()
Attach metadata to the index for enhanced identification and management.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->meta(['class' => 'MyApp2::User3']);
});
```
### alias()
Create an alias for the index to facilitate easier access and management.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->alias('bar_foo');
});
```
### routingRequired()
Make routing a mandatory requirement for the index to ensure efficient data placement and retrieval.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->routingRequired('bar_foo');
});
```
### addAnalyzer()
Configures or revises analyzers specific to an index, crucial for managing text tokenization and analysis.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->addAnalyzer('my_custom_analyzer')
->type('custom')
->tokenizer('punctuation')
->filter(['lowercase', 'english_stop'])
->charFilter(['emoticons']);
});
```
### addTokenizer()
Configures or revises tokenizers specific to an index, essential for text tokenization.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->addTokenizer('punctuation')
->type('pattern')
->pattern('[ .,!?]');
});
```
### addCharFilter()
Configures or revises character filters specific to an index, crucial for text normalization and analysis.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->addCharFilter('emoticons')
->type('mapping')
->mappings([":) => _happy_", ":( => _sad_"]);
});
```
### addFilter()
Configures or revises filters specific to an index, essential for text normalization and analysis.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->addFilter('english_stop')
->type('stop')
->stopwords('_english_');
});
```
### addNormalizer()
Configures or revises normalizers specific to an index, crucial for text normalization and analysis.
```php
Schema::create('contacts', function (Blueprint $index) {
$index->addNormalizer('my_normalizer')
->type('custom')
->charFilter([])
->filter(['lowercase', 'asciifolding']);
});
```
---
## Re-indexing Process
Re-indexing in Elasticsearch is a critical operation, especially when you need to alter the mapping of existing data. This process involves creating a new index with the desired mappings and copying the data from the old index to the new one. Here’s a step-by-step example using this package’s schema tools to safely re-index with updated mappings.
***
## Missed Mapping
To create a real-world scenario, let's consider a site log system that stores the URL, user IP, and location of visitors.
We'll intentionally overlook the mapping of the `location` field, which should be mapped as a `geo_point` type.
```php
Schema::create('site_logs', function (Blueprint $index) {
$index->keyword('url');
$index->ip('user_ip');
// Initially, the 'location' field might be overlooked or incorrectly mapped
});
```
Assuming there's a model for this index, you create a record like so:
```php
SiteLog::create([
'url' => 'https://example.com/contact-us',
'user_ip' => '0.0.0.0',
'location' => ['lat' => -7.3, 'lon' => 3.1] // This field is not correctly mapped yet
]);
```
After some time, you realize that the `location` field should be mapped as a `geo_point` type. If you try to filter records based on the `location` field, you'll get an error because the field is not correctly mapped.
***
## Re-indexing
1. ### Create a Temporary Index with Correct Mapping
Create a temporary index with the correct mapping for the location field.
```php
Schema::create('temp_site_logs', function (Blueprint $index) {
$index->keyword('url');
$index->ip('user_ip');
$index->geoPoint('location'); // Correct mapping for the 'location' field
});
```
2. ### Re-indexing Data to the Temporary Index
Copy all records from the original site\_logs index to the temp\_site\_logs index.
```php
$result = Schema::reIndex('site_logs', 'temp_site_logs');
```
**Important:** After re-indexing, inspect `$result->data` for any errors, and confirm the record count using:
```php
$copiedRecords = DB::connection('elasticsearch')->table('my_prefix_temp_site_logs')->count();
```
3. ### Delete the Original Index
Once you've confirmed that all data has been successfully copied, delete the original index.
```php
Schema::delete('site_logs');
```
4. ### Recreating the Original Index with Correct Mapping
Now, recreate the `site_logs` index with the correct mappings.
```php
Schema::create('site_logs', function (Blueprint $index) {
$index->keyword('url');
$index->ip('user_ip');
$index->geoPoint('location'); // Correct mapping
});
```
5. ### Re-indexing Data Back to the Original Index
Copy the data from the temporary index back to the original index.
```php
$result = Schema::reIndex('temp_site_logs', 'site_logs');
```
Again, verify the success of the re-indexing operation:
```php
// Confirm all records were copied
$copiedRecords = DB::connection('elasticsearch')->table('my_prefix_site_logs')->count();
```
6. ### Verifying the Correct Functionality
Now, with the location field correctly mapped, filtering operations should work as expected.
```php
$logs = SiteLog::whereGeoBox('location', [-10, 10], [10, -10])->get();
```
If issues persist, delete the index and repeat from step 4 to ensure the mapping is correct.
7. ### Delete the Temporary Index
Finally, delete the temporary index.
```php
Schema::delete('temp_site_logs');
```
***
---
## Extending the Base model
In this section, we'll dive into how to hook your Laravel models into Elasticsearch by extending the base model, allowing you to work with Elasticsearch indices as if they were regular Eloquent models.
## Extending Your Model
Every model you want to index in Elasticsearch should extend the `PDPhilip\Elasticsearch\Eloquent\Model`. **This base model extends Laravel's Eloquent model, so you can use it just like you would any other Eloquent model.**
```php
// app/Models/Product.php
namespace App\Models;
use PDPhilip\Elasticsearch\Eloquent\Model;
class Product extends Model
{
protected $connection = 'elasticsearch';
}
```
Just like a regular model, the index name will be inferred from the name of the model. In this example, the corresponding index for the `Product` model is `products`. In most cases, the elasticsearch connection won't be the default connection, and you'll need to include `protected $connection = 'elasticsearch'` in your model.
***
## Model IDs
### id alias for _id
`$model->id` and is an alias for Elasticsearch's `_id` field. This is to maintain consistency with Laravel's Eloquent ORM.
You can use either `$model->id` or `$model->_id` to access the ID of the model, but it's recommended to use `$model->id` for consistency with Laravel's Eloquent ORM.
```php
$model = MyModel::where('id','123')->first();
echo $model->id; // 123
echo $model->_id; // 123
```
### Laravel vs Elasticsearch generated IDs
By default, Elasticsearch will generate a unique ID for each document. However, this requires an additional API call to retrieve the ID after the document is created.
To avoid this, you can generate the ID on the Laravel side by using:
##### 1. `GeneratesUuids` trait
This trait will generate a time-based UUID for the ID field when creating a new model instance.
```php
use PDPhilip\Elasticsearch\Eloquent\Model;
+ use PDPhilip\Elasticsearch\Eloquent\GeneratesUuids;
class Product extends Model
{
+ use GeneratesUuids;
protected $connection = 'elasticsearch';
```
Example id:
```json
{
"id": "OWU3ZTllNDEtNmFmNS00MjdkLWE5MmEtNmJhMTBkZWI0Y2Vh"
}
```
##### 2. `GeneratesElasticIds` trait
This trait will generate an imitation of an Elasticsearch ID for the ID field when creating a new model instance.
```php
use PDPhilip\Elasticsearch\Eloquent\Model;
+ use PDPhilip\Elasticsearch\Eloquent\GeneratesElasticIds;
class Product extends Model
{
+ use GeneratesElasticIds;
protected $connection = 'elasticsearch';
```
Example id:
```json
{
"id": "GXUGv5UBt-dmV3b48pjF"
}
```
## Model properties
### `$table`
To change the inferred index name, pass in the `$table` property:
```php
//app/Models/Product.php
namespace App\Models;
use PDPhilip\Elasticsearch\Eloquent\Model;
class Product extends Model
{
protected $connection = 'elasticsearch';
+ protected $table = 'my_products';
}
```
> **Note:**
> You can also set [Dynamic indices](https://elasticsearch.pdphilip.com/eloquent/dynamic-indices) which will allow you to use the same model for multiple indices.
### Timestamps
By default, the base model will automatically set the `created_at` and `updated_at` fields. As is the case with Eloquent, you can disable this by setting the CREATED\_AT and UPDATED\_AT constants to null in your model
```php
//app/Models/Product.php
namespace App\Models;
use PDPhilip\Elasticsearch\Eloquent\Model;
class Product extends Model
{
protected $connection = 'elasticsearch';
+ const CREATED_AT = null;
+ const UPDATED_AT = null;
}
```
### Limits
If your query does not include a limit, (`->take(200)`) then Elasticsearch will default to 10.
This tends to be impractical in a Laravel context with hybrid data, thus **this integration overrides that value and defaults to 1000**.
You can change this default limit by:
1. Setting the `protected $defaultLimit` property in your model which will apply to that model only.
```php
use PDPhilip\Elasticsearch\Eloquent\Model;
class Product extends Model
{
+ protected $defaultLimit = 10000;
protected $connection = 'elasticsearch';
}
```
2. Setting the `ES_OPT_DEFAULT_LIMIT` environment variable in your `.env` file which will apply to all models that do not have a `$defaultLimit` property set.
```ini
ES_OPT_DEFAULT_LIMIT=10000
```
## Query Field Map
Elasticsearch requires specific field types for certain operations such as aggregation, sorting, or filtering. For example, while a `text` field cannot be used directly for sorting or term-based filtering, a corresponding `keyword` subfield is typically employed. However, this isn't restricted to just keywords; other field types may also need specific mappings depending on the operation.
To automate the process of using the correct field type at runtime, this package performs a mapping call unless the `bypass_map_validation` connection option is set to true. Enabling this setting avoids the overhead of an additional API call to fetch mappings for each query, which could affect performance. To facilitate efficient query processing without extra API calls, the model includes a `$queryFieldMap` property. This is an associative array where the keys are the column names and the values are the specific fields required for operations, ensuring the correct mappings are utilized.
```php
namespace App\Models;
use PDPhilip\Elasticsearch\Eloquent\Model;
class Product extends Model
{
protected $connection = 'elasticsearch';
+ protected $queryFieldMap = [
+ 'name' => 'name.keyword', // Example of a keyword subfield
+ 'date' => 'date.long' // Hypothetical example for date fields requiring long type
+ ];
}
```
This modification ensures all necessary operations are efficiently managed by leveraging the correct field mappings, enhancing performance and capability of the Elasticsearch integration in your Laravel application.
***
## Mutators & Casting
**You can use mutators and casting in your models just like you would with any other Eloquent model.**
In the context of the Laravel-Elasticsearch integration, the foundational BaseModel **inherits all the features of Laravel's Eloquent model**, including mutators and casting. You can use mutators and casts exactly like any standard Eloquent model.
For a comprehensive understanding of how to implement and use attribute mutators and casts within your models, refer to the official Laravel documentation on Eloquent Mutators & Casting: [Laravel - Eloquent: Mutators & Casting](https://laravel.com/docs/eloquent-mutators).
***
## Elastic Collections
Once a query is executed, the query meta is stored in the model instance as an `ElasticCollection`.
An `ElasticCollection` is a collection of an Elasticsearch record; akin to Laravel's `Eloquent\Collection` and augmented with Meta Data.
You can access the query meta by calling the `getMeta()` method on the model instance.
```php
$product = Product::where('color', 'green')->first();
return $product->getMeta()->toArray();
// or $product->getMetaAsArray();
```
returns:
```json
{
"score": 1,
"index": "es11_products",
"table_prefix": "es11_",
"table": "products",
"table_suffix": "",
"doc_count": null,
"sort": [],
"cursor": [],
"highlights": [],
"bucket": [], // New from v5.3+
}
```
### Track total hits
[Version 5.1+]
Method `->withTrackTotalHits(bool|int|null $val = true)`
Appends the `track_total_hits` parameter to the DSL query, setting value to `true` will count all the hits embedded in the query meta not capping to Elasticsearch default of 10k hits
```php
$products = Product::limit(5)->withTrackTotalHits(true)->get();
$totalHits = $products->getQueryMeta()->getTotalHits();
```
- `$val = true` - Returns total hits
- `$val = null` - Disables and uses ES default (Max 10k)
- `$val = false` - Does not count total hits at all, returns -1
- `$val = 50000` - Any int will count hits up to the given limit
This can be set by default for all queries by updating the connection config in `database.php`:
```php
'elasticsearch' => [
'driver' => 'elasticsearch',
.....
'options' => [
+ 'track_total_hits' => env('ES_TRACK_TOTAL_HITS', null),
....
],
],
```
### Get model meta value
[Version 5.3+]
Convenience method to get a specific meta value from the model instance.
```php
$product = Product::where('color', 'green')->first();
$score = $product->getMetaValue('score');
***
---
## Saving Models
Using Laravel-Elasticsearch integration allows developers to save models in a
way that aligns with the typical Eloquent patterns used in Laravel,
making it easier to switch from or use alongside traditional relational models.
## Save a New Model
### Option A: Attribute Assigning
Create a new model instance, assign attributes individually, and then save it to the Elasticsearch index. This method is direct and reflects common Laravel ORM practices.
```php
$log = new UserLog;
$log->user_id = $userId;
$log->title = $title;
$log->status = 1;
$log->save();
```
> Demonstrates creating and saving a new model instance by setting each
> attribute separately.
### Option B: Mass Assignment via `create()`
Use the `create()` method for mass assigning model attributes through an associative array, streamlining the creation and saving of a model instance.
```php
$log = UserLog::create([
'user_id' => $userId,
'title' => $title,
'status' => 1,
]);
```
> **Note:**
> The `create()` method respects the `$fillable` and `$guarded` attributes for secure attribute handling.
## Updating a Model
Update models in Elasticsearch following Eloquent's methodology: fetch a model, modify its attributes, and then call `save()`.
```php
$log = UserLog::where('status', 1)->first();
$log->status = 2;
$log->save();
```
> Fetch the first User Log with a status of 1, change its status to 2, then save
> the updated model.
For updates affecting multiple documents, employ the `update()` method on a query builder instance, which allows for bulk modifications based on query conditions.
```php
Product::where('status', 1)->update(['status' => 4]);
```
> $updates will indicate the number of documents updated.
## Inserting Models
Insert models similarly to Eloquent's approach, where you pass an array of arrays, each representing a document to be created.
```php
UserLog::insert([
['user_id' => 'John_Doe'],
['user_id' => 'Jane_Doe'],
]);
```
> Insert two records into the `UserLog` index.
For performance optimization, utilize `withoutRefresh()` to skip waiting for the index to refresh after insertions.
```php
UserLog::withoutRefresh()->insert([
['user_id' => 'John_Doe'],
['user_id' => 'Jane_Doe'],
]);
```
#### bulkInsert()
`bulkInsert()` is identical to `insert()` but will continue on errors and return an array of the results.
```php
People::bulkInsert([
[
'id' => '_edo3ZUBnJmuNNwymfhJ', // Will update (if id exists)
'name' => 'Jane Doe',
'status' => 1,
],
[
'name' => 'John Doe', // Will Create
'status' => 2,
],
[
'name' => 'John Dope',
'status' => 3,
'created_at' => 'xxxxxx', // Will fail
],
]);
```
Returns:
```json
{
"hasErrors": true,
"total": 3,
"took": 0,
"success": 2,
"created": 1,
"modified": 1,
"failed": 1,
"errors": [
{
"id": "Y-dp3ZUBnJmuNNwy7vkF",
"type": "document_parsing_exception",
"reason": "[1:45] failed to parse field [created_at] of type [date] in document with id 'Y-dp3ZUBnJmuNNwy7vkF'. Preview of field's value: 'xxxxxx'"
}
]
}
```
## Fast Saves
Elasticsearch provides a near real-time index, meaning there is a brief delay between indexing a document and its availability for search. Optimize write-heavy operations by skipping the wait with `withoutRefresh()`.
```php
$log->withoutRefresh()->save();
```
and
```php
UserLog::withoutRefresh()->create($attributes);
```
> **Caution:**
> Saving without refreshing the index may result in a slight delay before the data is available for search.
If you need to update attributes after saving then do not use `withoutRefresh()` as subsequent updates will not be saved.
```php
$log = new UserLog;
$log->user_id = $userId;
$log->title = $title;
$log->status = 1;
$log->withoutRefresh()->save();
$log->company_id = 'ABC-123';
$log->withoutRefresh()->save(); // company_id will not be saved
```
## First Or Create
The `firstOrCreate()` method either retrieves a model matching the specified attributes or creates a new one if no match exists, with distinct arrays for search attributes and new values.
```php
$book = Book::firstOrCreate(
['title' => $title, 'author' => $author], // search attributes
['description' => $description, 'stock' => 0] // attributes for creation
);
```
> Searches for a book with specified title and author. If found, returns it; if
> not, creates and returns a new book with additional details.
## First Or Create Without Refresh
Enhance `firstOrCreate()` with `withoutRefresh()` to forgo index refreshing when speed is prioritized over immediate data retrieval.
```php
$book = Book::withoutRefresh()->firstOrCreate(
['title' => $title,'author' => $author], //$attributes
['description' => $description, 'stock' => 0] //values
);
```
> Searches and creates without delay, making the book available in the index
> with a slight delay.
## Update Or Create Without Refresh
Enhance `updateOrCreate()` with `withoutRefresh()` to forgo index refreshing when speed is prioritized over immediate data retrieval.
```php
$book = Book::withoutRefresh()->updateOrCreate(
['title' => $title,'author' => $author], //$attributes
['description' => $description, 'stock' => 0] //values
);
```
> Searches and updates without delay, making the book available in the index
> with a slight delay.
## Create Or Fail
[Version 5.1+]
By default, when using `create($attributes)` where `$attributes `has an `id` that exists, the operation will upsert. `createOrFail` will throw a `BulkInsertQueryException` with status code `409` if the `id` exists
```php
Product::createOrFail([
'id' => 'some-existing-id',
'name' => 'Blender',
'price' => 30,
]);
```
## With Refresh
[Version 5.1+]
`withRefresh(bool|string $refresh)`
By default, inserting documents will wait for the shards to refresh, ie: `withRefresh(true)`, you can set the refresh flag with the following (as per ES docs):
- `true` (default): Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately.
- `wait_for`: Wait for the changes made by the request to be made visible by a refresh before replying. This doesn’t force an immediate refresh, rather, it waits for a refresh to happen.
- `false`: Take no refresh-related actions. The changes made by this request will be made visible at some point after the request returns.
```php
Product::withRefresh('wait_for')->create([
'name' => 'Blender',
'price' => 30,
]);
```
---
## Deleting Models
The Laravel-Elasticsearch integration facilitates model deletion in a manner that's highly consistent with the Laravel Eloquent ORM, offering a familiar and intuitive interface consistent with Laravel's Eloquent ORM.
***
## Single Model Deletion
To delete a single model, first, retrieve the model instance using the find method and then call the delete method on the instance. This operation removes the document from the Elasticsearch index corresponding to the model.
```php
$product = Product::find('IiLKG38BCOXW3U9a4zcn');
$product->delete();
```
> Find the product with the ID `IiLKG38BCOXW3U9a4zcn` and delete it.
***
## Mass Deletion
For deleting multiple models based on certain criteria, you can chain the delete method to a query.
```php
Product::whereNull('color')->delete();
```
> Delete all products where the color is null or the color field is missing.
***
## Truncating an Index
The truncate method removes all documents from an index without deleting the index itself. This is useful for quickly clearing all data while preserving the index settings and mappings.
```php
Product::truncate();
```
`truncate()` only affects documents, not mappings or index settings.
> Remove all documents from the `products` index but keep the index itself.
***
## Destroy by id
> `_id` under the hood of course
Single `id`
```php
Product::destroy('9iKKHH8BCOXW3U9ag1_4');
```
Multiple `id`s
```php
Product::destroy(
'4yKKHH8BCOXW3U9ag1-8',
'_iKKHH8BCOXW3U9ahF8Q'
);
```
Multiple `id`s as an array
```php
Product::destroy([
'4yKKHH8BCOXW3U9ag1-8',
'_iKKHH8BCOXW3U9ahF8Q'
]);
```
***
## Soft Deletes
Soft deletes add a `deleted_at` timestamp to the document.
These records are excluded from queries by default, but can be included or restored.
- `withTrashed()` → includes soft-deleted records.
- `restore()` → removes the deleted_at timestamp.
- `forceDelete()` → permanently deletes the documents.
To use soft deletes, include the `SoftDeletes` trait in your model:
```php
use PDPhilip\Elasticsearch\Eloquent\Model;
use PDPhilip\Elasticsearch\Eloquent\SoftDeletes;
class Product extends Model
{
use SoftDeletes;
}
```
With soft deletes enabled, you can **include deleted models** in your query results using the `withTrashed()` method:
```php
Product::withTrashed()->where('color', 'red')->get();
```
> Retrieve all products with the color red, including any soft-deleted products.
With soft deletes enabled, you can **restore** soft-deleted collections using the `restore()` query:
```php
Product::withTrashed()->where('color', 'red')->restore();
```
> Find all products with the color red and restore any that may have been soft-deleted.
To permanently remove a soft-deleted collection, you can use the **forceDelete** method:
```php
Product::withTrashed()->where('discontinued_at', '<', '2020-01-01')->forceDelete();
```
***
---
## Querying Models
Query models in Elasticsearch using the same Eloquent-style syntax you're familiar with
***
## All
Retrieve all records for a given model:
```php
$products = Product::all();
```
Equivalent to `get()` without clauses.
```php
$products = Product::get();
```
***
## Find
As with Eloquent, you can use the `find` method to retrieve a model by its primary key (id). The method will return a single model instance or `null` if no matching model is found.
```php
$product = Product::find('IiLKG38BCOXW3U9a4zcn');
```
> Find the product with id of `IiLKG38BCOXW3U9a4zcn` and return the model collection, or null if it does not exist.
```php
$product = Product::findOrFail('IiLKG38BCOXW3U9a4zcn');
```
> Find the product with id of `IiLKG38BCOXW3U9a4zcn` and return the model collection, or throw a `ModelNotFoundException` if no result is found.
***
## First
As with Eloquent, you can use the `first` method to retrieve the first model that matches the query. The method will return a single model instance or `null` if no matching model is found.
```php
$product = Product::where('status',1)->first();
```
> Find the first product with a status of 1 and return the model collection if it exists.
```php
$product = Product::where('status',1)->firstOrFail();
```
> Find the first product with a status of 1 and return the model collection, or throw a `ModelNotFoundException` if no result is found.
> **Note:**
> If the `ModelNotFoundException` is not caught, a 404 HTTP response is automatically sent back to the client
***
## Get
As with Eloquent, you can use the `get` method to retrieve all models that match the query. The method will return a model collection or an empty collection if no matching models are found.
```php
$product = Product::where('status',1)->get();
```
```json
{
"index":"products",
"body":{
"query":{
"match":{
"status":1
}
},
"_source":[
"*"
]
}
}
```
***
## Query and return DSL output
[Version 5.3+]
If you are uncertain about the query being generated, you can use the `dslQuery()` method to build your eloquent query and return the underlying Elasticsearch DSL array.
```php
$dsl = Product::dslQuery()->where('status',1)->get();
```
```json
// return $dsl:
{
"index":"products",
"body":{
"query":{
"match":{
"status":1
}
},
"_source":[
"*"
]
}
}
```
> **Note:**
> `dslQuery()` will only build and return the DSL array, it will not execute the query against Elasticsearch.
---
## Eloquent Queries
This section covers the most common query methods directly from Laravel, mapped to Elasticsearch DSL under the hood.
---
## Where
As with Eloquent, the `where` method is used to filter the query results based on the given conditions. The method accepts three parameters: the field name, an operator, and a value. The operator is optional and defaults to `=` if not provided.
```php
Product::where('status',1)->take(10)->get();
```
```json
{
"index": "products",
"body": {
"query": {
"term": {
"status": {
"value": 1
}
}
},
"size": 10
}
}
```
```php
Product::where('manufacturer.country', 'England')->take(10)->get();
```
```json
{
"index": "products",
"body": {
"query": {
"term": {
"manufacturer.country.keyword": {
"value": "England"
}
}
},
"size": 10
}
}
```
```php
Product::where('status','>=', 3)->take(10)->get();
```
```json
{
"index": "products",
"body": {
"query": {
"range": {
"status": {
"gte": 3
}
}
},
"size": 10
}
}
```
```php
Product::where('color','!=', 'red')->take(10)->get(); //*See notes
```
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must_not": [
{
"term": {
"color.keyword": {
"value": "red"
}
}
}
]
}
},
"size": 10
}
}
```
> **Note:**
> This query also matches documents where the `color` field is missing. To exclude nulls, combine with `whereNotNull('color')`.
### See ES-specific queries for more complex queries like
* [WhereExact](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-exact)
* [WherePhrase](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-phrase)
* [WherePhrasePrefix](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-phrase-prefix)
* [WhereTimestamp](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-timestamp)
* [WhereRegex](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-regex)
***
## Query Options **NEW**
Each query clause can have additional Elasticsearch specific options passed into them. You can either pass in an array of options, like:
```php
Person::where('name', 'joHn', ['case_insensitive' => true])->get();
```
```json
{
"index": "people",
"body": {
"query": {
"term": {
"name.keyword": {
"value": "joHn",
"case_insensitive": true
}
}
},
"size": 1000
}
}
```
Or use the package callback helper:
```php
use PDPhilip\Elasticsearch\Query\Options\TermOptions;
Person::where('name', 'joHn', function (TermOptions $options) {
$options->asCaseInsensitive();
})->get();
```
```json
{
"index": "people",
"body": {
"query": {
"term": {
"name.keyword": {
"value": "joHn",
"case_insensitive": true
}
}
},
"size": 1000
}
}
```
This is equivalent to passing `['case_insensitive' => true]`, but with better IDE support and method chaining.
### Query Options Helpers
The full set of helpers are:
```php
use PDPhilip\Elasticsearch\Query\Options\TermOptions; // for where() and whereTerm()
use PDPhilip\Elasticsearch\Query\Options\TermsOptions; //for whereIn()
use PDPhilip\Elasticsearch\Query\Options\MatchOptions; // for whereMatch()
use PDPhilip\Elasticsearch\Query\Options\NestedOptions; // for whereNestedObject()
use PDPhilip\Elasticsearch\Query\Options\PhraseOptions; // for wherePhrase()
use PDPhilip\Elasticsearch\Query\Options\PhrasePrefixOptions; // for wherePhrasePrefix()
use PDPhilip\Elasticsearch\Query\Options\PrefixOptions; // for wherePrefix()
use PDPhilip\Elasticsearch\Query\Options\RegexOptions; // for whereRegex()
use PDPhilip\Elasticsearch\Query\Options\FuzzyOptions; // for whereTermFuzzy()
use PDPhilip\Elasticsearch\Query\Options\SearchOptions; //All multi_match queries
use PDPhilip\Elasticsearch\Query\Options\DateOptions; // for whereDate()
```
For all the options in one helper you can use:
```php
use PDPhilip\Elasticsearch\Query\Options\ClauseOptions;
```
> **Caution:**
> Make sure to use the correct options for the query type, else Elasticsearch will throw an error.
## Where using LIKE
Using the `like` operator in your where clause works differently here than in SQL. Since Elasticsearch will match tokens, you can use a normal `where` clause to search for partial matches (assuming text field with the standard analyser).
For this package, you can use the `like` operator to search for partial matches within tokens. The package will automatically convert the `like` operator to a wildcard query, and will search for the term in the field. For example, to search for products with a color that contains the letters `bl` (blue, black, etc.), you can use the following query:
```php
Product::where('color', 'like', 'bl')->orderBy('color.keyword')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"wildcard": {
"color.keyword": {
"value": "*bl*"
}
}
},
"sort": [
{
"color.keyword": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
***
## WhereNot
The `whereNot` method is used to exclude documents that match the condition.
```php
Product::whereNot('status', 1)->get();
```
> Find all products that do not have a status of 1, identical to `where('status', '!=', 1)`
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must_not": [
{
"term": {
"status": 1
}
}
]
}
},
"size": 1000
}
}
```
***
## AND statements
The `where` method can be chained to add more conditions to the query. This will be read as `AND` in the query.
```php
Product::where('is_active', true)->where('in_stock', '<=', 50)->get();
```
> Find all products that are active and have 50 or less in stock
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"is_active": {
"value": true
}
}
},
{
"range": {
"in_stock": {
"lte": 50
}
}
}
]
}
},
"size": 1000
}
}
```
***
## OR Statements
The `orWhere` method can be used to add an `OR` condition to the query.
```php
Product::where('is_active', false)->orWhere('in_stock', '>=', 100)->get();
```
> Find all products that are not active or have 100 or more in stock
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"term": {
"is_active": {
"value": false
}
}
}
]
}
},
{
"bool": {
"must": [
{
"range": {
"in_stock": {
"gte": 100
}
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
***
## Chaining OR/AND statements
You can chain `where` and `orWhere` methods to create complex queries.
```php
Product::where('type', 'coffee')->where('is_approved', true)
->orWhere('type', 'tea')->where('is_approved', false)
->get();
```
> Find all products that are either coffee and approved, or tea and not approved. The query reads as: Where type is coffee and is approved, or where type is tea and is not approved.
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"term": {
"type.keyword": {
"value": "coffee"
}
}
},
{
"term": {
"is_approved": {
"value": true
}
}
}
]
}
},
{
"bool": {
"must": [
{
"term": {
"type.keyword": {
"value": "tea"
}
}
},
{
"term": {
"is_approved": {
"value": false
}
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
> **Note:**
> **Order of chaining matters** , It reads naturally from left to right having
>
> * `where() as AND where `
> * `orWhere() as OR where `
>
> In the above example, the query would be:
> `(type:"coffee" AND is_approved:true) OR (type:"tea" AND is_approved:false)`
***
## WhereIn
The `whereIn` method is used to include documents that match any of the values passed in the array.
```php
Product::whereIn('status', [1,5,11])->get();
```
> Find all products with a status of 1, 5, or 11
```json
{
"index": "products",
"body": {
"query": {
"terms": {
"status": [
1,
5,
11
]
}
},
"size": 1000
}
}
```
> **Note:**
> The `whereIn` method is effectively a concise OR statement on the same field. The above query is equivalent to:
>
> ```php
> $products = Product::where('status', 1)->orWhere('status', 5)->orWhere('status', 11)->get();
> ```
***
## WhereNotIn
The `whereNotIn` method is used to exclude documents that match any of the values passed in the array.
```php
Product::whereNotIn('color', ['red','green'])->get();
```
> Find all products that do not have a color of red or green
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must_not": [
{
"terms": {
"color.keyword": [
"red",
"green"
]
}
}
]
}
},
"size": 1000
}
}
```
> **Note:**
> The `whereNotIn` method is effectively a concise AND NOT statement on the same field. The above query is equivalent to:
>
> ```php
> Product::where('color', '!=', 'red')->where('color', '!=', 'green')->get();
> ```
***
## WhereNull
> Can be read as Where `field` does not exist
In traditional SQL databases, whereNull is commonly used to query records where a specific column's value is NULL, indicating the absence of a value. However, in Elasticsearch, the concept of NULL applies to the absence of a field as well as the field having a value of NULL.
**Therefore, in the context of the Elasticsearch implementation within Laravel, `whereNull` and `WhereNotNull` have been adapted to fit the common Elasticsearch requirement to query the existence or non-existence of a field as well as the null value of the field.**
```php
Product::whereNull('color')->get();
```
> Find all products that do not have a color field
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "color"
}
}
]
}
},
"size": 1000
}
}
```
***
## WhereNotNull
> Can be read as Where `field` Exists
> **Note:**
> Using `whereNotNull` is more common than it's counterpart (`whereNull`) since any negation style query of a field **will include documents that do not have the field**. Thus to negate values and only include documents that have the field, `whereNotNull` is used.
```php
Product::whereNotIn('color', ['red','green'])->whereNotNull('color')->get();
```
> Find all products that do not have a color of red or green, and ensure that the color field exists
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must_not": [
{
"terms": {
"color.keyword": [
"red",
"green"
]
}
}
],
"must": [
{
"exists": {
"field": "color"
}
}
]
}
},
"size": 1000
}
}
```
***
## WhereBetween
As with Eloquent, the `whereBetween` method is used to filter the query results based on the given range. The method accepts two parameters: the field name and an array containing the minimum and maximum values of the range.
```php
Product::whereBetween('in_stock', [10, 100])->get();
```
> Find all products with an in_stock value between 10 and 100 (including 10 and 100)
```json
{
"index": "products",
"body": {
"query": {
"range": {
"in_stock": {
"gte": 10,
"lte": 100
}
}
},
"size": 1000
}
}
```
```php
Product::whereBetween('orders', [1, 20])->orWhereBetween('orders', [100, 200])->get();
```
> Find all products with an orders value between 1 and 20, or between 100 and 200 (including 1, 20, 100, and 200)
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"range": {
"orders": {
"gte": 1,
"lte": 20
}
}
}
]
}
},
{
"bool": {
"must": [
{
"range": {
"orders": {
"gte": 100,
"lte": 200
}
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
***
## Grouped Queries
As with native Laravel Eloquent, `where` (and alike) clauses can accept a `$query` closure to group multiple queries together.
```php
Product::where(function ($query) {
$query->where('status', 1)
->orWhere('status', 2);
})->get();
```
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"term": {
"status": {
"value": 1
}
}
}
]
}
},
{
"bool": {
"must": [
{
"term": {
"status": {
"value": 2
}
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
A more advanced example:
```php
Product::whereNot(function ($query) {
$query->where('color', 'lime')->orWhere('color', 'blue');
})->orWhereNot(function ($query) {
$query->where('status', 2)->where('is_active', false);
})->orderBy('status')->get();
```
> Find all products that do not have a color of lime or blue, or do not have a status of 2 and are not active, and order the results by status
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must_not": [
{
"bool": {
"should": [
{
"bool": {
"must": [
{
"term": {
"color.keyword": {
"value": "lime"
}
}
}
]
}
},
{
"bool": {
"must": [
{
"term": {
"color.keyword": {
"value": "blue"
}
}
}
]
}
}
]
}
}
]
}
},
{
"bool": {
"must_not": [
{
"bool": {
"must": [
{
"term": {
"status": {
"value": 2
}
}
},
{
"term": {
"is_active": {
"value": false
}
}
}
]
}
}
]
}
}
]
}
},
"sort": [
{
"status": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
***
## Dates
Elasticsearch by default converts a date into a timestamp, and applies the `strict_date_optional_time||epoch_millis`
format. If you have not changed the default format for your index then acceptable values are:
* 2022-01-29
* 2022-01-29T13:05:59
* 2022-01-29T13:05:59+0:00
* 2022-01-29T12:10:30Z
* 1643500799 (timestamp)
With Carbon
```php
Carbon::now()->modify('-1 week')->toIso8601String()
```
You can use these values in a normal [where](#where) clause, or use the built-in date clause, ie:
**WhereDate()**
```php
Product::whereDate('created_at', '2022-01-29')->get();
```
> **Note:**
> The usage for `whereMonth` / `whereDay` / `whereYear` / `whereTime` has been disabled for the current version of this plugin
***
## Empty strings values
**Avoid saving fields with empty strings**, as Elasticsearch will treat them as a value and not as a null field. Rather, use `null` or simply do not include the field when writing the document.
Good example ✅
```php
$product = new Product();
$product->name = 'Glass bowl';
$product->color = null;
$product->save();
```
or
```php
$product = new Product();
$product->name = 'Glass bowl';
$product->save();
```
Bad example: ❌
```php
$product = new Product();
$product->name = 'Glass bowl';
$product->color = '';
$product->save();
```
If you need to find products without a color, the above product will not be included in the results. Further, querying for products with an empty string requires special handling since `where('color', '')` will not work as expected.
> **Note:**
> If you need to query for an empty string, you can use the following:
>
> ```php
> Product::whereIn('color', [''])->get();
> ```
>
> ```php
> Product::whereExact('color', '')->get(); //specific to this package
> ```
***
---
## Elasticsearch Specific Queries
Elasticsearch offers a rich set of query capabilities, including geospatial queries and regex-based searches, which go beyond the traditional query types found in SQL databases. The Laravel-Elasticsearch integration provides a seamless interface to leverage these powerful features directly within your Laravel models.
***
## Where Match **NEW**
This method allows you to query for matches within a field. This is useful when you need to search for a specific value within a text field.
```php
Person::whereMatch('description', 'John Smith')->get();
```
```json
{
"index": "people",
"body": {
"query": {
"match": {
"description": {
"query": "John Smith",
"operator": "and"
}
}
},
"size": 1000
}
}
```
### Where Match options
```php
use PDPhilip\Elasticsearch\Query\Options\MatchOptions;
Person::whereMatch('description', 'Jhn Smth', function (MatchOptions $options) {
$options->operator('or');
$options->analyzer('my_custom_analyzer');
$options->asFuzzy();
})->get();
```
```json
{
"index": "products",
"body": {
"query": {
"match": {
"description": {
"query": "Jhn Smth",
"operator": "or",
"analyzer": "my_custom_analyzer",
"fuzziness": "AUTO"
}
}
},
"size": 1000
}
}
```
***
## Where Term
This method allows you to query for exact case-sensitive matches within a field. This is useful when you need to find a specific value.
Make sure the field is mapped as a `keyword` or uses `.keyword` subfield. Otherwise, `term` queries will fail.
```php
Person::whereTerm('name', 'John Smith')->get();
//Or
Person::whereExact('name', 'John Smith')->get(); //Alias
```
> This will only return the documents where the name field is exactly 'John Smith'. 'john smith' or 'John' will not be returned.
```json
{
"index": "people",
"body": {
"query": {
"term": {
"name.keyword": {
"value": "John Smith"
}
}
},
"size": 1000
}
}
```
### Where Term options
```php
use PDPhilip\Elasticsearch\Query\Options\TermOptions;
Person::whereTerm('name', 'John Smith',function (TermOptions $options) {
$options->asCaseInsensitive();
})->get();
```
```json
{
"index": "people",
"body": {
"query": {
"term": {
"name.keyword": {
"value": "John Smith",
"case_insensitive": true
}
}
},
"size": 1000
}
}
```
***
## Where Prefix
This method allows you to query for exact case-sensitive values that start with the given value within a field.
Make sure the field is mapped as a `keyword` or uses `.keyword` subfield. Otherwise, `term` queries will fail.
```php
Person::wherePrefix('name', 'John Sm')->get();
//Or
Person::whereStartsWith('name', 'John Sm')->get(); //Alias
```
```json
{
"index": "people",
"body": {
"query": {
"prefix": {
"name.keyword": {
"value": "John Sm"
}
}
},
"size": 1000
}
}
```
## Where Phrase
This method allows you to query for exact phrases within a field. This is useful when you need to search for a specific sequence of words within a text field.
Under the hood, this method uses the `match_phrase` query from Elasticsearch's Query DSL.
```php
Person::wherePhrase('description', 'loves espressos')->get();
```
> This will only return the documents where the description field contains the exact phrase 'loves espressos'. Individual tokens like 'loves' or 'espressos' will not be returned in isolation.
```json
{
"index": "people",
"body": {
"query": {
"match_phrase": {
"description": {
"query": "loves espressos"
}
}
},
"size": 1000
}
}
```
### Where Phrase Options
```php
use PDPhilip\Elasticsearch\Query\Options\PhraseOptions;
Person::wherePhrase('description', 'loves espressos',function (PhraseOptions $options) {
$options->analyzer('english');
$options->slop(1);
})->get();
```
> This will only return the documents where the description field contains the exact phrase 'loves espressos'. Individual tokens like 'loves' or 'espressos' will not be returned in isolation.
```json
{
"index": "people",
"body": {
"query": {
"match_phrase": {
"description": {
"query": "loves espressos",
"analyzer": "english",
"slop": 1
}
}
},
"size": 1000
}
}
```
***
## Where Phrase Prefix
Similar to `wherePhrasePrefix`, this method allows you to query for exact phrases where the last word starts with a particular prefix.
Under the hood, this method uses the `match_phrase_prefix` query from Elasticsearch's Query DSL.
```php
Person::wherePhrasePrefix('description', 'loves es')->get();
```
> This will only return the documents where the description field contains the phrase 'loves es....'. Ex: 'loves espresso', 'loves essays' and 'loves eskimos' etc
```json
{
"index": "people",
"body": {
"query": {
"match_phrase_prefix": {
"description": {
"query": "loves es"
}
}
},
"size": 1000
}
}
```
### Where Phrase Prefix Options
```php
use PDPhilip\Elasticsearch\Query\Options\PhrasePrefixOptions;
Person::wherePhrasePrefix('description', 'loves es',function (PhrasePrefixOptions $options) {
$options->analyzer('my_custom_analyzer');
$options->maxExpansions(75);
$options->slop(1);
})->get();
```
```json
{
"index": "people",
"body": {
"query": {
"match_phrase_prefix": {
"description": {
"query": "loves es",
"analyzer": "my_custom_analyzer",
"max_expansions": 75,
"slop": 1
}
}
},
"size": 1000
}
}
```
***
## Where Regex
The `WhereRegex` method allows you to query for documents based on a regular expression pattern within a field.
```php
$regex1 = Product::whereRegex('color', 'bl(ue)?(ack)?')->get();
$regex2 = Product::whereRegex('color', 'bl...*')->get();
```
> The first example will return documents where the color field matches the pattern 'bl(ue)?(ack)?', which means it can be 'blue' or 'black'. The second example will return documents where the color field matches the pattern 'bl...\*', which means it starts with 'bl' and has at least three more characters. Both should return Blue or Black from the colors field.
```json
{
"index": "products",
"body": {
"query": {
"regexp": {
"color.keyword": {
"value": "bl(ue)?(ack)?"
}
}
},
"size": 1000
}
}
{
"index": "products",
"body": {
"query": {
"regexp": {
"color.keyword": {
"value": "bl...*"
}
}
},
"size": 1000
}
}
```
### Where Regex Options
```php
use PDPhilip\Elasticsearch\Query\Options\RegexOptions;
Product::whereRegex('color', 'bl(ue)?(ack)?', function (RegexOptions $options) {
$options->flags('ALL');
$options->caseInsensitive(true);
$options->maxDeterminizedStates(20000);
})->get();
```
```json
{
"index": "products",
"body": {
"query": {
"regexp": {
"color.keyword": {
"value": "bl(ue)?(ack)?",
"flags": "ALL",
"case_insensitive": true,
"max_determinized_states": 20000
}
}
},
"size": 1000
}
}
```
***
## Where Fuzzy **NEW**
The `whereFuzzy` method allows you to query for documents based on a fuzzy search within a field. This is useful when you need to search for approximate matches to a given value.
`whereFuzzy` is a term-level query.
```php
Product::whereFuzzy('description', 'qick brwn fx')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"fuzzy": {
"description": {
"value": "qick brwn fx"
}
}
},
"size": 1000
}
}
```
### Where Fuzzy Options
```php
use PDPhilip\Elasticsearch\Query\Options\FuzzyOptions;
Product::whereFuzzy('description', 'qick brwn fx',function (FuzzyOptions $options) {
$options->fuzziness('AUTO');
$options->maxExpansions(50);
$s>prefixLength(0);
$options->transpositions(true);
$options->rewrite('constant_score_blended');
})->get();
```
```json
{
"index": "products",
"body": {
"query": {
"fuzzy": {
"description": {
"value": "qick brwn fx",
"fuzziness": "AUTO",
"max_expansions": 50,
"prefix_length": 0,
"transpositions": true,
"rewrite": "constant_score_blended"
}
}
},
"size": 1000
}
}
```
## Where Geo Distance
The `whereGeoDistance` method filters results based on their proximity to a given point, specified by latitude and longitude, within a certain radius.
**Method Signature**
```php
/**
* Filter results based on a geo-point field within a defined box on a map.
*
* @param string $field
* @param string $distance [numeric value + distance unit (e.g., km, mi)]
* @param array $point [latitude, longitude]
* @param string $distanceType arc|plane
* @param string $validationMethod STRICT|IGNORE_MALFORMED|COERCE
* @return $this
*/
whereGeoDistance($column, string $distance, array $location, $distanceType = 'arc', $validationMethod = 'STRICT')
orWhereGeoDistance($column, string $distance, array $location, $distanceType = 'arc', $validationMethod = 'STRICT')
whereNotGeoDistance($column, string $distance, array $location, $distanceType = 'arc', $validationMethod = 'STRICT')
orWhereNotGeoDistance($column, string $distance, array $location, $distanceType = 'arc', $validationMethod = 'STRICT')
```
```php
// Specify the central point and radius
$point = [
'lat' => 51.509865,
'lon' => -0.118092,
]; //or:
$point = [51.509865, -0.118092]; // [latitude, longitude]
$distance = '20km';
// Retrieve UserLogs with Status 7 where 'agent.geo' is within 20km of center of London
UserLog::where('status', 7)->whereGeoDistance('agent.geo', $distance, $point)->get();
```
```json
{
"index": "user_logs",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 7
}
}
},
{
"geo_distance": {
"agent.geo": [
51.509865,
-0.118092
],
"distance": "20km"
}
}
]
}
},
"size": 1000
}
}
```
The `$distance` parameter is a string combining a numeric value and a distance unit (e.g., km for kilometers, mi for miles). Refer to the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#distance-units) on distance units for more information.
***
## Where Geo Box
The `whereGeoBox` method allows you to retrieve documents based on geospatial data, specifically targeting documents where a geo-point field falls within a defined "box" on a map. This is particularly useful for applications requiring location-based filtering, such as finding all events within a specific geographical area.
**Method Signature**
```php
/**
* Filter results based on a geo-point field within a defined box on a map.
*
* @param string $field
* @param array $topLeft [latitude, longitude]
* @param array $bottomRight [latitude, longitude]
* @param string $validationMethod STRICT|IGNORE_MALFORMED|COERCE
* @return $this
*/
whereGeoBox(string $field, array $topLeft, array $bottomRight, $validationMethod = 'STRICT')
orWhereGeoBox(string $field, array $topLeft, array $bottomRight, $validationMethod = 'STRICT')
whereNotGeoBox(string $field, array $topLeft, array $bottomRight, $validationMethod = 'STRICT')
orWhereNotGeoBox(string $field, array $topLeft, array $bottomRight, $validationMethod = 'STRICT')
```
```php
// Define the top-left and bottom-right coordinates of the box
$topLeft = [-10, 10]; // [latitude, longitude]
$bottomRight = [10, -10]; // [latitude, longitude]
// Retrieve UserLogs where 'agent.geo' falls within the defined box
UserLog::where('status', 7)->whereGeoBox('agent.geo', $topLeft, $bottomRight)->get();
```
```json
{
"index": "user_logs",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 7
}
}
},
{
"geo_bounding_box": {
"agent.geo": {
"top_left": [
-180,
90
],
"bottom_right": [
180,
-90
]
}
}
}
]
}
},
"size": 1000
}
}
```
> **Note:**
> The field **must be of type geo_point for both `whereGeoDistance` and `whereGeoBox` methods otherwise your shards will fail**, make sure to set the geo field in your migration, ex:
>
> ```php
> Schema::create('user_logs',function (IndexBlueprint $index){
> $index->geoPoint('agent.geo');
> });
> ```
>
> For more details, see [Migrations](https://elasticsearch.pdphilip.com/schema/migrations)
***
## Where Script **NEW**
Filter documents based on custom scripts.
```php
$script = "doc['orders'].size() > 0 && doc['orders'].value >= 29";
$options = ['params' => ['value' => 29]];
Product::whereScript($script, $options)->get();
```
```json
{
"index": "products",
"body": {
"query": {
"script": {
"script": {
"source": "doc['orders'].size() > 0 && doc['orders'].value >= 29",
"params": {
"value": 29
}
}
}
},
"size": 1000
}
}
```
## Where Timestamp
This method allows you to query for timestamps on a known field and will **sanitize the input to ensure it is a valid timestamp for both seconds and milliseconds**.
```php
Product::whereTimestamp('last_viewed', '<=', 1713911889521)->get();
```
> This will only return the documents where the last\_viewed field is less than or equal to the timestamp 1713911889521 ms.
```json
{
"index": "products",
"body": {
"query": {
"range": {
"last_viewed": {
"lte": 1713911889521
}
}
},
"size": 1000
}
}
```
***
## RAW DSL Queries
For scenarios where you need the utmost flexibility and control over your Elasticsearch queries, the Laravel-Elasticsearch integration provides the capability to directly use Elasticsearch's Query DSL (Domain Specific Language). The results will still be returned as collections of Eloquent models.
```php
$bodyParams = [
'query' => [
'match' => [
'color' => 'silver',
],
],
];
Product::rawSearch($bodyParams, $optionsParams = []); //Will search within the products index
Product::rawDsl($dsl); //Will use the given $dsl as is and return the raw response from Elasticsearch
```
> The DSL example above uses the `match` query to search for products with the color 'silver'
***
## RAW Aggregation Queries
Similar to raw search queries, you can also execute raw aggregation queries using Elasticsearch's Aggregation DSL. This allows you to perform complex aggregations on your data and retrieve the results in a structured format.
```php
$body = [
'aggs' => [
'price_ranges' => [
'range' => [
'field' => 'price',
'ranges' => [
['to' => 100],
['from' => 100, 'to' => 500],
['from' => 500, 'to' => 1000],
['from' => 1000],
],
],
'aggs' => [
'sales_over_time' => [
'date_histogram' => [
'field' => 'datetime',
'fixed_interval' => '1d',
],
],
],
],
],
];
return Product::rawAggregation($body);
```
> The aggregation example above uses the `range` aggregation to group products into price ranges and the `date_histogram` aggregation to group sales over time within each price range.
***
## To DSL
This method returns the parsed DSL query from the query builder. This can be useful when you need to inspect the raw query being generated by the query builder.
```php
$query = Product::where('price', '>', 100)->toDsl();
```
> This will return the raw DSL query generated by the query builder instance.
> **Note:**
> `toDsl()` and the inherited `toSQL()` are the same method. You can use them interchangeably.
***
---
## Cross Fields Search Queries
Cross fields search queries tap into Elasticsearch's text analysis features, allowing for sophisticated searches across text fields.
---
## Multi-match query
These methods enable full-text search across all (or some) fields and integrate seamlessly with standard Eloquent methods like `get()`, `first()`, `aggregate()`, and `paginate()`.
> **Note:**
> - `search{Method}` → across all fields/full-text queries
> - `where{Method}` → single-field queries
```php
MyModel::searchTerm('XYZ')->get();
```
```json
{
"index": "my_models",
"body": {
"query": {
"multi_match": {
"query": "XYZ",
"type": "best_fields"
}
},
"size": 1000
}
}
```
---
## Search Term
- `searchTerm($term, $fields = ['*'], $options = [])`
- `orSearchTerm($term, $fields = ['*'], $options = [])`
- Type: `best_fields`
```php
Book::searchTerm('Eric')->orSearchTerm('Lean')->searchTerm('Startup')->get();
```
> Search for books that contain 'Eric' or ('Lean' and 'Startup')
```json
{
"index": "books",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"multi_match": {
"query": "Eric",
"type": "best_fields"
}
}
]
}
},
{
"bool": {
"must": [
{
"multi_match": {
"query": "Lean",
"type": "best_fields"
}
},
{
"multi_match": {
"query": "Startup",
"type": "best_fields"
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
### Search Term Fuzzy
`searchFuzzy()` is a convenience method for `searchTerm()` with the `fuzziness` option set to `AUTO`. This allows for fuzzy matching, which is useful for handling typos or variations in spelling.
- `searchFuzzy($term, $fields = ['*'], $options = [])`
- `orSearchFuzzy($term, $fields = ['*'], $options = [])`
- Type: `best_fields`
```php
Book::searchFuzzy('Erc')->orSearchFuzzy('Leen')->orSearchFuzzy('Startop')->get();
```
> Fuzzy Search for books that contain 'Erc' or ('Leen' and 'Startop')
```json
{
"index": "books",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"multi_match": {
"query": "Erc",
"type": "best_fields",
"fuzziness": "AUTO"
}
}
]
}
},
{
"bool": {
"must": [
{
"multi_match": {
"query": "Leen",
"type": "best_fields",
"fuzziness": "AUTO"
}
},
{
"multi_match": {
"query": "Startop",
"type": "best_fields",
"fuzziness": "AUTO"
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
---
## Search Term Most
- `searchTermMost($term, $fields = ['*'], $options = [])`
- `orSearchTermMost($term, $fields = ['*'], $options = [])`
- Type: `most_fields`
```php
Book::searchTermMost('quick brown fox', [ "title", "title.original", "title.shingles" ])->get();
```
> Search for books that contain 'quick brown fox' in the 'title', 'title.original', or 'title.shingles' fields
```json
{
"index": "books",
"body": {
"query": {
"multi_match": {
"query": "quick brown fox",
"type": "most_fields",
"fields": [
"title",
"title.original",
"title.shingles"
]
}
},
"size": 1000
}
}
```
---
## Search Term Cross
- `searchTermCross($term, $fields = ['*'], $options = [])`
- `orSearchTermCross($term, $fields = ['*'], $options = [])`
- Type: `cross_fields`
```php
Person::searchTermCross('Will Smith', [ 'first_name','last_name'],['operator' => 'and'])->get();
```
> Search for people with the first name 'Will' and the last name 'Smith'
```json
{
"index": "people",
"body": {
"query": {
"multi_match": {
"query": "Will Smith",
"operator": "and",
"type": "cross_fields",
"fields": [
"first_name",
"last_name"
]
}
},
"size": 1000
}
}
```
---
## Search Phrase
- `searchPhrase($phrase, $fields = ['*'], $options = [])`
- `orSearchPhrase($phrase, $fields = ['*'], $options = [])`
- Type: `phrase`
```php
Product::searchPhrase('United States')->orSearchPhrase('United Kingdom')->get();
```
> Search for products that contain either 'United States' or 'United Kingdom' in any field, then sum the 'orders' field.
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"multi_match": {
"query": "United States",
"type": "phrase"
}
}
]
}
},
{
"bool": {
"must": [
{
"multi_match": {
"query": "United Kingdom",
"type": "phrase"
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
---
## Search Phrase Prefix
- `searchPhrasePrefix($phrase, $fields = ['*'], $options = [])`
- `orSearchPhrasePrefix($phrase, $fields = ['*'], $options = [])`
- Type: `phrase_prefix`
```php
Person::searchPhrasePrefix('loves espressos and te')->get();
```
> Search for people who have the phrase 'loves espressos and' with a prefix of 'ru' in the next word in any field. Ex:
- 'loves espressos and tea'
- 'loves espressos and tennis'
- 'loves espressos and tequila'
```json
{
"index": "people",
"body": {
"query": {
"multi_match": {
"query": "loves espressos and te",
"type": "phrase_prefix"
}
},
"size": 1000
}
}
```
---
## Search Bool Prefix
- `searchBoolPrefix($phrase, $fields = ['*'], $options = [])`
- `orSearchBoolPrefix($phrase, $fields = ['*'], $options = [])`
- Type: `bool_prefix`
- Scoring behaves like `most_fields`, but using a `match_bool_prefix` query instead of a `match` query.
```php
Person::searchBoolPrefix('loves espressos and te')->get();
```
> Search for people who have the phrase 'loves espressos and' with a prefix of 'ru' in the next word in any field. Ex:
- 'loves espressos and tea'
- 'loves espressos and tennis'
- 'loves espressos and tequila'
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "loves espressos and te",
"type": "bool_prefix"
}
},
"size": 1000
}
}
```
### Fuzzy Bool Prefix
- `searchFuzzyPrefix()` is a convenience method for `searchBoolPrefix()` with the `fuzziness` option set to `AUTO`
- `searchFuzzyPrefix($phrase, $fields = ['*'], $options = [])`
- `orSearchFuzzyPrefix($phrase, $fields = ['*'], $options = [])`
- Type: `bool_prefix`
```php
Person::searchFuzzyPrefix('lovs esprssos and te')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "lovs esprssos and te",
"type": "bool_prefix",
"fuzziness": "AUTO"
}
},
"size": 1000
}
}
```
---
## Parameter: $fields
By default, all fields will be searched through; you can specify which to search through as well as optionally boost certain fields using a caret `^`:
```php
People::searchTerm('John',['name^3','description^2','friends.name'])->get();
```
> Search for people with the name 'John' in the `name` field, `description` field, and `friends.name` field. The `name` field is boosted by 3, and the `description` field is boosted by 2, which will affect the relevance score used for sorting.
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "John",
"type": "best_fields",
"fields": [
"name^3",
"description^2",
"friends.name"
]
}
},
"size": 1000
}
}
```
---
## Parameter: $options
Allows you to set any options for the `multi_match` clause to use, ex:
- analyzer
- boost
- operator
- minimum_should_match
- fuzziness
- lenient
- prefix_length
- max_expansions
- fuzzy_rewrite
- zero_terms_query
```php
use PDPhilip\Elasticsearch\Query\Options\SearchOptions;
Product::searchTerm('espreso tme', function (SearchOptions $options) {
$options->type('most_fields');
$options->searchFuzzy();
$options->analyzer('my_custom_analyzer');
$options->boost(2);
$options->operator('OR');
$options->minimumShouldMatch(2);
$options->autoGenerateSynonymsPhraseQuery(true);
$options->fuzzyTranspositions(true);
$options->fuzzyRewrite('constant_score');
$options->lenient(true);
$options->zeroTermsQuery('all');
$options->constantScore(true);
})->get();
```
```json
{
"index": "products",
"body": {
"query": {
"constant_score": {
"filter": {
"multi_match": {
"query": "espreso tme",
"type": "best_fields",
"fuzziness": "AUTO",
"analyzer": "my_custom_analyzer",
"boost": 2,
"operator": "OR",
"minimum_should_match": 2,
"auto_generate_synonyms_phrase_query": true,
"fuzzy_transpositions": true,
"fuzzy_rewrite": "constant_score",
"lenient": true,
"zero_terms_query": "all"
}
}
}
},
"size": 1000
}
}
```
---
## Highlighting
Highlighting allows you to display search results with the matched terms highlighted:
`highlight($fields = [], $preTag = '', $postTag = '', $globalOptions = [])`
The `highlighted` results are stored in the model's metadata and can be accessed via a built-in model attribute using:
- `$model->searchHighlights`: returns on object with the found highlights for each field.
- `$model->searchHighlightsAsArray`: returns an associative array with the found highlights for each field.
**The values of the highlights are always in an array, even if there is only one fragment.**
> **Note:**
> In `$model->searchHighlights` only the top level fields are the object keys (as is with normal model attributes).
> Ex: `$model->searchHighlights->manufacturer['location']['country']`
> The array in `$model->searchHighlightsAsArray` is a flat associative array with the field names (in dot notation) as keys.
> Ex: `$model->searchHighlights[manufacturer.location.country]`
```php
$highlights = [];
$products = Product::searchTerm('espresso')->highlight()->get();
foreach ($products as $product) {
$highlights[$product->id] = $product->searchHighlights;
}
```
> Search for products containing 'espresso' in any field. All hits on `espresso` will be stored in the highlights metadata as an array under the field where the hit occurred.
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "and",
"type": "best_fields"
}
},
"highlight": {
"pre_tags": "",
"post_tags": ""
},
"size": 1000
}
}
```
You can filter the fields to highlight:
```php
$highlights = [];
$products = Product::searchTerm('espresso')->highlight(['description'],'','')->get();
foreach ($products as $product) {
$highlights[$product->_id] = $product->searchHighlights->description ?? [];
}
```
> Search for products containing 'espresso' in any field. Only hits on `espresso` in the `description` field will highlighted and wrapped in `strong` tags. All results will be returned.
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "espresso",
"type": "best_fields"
}
},
"highlight": {
"fields": [
{
"description": {}
}
],
"pre_tags": "",
"post_tags": ""
},
"size": 1000
}
}
```
```php
$highlightFields = [
'name' => [
'pre_tags' => [''],
'post_tags' => [''],
],
'description' => [
'pre_tags' => [''],
'post_tags' => [''],
],
'manufacturer.name' => [
'pre_tags' => [''],
'post_tags' => [''],
],
];
$highlights = [];
$products = Product::searchTerm('espresso')->highlight($highlightFields)->get();
foreach ($products as $product) {
$highlights[$product->_id]['name'] = $product->searchHighlights->name ?? [];
$highlights[$product->_id]['description'] = $product->searchHighlights->description ?? [];
$highlights[$product->_id]['manufacturer'] = $product->searchHighlights->manufacturer['name'] ?? [];
}
```
> Search for products containing 'espresso' in any field. Hits on `espresso` in the `name` field will be highlighted with a primary color, hits in the `description` field will be highlighted with a secondary color, and any hits in the `manufacturer.name` field will be highlighted with a sky color.
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "espresso",
"type": "best_fields"
}
},
"highlight": {
"fields": [
{
"name": {
"pre_tags": [
""
],
"post_tags": [
""
]
}
},
{
"description": {
"pre_tags": [
""
],
"post_tags": [
""
]
}
},
{
"manufacturer.name": {
"pre_tags": [
""
],
"post_tags": [
""
]
}
}
],
"pre_tags": "",
"post_tags": ""
},
"size": 1000
}
}
```
Global options can be set for all fields:
```php
$options = [
'number_of_fragments' => 3,
'fragment_size' => 150,
];
$highlights = [];
$products = Product::searchTerm('espresso')->highlight([],'','',$options)->get();
foreach ($products as $product) {
$highlights[$product->_id] = $product->searchHighlights;
}
```
> Search for products containing 'espresso' in any field. A maximum of 3 fragments will be returned for each field, with each fragment being a maximum of 150 characters long.
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "espresso",
"type": "best_fields"
}
},
"highlight": {
"number_of_fragments": 3,
"fragment_size": 150,
"pre_tags": "",
"post_tags": ""
},
"size": 1000
}
}
```
#### `$model->withHighlights->field`
This built in attribute will get all the model's data, parse any user defined mutators, then overwrite any fields that have highlighted data. This is useful when you want to display the highlighted data in a view.
> **Note:**
> This is not an instance of the model, so you cannot call any model methods on it (like save, update, etc). This is intentional to avoid accidentally saving the highlighted data to the database.
> **Note:**
> For multiple fragments, the values are concatenated with `.....`
```php
@foreach ($products as $product)
{!! $product->withHighlights->name !!}
{!! $product->withHighlights->description !!}
@endforeach
```
---
## Query String Queries
Query String Queries leverage Elasticsearch’s powerful `query_string` syntax to perform advanced full-text and structured searches directly on your Laravel models
---
## Search Query String
[Version 5.2+]
- `searchQueryString($query, $fields = null, $options = [])`
- `orSearchQueryString($query, $fields = null, $options = [])`
- `searchNotQueryString($query, $fields = null, $options = [])`
- `orSearchNotQueryString($query, $fields = null, $options = [])`
```php
Book::searchQueryString('Eric OR (Lean AND Startup)')->get();
```
```json
{
"index": "books",
"body": {
"query": {
"query_string": {
"query": "Eric OR (Lean AND Startup)"
}
},
"size": 1000
}
}
```
You can combine `searchQueryString` with other eloquent methods, for example:
```php
Book::searchQueryString('Eric OR (Lean AND Startup)')
->orderByDesc('sales')->get();
```
```json
{
"index": "books",
"body": {
"query": {
"query_string": {
"query": "Eric OR (Lean AND Startup)"
}
},
"sort": [
{
"sales": {
"order": "desc"
}
}
],
"size": 1000
}
}
```
```php
Book::searchQueryString('Eric OR (Lean AND Startup)')
->where('sales', '>', 100)->get();
```
```json
{
"index": "books",
"body": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Eric OR (Lean AND Startup)"
}
},
{
"range": {
"sales": {
"gt": 100
}
}
}
]
}
},
"size": 1000
}
}
```
```php
Book::searchQueryString('Eric OR (Lean AND Startup)')
->where('sales', '>=', 100)->limit(10)->skip(3)->get();
```
```json
{
"index": "books",
"body": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Eric OR (Lean AND Startup)"
}
},
{
"range": {
"sales": {
"gte": 100
}
}
}
]
}
},
"from": 3,
"size": 10
}
}
```
```php
Book::searchQueryString('Eric OR (Lean AND Startup)')
->where('published', true)->paginate(10);
```
```php
Book::searchQueryString('Eric OR (Lean AND Startup)')
->where('published', true)->count()
```
---
## Parameter: $fields
By default, all fields will be searched through; you can specify which to search through as well as optionally boost certain fields using a caret `^`:
```php
People::searchQueryString('John',['name^3','description^2','friends.name'])->get();
```
> Search through people for 'John' in the `name` field, `description` field, and `friends.name` field. The `name` field is boosted by 3, and the `description` field is boosted by 2, which will affect the relevance score used for sorting.
```json
{
"index": "people",
"body": {
"query": {
"query_string": {
"query": "John",
"fields": [
"name^3",
"description^2",
"friends.name"
]
}
},
"size": 1000
}
}
```
---
## Parameter: $options
Allows you to set any options for the `query_string` clause to use, ex:
- type
- `best_fields (Default), most_fields, cross_fields, phrase, phrase_prefix, bool_prefix`
- allow_leading_wildcard
- analyze_wildcard
- analyzer
- auto_generate_synonyms_phrase_query
- boost
- default_operator
- fuzziness
- fuzzy_max_expansions
- fuzzy_prefix_length
- fuzzy_transpositions
- fuzzy_rewrite
- lenient
- max_determinized_states
- minimum_should_match
- quote_analyzer
- phrase_slop
- quote_field_suffix
- rewrite
- time_zone
```php
use PDPhilip\Elasticsearch\Query\Options\QueryStringOptions;
Product::searchQueryString('espreso tme', function (QueryStringOptions $options) {
// Match behavior and field resolution
$options->type('most_fields'); // Default: best_fields
$options->analyzer('my_custom_analyzer');
// Fuzziness and tolerance
$options->fuzziness(2); // Allow up to 2 edit distances
$options->fuzzyPrefixLength(1);
$options->fuzzyTranspositions(true);
$options->fuzzyMaxExpansions(50);
$options->rewrite('constant_score');
// Scoring and matching
$options->boost(2.0);
$options->defaultOperator('OR'); // Default: OR
$options->minimumShouldMatch('2');
$options->autoGenerateSynonymsPhraseQuery(true);
// Wildcards and parsing
$options->allowLeadingWildcard(false);
$options->analyzeWildcard(true);
$options->lenient(true);
// Phrase matching
$options->phraseSlop(2);
$options->quoteFieldSuffix('.exact');
// Misc
$options->timeZone('UTC');
})->get();
```
```json
{
"index": "products",
"body": {
"query": {
"query_string": {
"query": "espreso tme",
"type": "most_fields",
"analyzer": "my_custom_analyzer",
"fuzziness": 2,
"fuzzy_prefix_length": 1,
"fuzzy_transpositions": true,
"fuzzy_max_expansions": 50,
"rewrite": "constant_score",
"boost": 2,
"default_operator": "OR",
"minimum_should_match": "2",
"auto_generate_synonyms_phrase_query": true,
"allow_leading_wildcard": false,
"analyze_wildcard": true,
"lenient": true,
"phrase_slop": 2,
"quote_field_suffix": ".exact",
"time_zone": "UTC"
}
},
"size": 1000
}
}
```
> **Note:**
> You can also pass the options as an associative array:
>
> ```php
> Product::searchQueryString('espreso tme', [
> 'type' => 'most_fields',
> 'fuzziness' => 2,
> 'fuzzy_prefix_length' => 1,
> 'fuzzy_transpositions' => true,
> 'boost' => 2.0,
> 'default_operator' => 'OR',
> 'minimum_should_match' => '2',
> 'auto_generate_synonyms_phrase_query' => true,
> 'analyzer' => 'my_custom_analyzer',
> 'lenient' => true,
> 'phrase_slop' => 2,
> ])->get();
> ```
## Query String: What’s Possible
### 1. Basics & Ranges
```php
User::searchQueryString('age:35')->get();
User::searchQueryString('age:>=35')->get();
User::searchQueryString('age:<=18')->get();
User::searchQueryString('age:(NOT 35)')->get();
```
### 2. Boolean Logic (AND / OR / NOT)
```php
// AND
User::searchQueryString('(age:35) AND (title:admin)')->get();
// OR
User::searchQueryString('name:(doe OR toe)')->get();
// NOT
User::searchQueryString('name:(NOT doe)')->get();
// AND NOT & OR NOT (composed with helpers)
User::searchNotQueryString('name:doe')->searchNotQueryString('name:toe')->get();
User::searchNotQueryString('name:doe')->orSearchNotQueryString('age:35')->get();
```
### 3. Search Scope (All fields vs Specific fields)
```php
// All fields
Product::searchQueryString('sweet')->get();
Product::searchQueryString('NOT sweet')->get();
// Specific field
Product::searchQueryString('sweet', 'details.type')->get();
Product::searchQueryString('false', 'details.gluten_free')->get();
// Fielded in query
Product::searchQueryString('details.gluten_free:false')->get();
```
### 4. Fuzziness (typo-tolerant)
```php
use PDPhilip\Elasticsearch\Query\Options\QueryStringOptions;
// Default fuzziness (~2 edits)
Product::searchQueryString('esprso~')->get();
// Custom fuzziness settings
Product::searchQueryString('espro~', function (QueryStringOptions $options) {
$options->fuzziness(3); // more tolerant
})->get();
Product::searchQueryString('espreso~', function (QueryStringOptions $options) {
$options->fuzziness(1); // tighter match, fewer results
})->get();
```
### 5. Default Operator & Minimum Should Match
```php
use PDPhilip\Elasticsearch\Query\Options\QueryStringOptions;
// Default operator (space = OR)
Product::searchQueryString('sweet soda')->get();
Product::searchQueryString('sweet soda', function (QueryStringOptions $options) {
$options->defaultOperator('AND'); // require both terms
})->get();
// Minimum should match on OR group
Product::searchQueryString('drink OR water OR soda')->get();
Product::searchQueryString('drink OR water OR soda', function (QueryStringOptions $options) {
$options->minimumShouldMatch(2); // must match at least two terms
})->get();
```
### 6. Phrases & Slop (ordered proximity)
```php
// Allow gaps between terms
Product::searchQueryString('"fresh juice"', fn($o) => $o->phraseSlop(1))->get();
// Exact adjacency only
Product::searchQueryString('"fresh juice"', fn($o) => $o->phraseSlop(0))->get();
```
### 7. Wildcards & Safety
```php
use PDPhilip\Elasticsearch\Query\Options\QueryStringOptions;
// Leading wildcard disabled → throws (not allowed by default)
Product::searchQueryString('*pro', function (QueryStringOptions $options) {
$options->allowLeadingWildcard(false);
})->get();
// Leading wildcard enabled → allowed
Product::searchQueryString('*pro', function (QueryStringOptions $options) {
$options->allowLeadingWildcard(true);
})->get();
```
### 8. Leniency (parse forgiving on types)
```php
// Strict → throws when querying numeric field with text
Product::searchQueryString('ABC', 'price')->get();
// Lenient → ignores type errors, returns 0 matches
Product::searchQueryString('ABC', 'price', fn($o) => $o->lenient(true))->get();
```
### 9. Query Types (best_fields, cross_fields, phrase, etc.)
```php
use PDPhilip\Elasticsearch\Query\Options\QueryStringOptions;
// phrase vs phrase_prefix (single field)
Product::searchQueryString('club sand', 'name', function (QueryStringOptions $options) {
$options->type('phrase'); // requires exact phrase match
})->get();
Product::searchQueryString('club sand', 'name', function (QueryStringOptions $options) {
$options->type('phrase_prefix'); // allows partial phrase match ("Club sandwich deluxe")
})->get();
// best_fields vs cross_fields (multi-field resolution)
Product::searchQueryString('vanilla pizza', ['name', 'details.product'], function (QueryStringOptions $options) {
$options->defaultOperator('AND')->type('best_fields'); // both terms must appear in the same field
})->get();
Product::searchQueryString('vanilla pizza', ['name', 'details.product'], function (QueryStringOptions $options) {
$options->defaultOperator('AND')->type('cross_fields'); // terms can appear across multiple fields
})->get();
// bool_prefix (autocomplete-style)
Product::searchQueryString('cheese piz', 'name', function (QueryStringOptions $options) {
$options->type('bool_prefix'); // prefix matching for partial terms
})->get();
```
### 10. Field Boosting (ranking control)
```php
use PDPhilip\Elasticsearch\Query\Options\QueryStringOptions;
// Boost name vs product and compare first() result
Product::searchQueryString('coffee', ['name^3', 'details.product'], function (QueryStringOptions $options) {
$options->type('cross_fields'); // boosts name field more heavily
})->first();
Product::searchQueryString('coffee', ['name', 'details.product^3'], function (QueryStringOptions $options) {
$options->type('cross_fields'); // boosts product field more heavily
})->first();
```
### 11. Regex (advanced matching)
```php
User::searchQueryString('name:/joh?n(ath[oa]n)/')->get();
```
### 12. Numeric Ranges (inclusive vs exclusive)
```php
// Inclusive upper bound
Product::searchQueryString('price:[5 TO 19]')->get();
// Exclusive upper bound (right brace)
Product::searchQueryString('price:[5 TO 19}')->get();
```
### 13. Mixed Boolean Operators (must/forbid/optional)
```php
// vanilla optional, +pizza required, -ice forbidden
Product::searchQueryString('vanilla +pizza -ice', fn($o) => $o->type('cross_fields'))->get();
```
## Simple Query string
> **Note:**
> `simple_query_string` is not supported in this package. It’s a restricted subset of `query_string` (no ranges, no regex, limited fielded clauses), and doesn’t justify separate maintenance.
> If you need “forgiving” parsing or full control, use:
> - `searchQueryString()` with `lenient` option
> - `rawDsl()` if you really must
---
## Aggregation Queries
Aggregations are powerful tools in Elasticsearch that allow you to summarize, compute statistics, and analyze data trends within your dataset. In the Laravel-Elasticsearch integration, aggregations are simplified to align with Eloquent's method of handling aggregate functions, making it intuitive for developers to perform complex data analysis.
***
## Basic Aggregations
You can use standard aggregate functions such as `count()`, `max()`, `min()`, `avg()`, and `sum()` directly on your Eloquent models, just like you would with a SQL database. These functions provide quick insights into your dataset.
```php
$totalSales = Sale::count(); // Total number of sales
$highestPrice = Sale::max('price'); // Maximum sale price
$lowestPrice = Sale::min('price'); // Minimum sale price
$averagePricePerSale = Sale::avg('price'); // Average sale price
$totalEarnings = Sale::sum('price'); // Sum of all sale prices
//Multiple fields at once
$highestPriceAndDiscountValue = Sale::max(['price','discount_amount']);
```
These aggregation functions are straightforward and mirror the typical usage in Laravel's Eloquent ORM, providing a seamless experience for developers.
Aggregations with Conditions
As the aggregation functions are part of the Eloquent ORM, you can also use them with conditions to filter the data you want to analyze.
```php
$averagePrice = Product::whereNotIn('color', ['red', 'green'])->avg('price');
```
> Average price of products excluding 'red' and 'green' colored products
***
## Grouped Aggregations
`agg()` is an optimization method that allows you to call multiple aggregation functions on a single field in one call.
This call saves you from making multiple queries to get different statistics for the same field.
```php
Product::where('is_active',true)->agg(['count','avg','min','max','sum'],'sales');
```
> Returns count, average, minimum, maximum, and sum of sales for active products
Available aggregation functions: `count`, `avg`, `min`, `max`, `sum`, `matrix`.
***
## Elasticsearch Aggregations
Elasticsearch offers advanced aggregation capabilities, including matrix stats aggregations, which provide comprehensive statistics about multiple fields. The Laravel-Elasticsearch integration simplifies the usage of these advanced features.
### Matrix Stats Aggregations
##### `matrix(string|array $fields,$options = [])`
```php
// Matrix stats for 'price' and `orders` fields, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->matrix('price');
// Matrix stats for both 'price' and 'orders' fields
Product::whereNotIn('color', ['red', 'green'])->matrix(['price', 'orders']);
```
Example result for `matrix(['price', 'orders']);`:
```json
{
"matrix_stats_price": {
"name": "price",
"count": 80,
"mean": 971.7610051512718,
"variance": 319573.9412606584,
"skewness": 0.09455383230430257,
"kurtosis": 1.8069082807157395,
"covariance": {
"price": 319573.9412606584,
"orders": 6096.144923266881
},
"correlation": {
"price": 1,
"orders": 0.14887562780257566
}
},
"matrix_stats_orders": {
"name": "orders",
"count": 80,
"mean": 120.7,
"variance": 5246.769620253164,
"skewness": -0.06873472971174674,
"kurtosis": 1.8427347935190153,
"covariance": {
"price": 6096.144923266881,
"orders": 5246.769620253164
},
"correlation": {
"price": 0.14887562780257566,
"orders": 1
}
}
}
```
##### Matrix Results Explained
The result of a matrix aggregation is a detailed statistical summary of the selected fields. Here's a breakdown of what each statistic represents:
* **doc\_count**: The total number of documents that matched the aggregation query.
* **fields**: An array containing the statistical data for each field included in the matrix aggregation.
* **name**: The name of the field.
* **count**: The number of values analyzed for this field.
* **mean**: The average value.
* **variance**: The variance indicating the data's spread.
* **skewness**: A measure of the asymmetry of the data distribution.
* **kurtosis**: A measure of the 'tailedness' of the data distribution.
* **covariance**: The covariance between the current field and other fields in the matrix, indicating how the fields vary together.
* **correlation**: The correlation between the current field and other fields, showing the strength and direction of a linear relationship.
***
### Boxplot Aggregations **NEW**
Boxplots help visualize distribution spread, quartiles, and outliers.
##### `boxplot(string|array $fields,$options = [])`
```php
// Boxplot for 'price' and `orders` fields, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->boxplot(['price', 'orders']);
```
Example result for `boxplot(['price', 'orders']);`:
```json
{
"boxplot_price": {
"min": 5.409999847412109,
"max": 1997.1500244140625,
"q1": 490.53751373291016,
"q2": 904.2349853515625,
"q3": 1437.7875366210938,
"lower": 5.409999847412109,
"upper": 1997.1500244140625
},
"boxplot_orders": {
"min": 3,
"max": 246,
"q1": 48.25,
"q2": 126,
"q3": 172,
"lower": 3,
"upper": 246
}
}
```
### Stats Aggregations **NEW**
##### `stats(string|array $fields,$options = [])`
```php
// Stats for 'price' and `orders` fields, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->stats(['price', 'orders']);
```
Example result for `stats(['price', 'orders']);`:
```json
{
"stats_price": {
"count": 80,
"min": 5.409999847412109,
"max": 1997.1500244140625,
"avg": 971.7610051512718,
"sum": 77740.88041210175
},
"stats_orders": {
"count": 80,
"min": 3,
"max": 246,
"avg": 120.7,
"sum": 9656
}
}
```
### Extended Stats Aggregations **NEW**
##### `extendedStats(string|array $fields,$options = [])`
```php
// Extended Stats for 'price' and `orders` fields, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->extendedStats(['price', 'orders']);
```
Example result for `extendedStats(['price', 'orders']);`:
```json
{
"extended_stats_price": {
"count": 80,
"min": 5.409999847412109,
"max": 1997.1500244140625,
"avg": 971.7610051512718,
"sum": 77740.88041210175,
"sum_of_squares": 100791897.45020083,
"variance": 315579.26699490024,
"variance_population": 315579.26699490024,
"variance_sampling": 319573.9412606585,
"std_deviation": 561.7644230412783,
"std_deviation_population": 561.7644230412783,
"std_deviation_sampling": 565.308713236103,
"std_deviation_bounds": {
"upper": 2095.2898512338284,
"lower": -151.7678409312848,
"upper_population": 2095.2898512338284,
"lower_population": -151.7678409312848,
"upper_sampling": 2102.378431623478,
"lower_sampling": -158.85642132093426
}
},
"extended_stats_orders": {
"count": 80,
"min": 3,
"max": 246,
"avg": 120.7,
"sum": 9656,
"sum_of_squares": 1579974,
"variance": 5181.185,
"variance_population": 5181.185,
"variance_sampling": 5246.769620253165,
"std_deviation": 71.98044873436119,
"std_deviation_population": 71.98044873436119,
"std_deviation_sampling": 72.43458856273821,
"std_deviation_bounds": {
"upper": 264.6608974687224,
"lower": -23.260897468722376,
"upper_population": 264.6608974687224,
"lower_population": -23.260897468722376,
"upper_sampling": 265.5691771254764,
"lower_sampling": -24.169177125476423
}
}
}
```
### Cardinality Aggregations **NEW**
##### `cardinality(string|array $fields,$options = [])`
```php
// Cardinality for 'price' and `orders` fields, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->cardinality(['price', 'orders']);
```
Example result for `cardinality(['price', 'orders']);`:
```json
{
"cardinality_price": 80,
"cardinality_orders": 66
}
```
### Median Absolute Deviation Aggregations **NEW**
##### `medianAbsoluteDeviation(string|array $fields,$options = [])`
```php
// Median Absolute Deviation for 'price' and `orders` fields, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->medianAbsoluteDeviation(['price', 'orders']);
```
Example result for `medianAbsoluteDeviation(['price', 'orders']);`:
```json
{
"median_absolute_deviation_price": 452.6300048828125,
"median_absolute_deviation_orders": 63.5
}
```
### Percentiles Aggregations **NEW**
##### `percentiles(string|array $fields,$options = [])`
```php
// Percentiles for 'price' and `orders` fields, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->percentiles(['price', 'orders']);
```
Example result for `percentiles(['price', 'orders']);`:
```json
{
"percentiles_price": {
"1.0": 13.144099817276,
"5.0": 124.55599937438966,
"25.0": 490.53751373291016,
"50.0": 904.2349853515625,
"75.0": 1437.7875366210938,
"95.0": 1827.5749877929686,
"99.0": 1904.7594665527336
},
"percentiles_orders": {
"1.0": 5.37,
"5.0": 11,
"25.0": 48.25,
"50.0": 126,
"75.0": 172,
"95.0": 237.1,
"99.0": 242.04999999999995
}
}
```
### String Stats Aggregations **NEW**
##### `stringStats(string|array $fields,$options = [])`
> **Note:**
> String stats are calculated based on the length and entropy of the string fields.
>
> The fields must be of type `keyword` and the keyword field must be explicitly given
```php
// String stats for the 'name' and 'description' field, excluding 'red' and 'green' colored products
Product::whereNotIn('color', ['red', 'green'])->stringStats(['name.keyword', 'description.keyword']);
```
Example result for `stringStats(['name.keyword', 'description.keyword']);`:
```json
{
"string_stats_name.keyword": {
"count": 80,
"min_length": 8,
"max_length": 25,
"avg_length": 15.1875,
"entropy": 4.881605005180552
},
"string_stats_description.keyword": {
"count": 80,
"min_length": 190,
"max_length": 205,
"avg_length": 197.3625,
"entropy": 4.530787852109844
}
}
```
---
## Distinct and GroupBy
In the Laravel-Elasticsearch integration, distinct() and groupBy() methods play a pivotal role in data aggregation, especially when retrieving unique field values or aggregating grouped data summaries.
***
The `distinct()` and `groupBy()` methods are used to retrieve unique values of a given field. But they differ significantly in how they're implemented under the hood
`distinct()` uses: **Nested Term Aggs**
- Full results, cannot paginate
- Sort on column names and by doc_count (ES default is _doc_count desc)
- Ideal For:
- Quick unique field lookups by doc count
`groupBy()` uses: **Composite Aggregation**
- Can paginate
- Sort on column names only (no doc_count)
- Ideal For:
- Paginated results
- Fetching relationship data
- Aggregating on the grouped data (eg: sum, avg, min, max)
## Distinct
> **Caution:**
> Implementation of `distinct()` in the context of this package differs from native Laravel.
>
> - In Laravel's Eloquent SQL, `distinct()` is used within the query building clauses.
> - In Laravel-Elasticsearch, `distinct()` executes the query (like `get()`).
### Basic Usage
##### `distinct(string|array $field, $includeDocCount = false)`
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->distinct('user_id');
// Alternative syntax, explicitly selecting the field
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->select('user_id')->distinct();
```
> Retrieves All the unique user_ids of users logged in the last 30 days
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-21T18:41:18+00:00"
}
}
},
"size": 0,
"aggs": {
"by_user_id": {
"terms": {
"field": "user_id",
"size": 1000
}
}
}
},
"_source": [
"user_id"
]
}
```
### Note on Limits
Distinct will run a `Nested Term Aggs` on a given query. The aggregation will return the top unique values, limited to 10 by default.
- ex: `UserLog::distinct('user_id')` - will run a term aggregation on all records and return the top 10 ordered by doc_count desc
- ex: `UserLog::limit(5)->distinct('user_id')` - will run a term aggregation on all records and return the top 5 ordered by doc_count desc
- ex: `UserLog::limit(10000)->distinct('user_id')` - will run a term aggregation on all records and return the top 10,000 ordered by doc_count desc
### Multiple Fields
Multiple fields perform a nested terms aggregation. The results is a unique combination of the fields provided.
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->distinct(['status', 'log_code']);
```
> Retrieves All the unique combinations of status and log_code of users logged in the last 30 days
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-21T18:42:19+00:00"
}
}
},
"size": 0,
"aggs": {
"by_status": {
"terms": {
"field": "status",
"size": 10
},
"aggs": {
"by_log_code": {
"terms": {
"field": "log_code",
"size": 10
}
}
}
}
}
},
"_source": [
"status",
"log_code"
]
}
```
### Ordering by Aggregation Count
`_count` is a special internal alias used by the package to sort by aggregation result count.
`_count` is not available with `groupBy()`.
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->orderByDesc('_count')
->distinct('user_id');
```
> Retrieves All the unique user_ids of users logged in the last 30 days
Ordered by most logs by a user
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-21T18:47:14+00:00"
}
}
},
"size": 0,
"aggs": {
"by_user_id": {
"terms": {
"field": "user_id",
"size": 10,
"order": [
{
"_count": "desc"
}
]
}
}
}
},
"_source": [
"user_id"
]
}
```
### Return with doc count
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->orderByDesc('_count')
->distinct('user_id',true);
//OR
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->orderByDesc('_count')
->select('user_id')
->distinct(true);
```
> Retrieves All the unique user_ids of users logged in the last 30 days
Ordered by most logs by a user
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-21T18:47:14+00:00"
}
}
},
"size": 0,
"aggs": {
"by_user_id": {
"terms": {
"field": "user_id",
"size": 10,
"order": [
{
"_count": "desc"
}
]
}
}
}
},
"_source": [
"user_id"
]
}
```
```json
{
"user_id": "1",
"user_id_count": 24
},
{
"user_id": "2",
"user_id_count": 18
},
{
"user_id": "3",
"user_id_count": 8
},
```
### Distinct with Relations
[Version 5.3+]
As of v5.3, distinct results are returned as [ElasticCollections](https://elasticsearch.pdphilip.com/eloquent/the-base-model#elastic-collections), enabling the use of Laravel's rich collection methods for further manipulation or processing.
If a model relation is defined and you've **aggregated on the foreign_key**, you can load the related model
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->with('user')
->orderByDesc('_count')
->select('user_id')
->distinct(true);
```
> Retrieves All the unique user_ids of users logged in the last 30 days
Ordered by most logs by a user
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-21T18:47:14+00:00"
}
}
},
"size": 0,
"aggs": {
"by_user_id": {
"terms": {
"field": "user_id",
"size": 10,
"order": [
{
"_count": "desc"
}
]
}
}
}
},
"_source": [
"user_id"
]
}
```
```json
{
"user_id": "1",
"user_id_count": 24,
"user": {
"id": "1",
"name": "John Doe",
...
}
},
{
"user_id": "2",
"user_id_count": 18,
"user": {
"id": "2",
"name": "Jane Smith",
...
}
},
{
"user_id": "3",
"user_id_count": 8,
"user": {
"id": "3",
"name": "Alice Johnson",
...
}
},
```
---
## Bulk Distinct
[Version 5.3+]
##### `bulkDistinct(array $fields, $includeDocCount = false)`
To perform distinct on multiple fields in a single query, you can use the `bulkDistinct()` method.
This operates similarly to `distinct()`, but specifying multiple fields does not create nested aggregations. Instead, it runs separate term aggregations for each field provided.
As with `distinct()`, setting `true` for `$includeCount` will return the doc_count for each unique value.
```php
$top3 = UserSession::where('created_at', '>=', Carbon::now()->subDays(30))
->limit(3)
->bulkDistinct(['country', 'device', 'browser_name'], true);
```
```json
{
"index": "user_sessions",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-12-21T13:02:07+00:00"
}
}
},
"size": 0,
"aggs": {
"by_country": {
"terms": {
"field": "country.keyword",
"size": 3
}
},
"by_device": {
"terms": {
"field": "device.keyword",
"size": 3
}
},
"by_browser_name": {
"terms": {
"field": "browser_name.keyword",
"size": 3
}
}
}
},
"_source": [
"country",
"device",
"browser_name"
]
}
```
```json
[
{
"country": "US",
"country_count": 737
},
{
"country": "ZA",
"country_count": 148
},
{
"country": "GB",
"country_count": 105
},
{
"device": "desktop",
"device_count": 2287
},
{
"device": "mobile",
"device_count": 1574
},
{
"device": "tablet",
"device_count": 1139
},
{
"browser_name": "Chrome",
"browser_name_count": 1569
},
{
"browser_name": "Safari",
"browser_name_count": 1496
},
{
"browser_name": "Firefox",
"browser_name_count": 1139
}
]
```
---
## GroupBy
##### `groupBy(string|array $field)`
### Basic Usage
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->groupBy(['user_id'])
->get();
```
> Retrieves All the unique user_ids of users logged in the last 30 days
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-22T09:20:46+00:00"
}
}
},
"size": 0,
"aggs": {
"group_by": {
"composite": {
"sources": [
{
"user_id": {
"terms": {
"field": "user_id.keyword"
}
}
}
]
}
}
}
},
"_source": [
"*"
]
}
```
### GroupBy take and skip
If no limit is provided, then the aggregation will return the first 10 results as per Elasticsearch default.
You can set the limit to whatever you want, but it is not necessary to cover all the hits of your query since the composite is pagable.
```php
UserLog::groupBy(['user_id'])
->take(15)->skip(4)
->get();
```
```json
{
"index": "user_logs",
"body": {
"size": 0,
"aggs": {
"group_by": {
"composite": {
"sources": [
{
"user_id": {
"terms": {
"field": "user_id.keyword"
}
}
}
],
"size": 15,
"after": {
"user_id": "158e6cc9-fa3e-3258-99c2-571d052e923c"
}
}
}
}
},
"_source": [
"*"
]
}
```
### Multiple Fields
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->groupBy(['user_id','status'])
->get();
```
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-22T11:26:48+00:00"
}
}
},
"size": 0,
"aggs": {
"group_by": {
"composite": {
"sources": [
{
"user_id": {
"terms": {
"field": "user_id.keyword"
}
}
},
{
"status": {
"terms": {
"field": "status"
}
}
}
]
}
}
}
},
"_source": [
"*"
]
}
```
### GroupBy with Relations
> **Tip:**
> A common use case is to load related data from the grouped results
>
> **Ensure that you have selected the foreign key in the query to load the related data.**
>
The results from `groupBy()` queries are returned as [ElasticCollections](https://elasticsearch.pdphilip.com/eloquent/the-base-model#elastic-collections), enabling the use of Laravel's rich collection methods for further manipulation or processing.
Example: User has many user logs:
```php
// Loading related user data from the distinct user_ids
$users = UserLog::with('user')
->where('created_at', '>=', Carbon::now()->subDays(30))
->groupBy(['user_id'])
->get();
```
> Loads the related user data for the distinct user_ids
### GroupBy Pagination with Relations
Example:
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->groupBy(['user_id'])
->paginate(10)->through(function ($userLog) {
$userLog->load('user');
return $userLog;
}
);
```
---
## GroupBy Ranges
[Version 5.3+]
##### `groupByRanges(string $field, array $ranges)`
`groupByRanges()` performs a [range aggregation](https://www.elastic.co/docs/reference/aggregations/search-aggregations-bucket-range-aggregation) on the specified field.
This is useful to count records that fall within specific ranges, or aggregate data based on those ranges.
### Range parameter
The range paramter is an array of ranges. Each range can be defined as:
- An associative array with `from` and/or `to` keys with optional `key` key for naming the range
- An array of `[$from, $to]` values
```php
$ranges = [
[
'key' => 'low-stock',
'to' => 5,
],
[
'key' => 'medium-stock',
'from' => 5,
'to' => 15,
],
[
'key' => 'high-stock',
'from' => 15,
]
];
$groups = Item::groupByRanges('stock', $ranges)
->get();
```
```json
{
"index": "items",
"body": {
"size": 0,
"aggs": {
"stock_range": {
"range": {
"field": "stock",
"ranges": [
{
"key": "low-stock",
"to": 5
},
{
"key": "medium-stock",
"from": 5,
"to": 15
},
{
"key": "high-stock",
"from": 15
}
]
}
}
}
},
"_source": [
"*"
]
}
```
```json
[
{
"count_stock_range_low-stock": 2,
},
{
"count_stock_range_medium-stock": 6
},
{
"count_stock_range_high-stock": 2
}
]
```
### GroupBy Ranges with aggregations
You can also perform sub-aggregations on the grouped ranges. For example, to get the average, min and max price of items in each stock range:
```php
$ranges = [
[
'key' => 'low-stock',
'to' => 5,
],
[
'key' => 'medium-stock',
'from' => 5,
'to' => 15,
],
[
'key' => 'high-stock',
'from' => 15,
]
];
$groups = Item::groupByRanges('stock', $ranges)
->agg(['min', 'max', 'avg'], 'price');
```
```json
{
"index": "items",
"body": {
"size": 0,
"aggs": {
"stock_range": {
"range": {
"field": "stock",
"ranges": [
{
"to": 5,
"key": "low-stock"
},
{
"from": 5,
"to": 15,
"key": "medium-stock"
},
{
"from": 15,
"key": "high-stock"
}
]
},
"aggs": {
"min_price": {
"min": {
"field": "price"
}
},
"max_price": {
"max": {
"field": "price"
}
},
"avg_price": {
"avg": {
"field": "price"
}
}
}
}
}
}
}
```
```json
[
{
"count_stock_range_low-stock": 2,
"min_price_stock_range_low-stock": 8,
"max_price_stock_range_low-stock": 1500,
"avg_price_stock_range_low-stock": 754
},
{
"count_stock_range_medium-stock": 6,
"min_price_stock_range_medium-stock": 6,
"max_price_stock_range_medium-stock": 900,
"avg_price_stock_range_medium-stock": 173.83333333333334
},
{
"count_stock_range_high-stock": 2,
"min_price_stock_range_high-stock": 3,
"max_price_stock_range_high-stock": 350,
"avg_price_stock_range_high-stock": 176.5
}
]
```
> **Note:**
> Aggregations include: `sum`, `avg`, `min`, `max`, `agg`, `matrix`, `boxplot`,`stats`,`extendedStats`, `cardinality`, `medianAbsoluteDeviation`, `stringStats` and `percentiles`.
## GroupBy Date Ranges
[Version 5.3+]
##### `groupByDateRanges(string $field, array $ranges, $options = [])`
`groupByDateRanges()` performs a [date range aggregation](https://www.elastic.co/docs/reference/aggregations/search-aggregations-bucket-daterange-aggregation) on the specified field.
Similar to `groupByRanges()`, but specifically for date fields where the `from` and `to` values can be expressed in date math format. It also accepts an optional `$options` array to specify date formats.
```php
$ranges = [
['to' => 'now-10M/M'],
['from' => 'now-10M/M'],
];
$options = ['format' => 'MM-yyyy'];
UserSession::groupByDateRanges('created_at', $ranges, $options)->get();
```
```json
{
"index": "user_sessions",
"body": {
"size": 0,
"aggs": {
"created_at_range": {
"date_range": {
"field": "created_at",
"format": "MM-yyyy",
"ranges": [
{
"to": "now-10M/M"
},
{
"from": "now-10M/M"
}
]
}
}
}
}
}
```
## Raw bucket values from meta
[Version 5.3+]
Since each `distinct()`, `bulkDistinct()`,`groupBy()`, `groupByRanges()`, and `groupByDateRanges()` are returned as an ElasticCollections, you can access the bucket meta values using the `getMetaValue('bucket')` method.
```php
$ranges = [
[
'key' => 'low-stock',
'to' => 5,
],
[
'key' => 'medium-stock',
'from' => 5,
'to' => 15,
],
[
'key' => 'high-stock',
'from' => 15,
]
];
$groups = Item::groupByRanges('stock', $ranges)
->agg(['min', 'max', 'avg'], 'price');
$buckets = $groups->map(function ($group) {
return $group->getMetaValue('bucket');
});
```
```json
// return $buckets;
[
{
"key": "low-stock",
"to": 5,
"doc_count": 2,
"min_price": {
"value": 8
},
"avg_price": {
"value": 754
},
"max_price": {
"value": 1500
}
},
{
"key": "medium-stock",
"from": 5,
"to": 15,
"doc_count": 6,
"min_price": {
"value": 6
},
"avg_price": {
"value": 173.83333333333334
},
"max_price": {
"value": 900
}
},
{
"key": "high-stock",
"from": 15,
"doc_count": 2,
"min_price": {
"value": 3
},
"avg_price": {
"value": 176.5
},
"max_price": {
"value": 350
}
}
]
```
---
## Nested Queries
Nested queries are a powerful feature of Elasticsearch that allows you to search within nested objects or arrays. This feature is especially useful when working with complex data structures that require deep search capabilities.
***
## Where Nested Object
This method allows you to query nested objects within an Elasticsearch document.
Nested fields are useful when you need to store large arrays of objects without flattening them into a single array or overindexing the parent index.
```php
BlogPost::whereNestedObject('comments', function (Builder $query) {
$query->where('comments.country', 'Peru')->where('comments.likes', 5);
})->get();
```
> This will return all the blog posts where the comments field contains an object with the country field set to 'Peru' and the likes field set to 5.
```json
{
"index": "blog_posts",
"body": {
"query": {
"nested": {
"path": "comments",
"query": {
"bool": {
"must": [
{
"term": {
"comments.country.keyword": {
"value": "Peru"
}
}
},
{
"term": {
"comments.likes": {
"value": 5
}
}
}
]
}
}
}
},
"size": 1000
}
}
```
> **Note:**
> Make sure the field is declared as `nested` in your index mapping. Otherwise, `whereNestedObject()` will return no results.
>
> ```php
> Schema::create('blog_posts', function (Blueprint $index) {
> $index->nested('comments');
> });
> ```
> **Note:**
> You may omit the path prefix (`comments.`) inside the closure for cleaner syntax.
>
> ```php
> $posts = BlogPost::whereNestedObject('comments', function (Builder $query) {
> $query->where('country', 'Peru')->where('likes', 5);
> // 'country' points to 'comments.country'
> // 'likes' points to 'comments.likes'
> })->get();
> ```
> ```json
> {
> "index": "blog_posts",
> "body": {
> "query": {
> "nested": {
> "path": "comments",
> "query": {
> "bool": {
> "must": [
> {
> "term": {
> "comments.country.keyword": {
> "value": "Peru"
> }
> }
> },
> {
> "term": {
> "comments.likes": {
> "value": 5
> }
> }
> }
> ]
> }
> }
> }
> },
> "size": 1000
> }
> }
> ```
***
## Where Nested Object with Filtered Inner Hits
- Option 1:
```php
MyModel::whereNestedObject($field, $closureQuery, true)
```
- Option 2:
```php
MyModel::filterNested($field, $closureQuery)
```
> **Note:**
> When Filtering Inner Hits you can include ordering and limiting the inner hits.
Example:
```php
// Query nested and filter inner hits
BlogPost::whereNestedObject('comments', function (Builder $query) {
$query->where('comments.country', 'Peru')
->where('comments.likes', 5)
->orderBy('likes', 'desc')
->limit(5);
},true)->get();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"nested": {
"path": "comments",
"query": {
"bool": {
"must": [
{
"term": {
"comments.country.keyword": {
"value": "Peru"
}
}
},
{
"term": {
"comments.likes": {
"value": 5
}
}
}
]
}
},
"inner_hits": {
"sort": [
{
"comments.likes": {
"order": "desc"
}
}
],
"size": 5
}
}
},
"size": 1000
}
}
```
## Note on Nested limits
The default max limit for Elasticsearch on nested values when using `inner_hits` is `100`. Trying to set a limit higher than that will result in an error.
You can change the limit by setting the `max_inner_result_window` value in your Elasticsearch configuration.
```php
Schema::create('blog_posts', function (IndexBlueprint $index) {
$index->nested('comments');
$index->settings('max_inner_result_window', 200);
});
```
Then you can:
```php
BlogPost::where('status', 5)->whereNestedObject('comments', function ($query) {
$query->orderByDesc('likes')->limit(200);
},true)->orderBy('created_at')->first();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 5
}
}
},
{
"nested": {
"path": "comments",
"query": {
"exists": {
"field": "comments"
}
},
"inner_hits": {
"sort": [
{
"comments.likes": {
"order": "desc"
}
}
],
"size": 200
}
}
}
]
}
},
"sort": [
{
"created_at": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
***
## Where Not Nested Object
Similar to `whereNestedObject`, this method allows you to query nested objects within an Elasticsearch document.
However, it **excludes documents that match the specified nested object query**.
```php
BlogPost::whereNotNestedObject('comments', function (Builder $query) {
$query->where('comments.country', 'Peru');
})->get();
```
> This will return all the blog posts where there are no comments from Peru
```json
{
"index": "blog_posts",
"body": {
"query": {
"bool": {
"must_not": [
{
"nested": {
"path": "comments",
"query": {
"term": {
"comments.country.keyword": {
"value": "Peru"
}
}
},
}
}
]
}
},
"size": 1000
}
}
```
***
## Where Not Nested Object with Filtered Inner Hits
Follow up the `whereNotNestedObject` clause with a `whereNestedObject` set to true `true` and an inner query for the desired filter
```php
BlogPost::whereNotNestedObject('comments', function (Builder $query) {
$query->where('country', 'Peru');
})->whereNestedObject('comments', function (Builder $query) {
$query->orderBy('likes', 'desc')->limit(5);
},true)->get();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"bool": {
"must_not": [
{
"nested": {
"path": "comments",
"query": {
"term": {
"comments.country.keyword": {
"value": "Peru"
}
}
}
}
}
],
"must": [
{
"nested": {
"path": "comments",
"query": {
"exists": {
"field": "comments"
}
},
"inner_hits": {
"sort": [
{
"comments.likes": {
"order": "desc"
}
}
],
"size": 5
}
}
}
]
}
},
"size": 1000
}
}
```
## Order By Nested Field
This method allows you to order the results of a query by a nested field.
This is useful when you need to sort the results of a query based on a nested field.
Parameters scope: `orderByNested($field, $direction = 'asc', $mode = 'avg')`
> **Note:**
> `orderByNested` orders the parent documents based on the nested field values, not the inner hits.
```php
BlogPost::where('status', 5)->orderByNested('comments.likes', 'desc', 'sum')->limit(5)->get();
```
> This will return all the blog posts where the status is 5, ordered by the sum of the likes of the comments in descending order.
```json
{
"index": "blog_posts",
"body": {
"query": {
"term": {
"status": {
"value": 5
}
}
},
"sort": [
{
"comments.likes": {
"mode": "avg",
"nested": {
"path": "comments"
},
"order": "asc"
}
}
],
"size": 5
}
}
```
***
## Filtered Inner Hits Examples
### 1. No Filtering:
```php
BlogPost::where('status', 5)->orderBy('created_at')->first();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"term": {
"status": {
"value": 5
}
}
},
"sort": [
{
"created_at": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
return example:
```json
{
"_id": "1IiLKG38BCOXW3U9a4zcn",
"title": "My first post",
"status": 5,
"comments": [
{
"name": "Damaris Ondricka",
"comment": "Quia quis facere cupiditate unde natus dolorem. Quia voluptatem in nam occaecati. Veritatis libero neque vitae.",
"country": "Peru",
"likes": 6
},
{
"name": "Cole Beahan",
"comment": "Officia ut dolorem itaque sapiente repellendus consequatur. Voluptas veniam quis eligendi. Aliquid voluptatem reiciendis ut.",
"country": "Sweden",
"likes": 0
},
{
"name": "April Von",
"comment": "Repudiandae rem aspernatur neque molestiae voluptatibus ut aut. Animi dolor id voluptas. Blanditiis a est nobis voluptatem sed sed illum esse.",
"country": "Switzerland",
"likes": 10
},
{
"name": "Ella Ruecker",
"comment": "Et deleniti ab cumque nobis ut ullam. Exercitationem qui sequi voluptatem delectus sunt nobis. Vel libero nihil quas inventore omnis. Harum corrupti consequatur quibusdam ut.",
"country": "UK",
"likes": 5
},
{
"name": "Mabelle Schinner",
"comment": "Aliquid molestiae quas vitae ipsam neque nam sed. Facere blanditiis repellendus sequi autem. Explicabo cupiditate porro quia animi ut minus tempora ut.",
"country": "Switzerland",
"likes": 7
}
]
}
```
### 2. Filter comments from Switzerland:
```php
BlogPost::where('status', 5)->whereNestedObject('comments', function ($query) {
$query->where('country', 'Switzerland'); //or comments.country
},true)->orderBy('created_at')->first();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 5
}
}
},
{
"nested": {
"path": "comments",
"query": {
"term": {
"comments.country.keyword": {
"value": "Switzerland"
}
}
},
"inner_hits": {}
}
}
]
}
},
"sort": [
{
"created_at": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
returns:
```json
{
"_id": "1IiLKG38BCOXW3U9a4zcn",
"title": "My first post",
"status": 5,
"comments": [
{
"name": "April Von",
"comment": "Repudiandae rem aspernatur neque molestiae voluptatibus ut aut. Animi dolor id voluptas. Blanditiis a est nobis voluptatem sed sed illum esse.",
"country": "Switzerland",
"likes": 10
},
{
"name": "Mabelle Schinner",
"comment": "Aliquid molestiae quas vitae ipsam neque nam sed. Facere blanditiis repellendus sequi autem. Explicabo cupiditate porro quia animi ut minus tempora ut.",
"country": "Switzerland",
"likes": 7
}
]
}
```
### 3. Filter comments from Switzerland ordered by likes:
```php
BlogPost::where('status', 5)->whereNestedObject('comments', function ($query) {
$query->where('country', 'Switzerland')->orderBy('likes');
},true)->orderBy('created_at')->first();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 5
}
}
},
{
"nested": {
"path": "comments",
"query": {
"term": {
"comments.country.keyword": {
"value": "Switzerland"
}
}
},
"inner_hits": {
"sort": [
{
"comments.likes": {
"order": "asc"
}
}
]
}
}
}
]
}
},
"sort": [
{
"created_at": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
returns:
```json
{
"_id": "1IiLKG38BCOXW3U9a4zcn",
"title": "My first post",
"status": 5,
"comments": [
{
"name": "Mabelle Schinner",
"comment": "Aliquid molestiae quas vitae ipsam neque nam sed. Facere blanditiis repellendus sequi autem. Explicabo cupiditate porro quia animi ut minus tempora ut.",
"country": "Switzerland",
"likes": 7
},
{
"name": "April Von",
"comment": "Repudiandae rem aspernatur neque molestiae voluptatibus ut aut. Animi dolor id voluptas. Blanditiis a est nobis voluptatem sed sed illum esse.",
"country": "Switzerland",
"likes": 10
}
]
}
```
### 4. Filter comments with likes greater than 5:
```php
BlogPost::where('status', 5)->whereNestedObject('comments', function ($query) {
$query->where('likes', '>', 5);
},true)->orderBy('created_at')->first();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 5
}
}
},
{
"nested": {
"path": "comments",
"query": {
"range": {
"comments.likes": {
"gt": 5
}
}
},
"inner_hits": {}
}
}
]
}
},
"sort": [
{
"created_at": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
returns:
```json
{
"_id": "1IiLKG38BCOXW3U9a4zcn",
"title": "My first post",
"status": 5,
"comments": [
{
"name": "Damaris Ondricka",
"comment": "Quia quis facere cupiditate unde natus dolorem. Quia voluptatem in nam occaecati. Veritatis libero neque vitae.",
"country": "Peru",
"likes": 6
},
{
"name": "April Von",
"comment": "Repudiandae rem aspernatur neque molestiae voluptatibus ut aut. Animi dolor id voluptas. Blanditiis a est nobis voluptatem sed sed illum esse.",
"country": "Switzerland",
"likes": 10
},
{
"name": "Mabelle Schinner",
"comment": "Aliquid molestiae quas vitae ipsam neque nam sed. Facere blanditiis repellendus sequi autem. Explicabo cupiditate porro quia animi ut minus tempora ut.",
"country": "Switzerland",
"likes": 7
}
]
}
```
### 4. Filter comments with likes greater than or equal to 5, limit 2:
```php
BlogPost::where('status', 5)->whereNestedObject('comments', function ($query) {
$query->where('likes', '>=', 5)->limit(2);
},true)->orderBy('created_at')->first();
```
```json
{
"index": "blog_posts",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 5
}
}
},
{
"nested": {
"path": "comments",
"query": {
"range": {
"comments.likes": {
"gte": 5
}
}
},
"inner_hits": {
"size": 2
}
}
}
]
}
},
"sort": [
{
"created_at": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
returns:
```json
{
"_id": "1IiLKG38BCOXW3U9a4zcn",
"title": "My first post",
"status": 5,
"comments": [
{
"name": "Damaris Ondricka",
"comment": "Quia quis facere cupiditate unde natus dolorem. Quia voluptatem in nam occaecati. Veritatis libero neque vitae.",
"country": "Peru",
"likes": 6
},
{
"name": "April Von",
"comment": "Repudiandae rem aspernatur neque molestiae voluptatibus ut aut. Animi dolor id voluptas. Blanditiis a est nobis voluptatem sed sed illum esse.",
"country": "Switzerland",
"likes": 10
}
]
}
```
---
## Ordering and Pagination
In the Laravel-Elasticsearch integration, ordering and pagination are essential features that enable developers to manage and present data effectively. These features are designed to work seamlessly within the Laravel ecosystem, designed to feel native to Laravel developers familiar with Eloquent
***
## Ordering
Elasticsearch inherently ranks search results based on relevance (using internal scoring and ranking algorithms). However, it is often necessary to sort results based on specific fields. The Laravel-Elasticsearch integration provides a simple and intuitive way to sort search results using the `orderBy` and `orderByDesc` methods.
> **Note:**
> You can only sort on `keyword`, `numeric`, or `date` fields. Sorting on **text** fields is not supported by Elasticsearch and will throw an error (No shards available)
***
## OrderBy
The `orderBy` method allows you to specify the field by which the results should be sorted and the direction of the sort (ascending or descending). This method is straightforward and aligns with the Laravel Eloquent's `orderBy` functionality.
```php
Product::orderBy('status')->get();
```
> Find all products and order them by their status in ascending order.
```json
{
"index": "products",
"body": {
"sort": [
{
"status": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
```php
Product::orderBy('created_at', 'desc')->get();
```
> Find all products and order them by creation date in descending order.
```json
{
"index": "products",
"body": {
"sort": [
{
"created_at": {
"order": "desc"
}
}
],
"size": 1000
}
}
```
If you have a field that is mapped as a `text` field with a `keyword` subfield, the package will automatically use the `keyword` subfield for sorting.
```php
Product::orderBy('name')->get();
```
> Find all products and order them by their name in ascending order via the keyword subfield.
```json
{
"index": "products",
"body": {
"sort": [
{
"name.keyword": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
### OrderByDesc
As with Laravel's standard eloquent, the `orderByDesc` method is provided to quickly sort results in descending order by a specified field, without needing to explicitly set the direction.
```php
Product::orderByDesc('created_at')->get();
```
> Find all products and order them by creation date in descending order.
```json
{
"index": "products",
"body": {
"sort": [
{
"created_at": {
"order": "desc"
}
}
],
"size": 1000
}
}
```
***
## Offset & Limit (skip & take)
As with Eloquent, you can use the `skip` and `take` methods in your query.
```php
Product::skip(10)->take(5)->get();
```
> Find all products and skip the first 10, then take the next 5.
```json
{
"index": "products",
"body": {
"from": 10,
"size": 5
}
}
```
***
## Pagination
Pagination works as expected in this Laravel-Elasticsearch integration.
```php
$products = Product::where('is_active',true)
$products = $products->paginate(50)
```
> Find all active products and paginate them with 50 products per page.
Pagination links (Blade)
```php
{{ $products->appends(request()->query())->links() }}
```
***
## Extending ordering for Elasticsearch features
The `orderBy` and `orderByDesc` methods are designed to be Laravel native, but they can be extended to support more advanced Elasticsearch features.
Full parameter scope:
* `orderBy($field, $direction = 'asc', $options = [])`
* `orderByDesc($field, $options = [])`
### $mode
##### `$options['mode'] = $mode;`
The `mode` parameter allows you to specify how Elasticsearch should handle sorting when multiple documents have the same value for the field being sorted.
Options: `min`, `max`, `sum`, `avg`, `median`
Default: `min` when sorting in ascending order, `max` when sorting in descending order.
### $missing
##### `$options['missing'] = $missing;`
Default: `_last`
The `missing` parameter allows you to specify how Elasticsearch should handle sorting when a document is missing the field being sorted.
Options: `_first`, `_last`, or a custom value.
Example with options:
```php
$options = [
'mode' => 'avg',
'missing' => '_first'
];
```
```php
// pricing_history is an array of prices ex: [9.99, 15.50, 29, 20.50]
Product::where('is_active',true)->orderBy('pricing_history', $options)->get();
```
> Find all active products and order them by the average price in the pricing history array in descending order.
```json
{
"index": "products",
"body": {
"query": {
"term": {
"is_active": {
"value": true
}
}
},
"sort": [
{
"pricing_history": {
"mode": "avg",
"missing": "_first",
"order": "asc"
}
}
],
"size": 1000
}
}
```
***
## OrderBy Geo Distance
The `orderByGeo` method is a specialized method for sorting by geo distance. This sorting method only works with `geo` fields and can be used to sort results based on their distance from a specified point.
Parameter scope:
* `orderByGeo(string $column, array $pin, $direction = 'asc', $options = [])`
* `orderByGeoDesc(string $column, array $pin, , $options = [])`
### `$pin`
The point(s) from which to calculate the distance. This can be a single point or an array of points.
* A point is an array with two values: `[lon, lat]` **(longitude, latitude) - Order matters** and Elasticsearch does not use the standard `[lat, lon]` format
* For multiple points, use an array of points: `[[lon1, lat1], [lon2, lat2], ...]`
* You can also specify the lat and lon keys: `['lat' => 51.50853, 'lon' => -0.12574]`
### `$direction`
* The direction in which to sort the results. Options: `asc`, `desc`
* `asc` sorts by shortest distance from the pin
* `desc` sorts by longest distance from the pin
### `$options`
```php
$options = [
'unit' => $unit,
'mode' => $mode,
'distance_type' => $type
];
```
- #### `$unit`
- The unit to use when computing sort values. Options: `m`, `km`, `mi`, `yd`, `ft`
- #### `$mode`
* Used for when a field has several geo points. By default, the **shortest distance is used when sorting in ascending order** and the **longest distance when sorting in descending order**.
* Options: `min`, `max`, `sum`, `avg`, `median`
- #### `$type`
* The type of distance calculation to use.
* Options: `arc`, `plane`
* Note: `plane` is faster, but inaccurate on long distances and close to the poles.
```php
// Lat:51.50853 & Lon:-0.12574 (London)
Product::where('is_active',true)->orderByGeo('manufacturer.location', [-0.12574, 51.50853])->get();
// OR
Product::where('is_active',true)->orderByGeo('manufacturer.location', ['lat' => 51.50853, 'lon' => -0.12574])->get();
```
> Find all active products and order them by the closest distance of the manufacturer's location to London.
```json
{
"index": "products",
"body": {
"query": {
"term": {
"is_active": {
"value": true
}
}
},
"sort": [
{
"_geo_distance": {
"manufacturer.location": [
-0.12574,
51.50853
],
"order": "asc"
}
}
],
"size": 1000
}
}
```
#### With options
```php
// Lat:48.85341 & Lon:2.3488 (Paris)
// Lat:51.50853 & Lon:-0.12574 (London)
$options = [
'unit' => 'km',
'mode' => 'avg',
'distance_type' => 'plane'
];
Product::where('is_active',true)->orderByGeoDesc('manufacturer.location', [[2.3488, 48.85341], [-0.12574, 51.50853]], $options)->get();
//Or
Product::where('is_active',true)->orderByGeoDesc('manufacturer.location', [['lat' => 48.85341, 'lon' => 2.3488], ['lat' => 51.50853, 'lon' => -0.12574]], $options)->get();
```
> Find all active products and order them by the longest average distance of the manufacturer's location to Paris and London.
```json
{
"index": "products",
"body": {
"query": {
"term": {
"is_active": {
"value": true
}
}
},
"sort": [
{
"_geo_distance": {
"manufacturer.location": [
[
2.3488,
48.85341
],
[
-0.12574,
51.50853
]
],
"order": "desc",
"unit": "km",
"distance_type": "plane",
"mode": "avg"
}
}
],
"size": 1000
}
}
```
***
---
## Chunking
In scenarios where you're dealing with large datasets, Laravel's chunk method becomes invaluable, allowing you to process large sets of results in manageable portions. This approach is particularly useful in memory-intensive operations, such as batch updates or exports, where loading the entire dataset into memory at once could lead to performance issues. The Laravel-Elasticsearch integration adapts this familiar concept from Laravel Eloquent ORM to work within the context of Elasticsearch data.
---
## Basic Chunking
The `chunk` method breaks down the query results into smaller "chunks" of a specified size, executing a callback function for each chunk. This allows you to operate on a subset of the results at a time, reducing memory usage.
```php
Product::chunk(1000, function ($products) {
foreach ($products as $product) {
// Example operation: Increase price by 10%
$product->price *= 1.1;
$product->save();
}
});
```
> Retrieves products in batches of 1000. For each batch, it iterates over each
> product, applying a 10% price increase and then saving the changes.
---
## Chunking, Under the Hood - PIT
- The chunking method uses Elasticsearch's `Point In Time` (PIT) to iterate over the results. This allows for more efficient pagination and avoids potential issues that may arise when records change during the operation.
- The default ordering will be `_shard_doc` in ascending order. You can introduce your own ordering by using the `orderBy` method before calling `chunk` however, this may affect the grouping of results.
- Once the chunking operation is complete, the PIT will be automatically closed.
---
## Chunk By Id
`chunkById($count, callable $callback, $column = '_id', $alias = null)`
The `chunkById()` method is a specialized version of the `chunk` method that paginates results based on a unique identifier **ensuring that each chunk contains unique records**.
Any ordering clauses will be ignored as this method uses the unique identifier to paginate the results.
If no identifier is provided (default value `id`), the chunk will use PIT ordered by `_shard_doc` in ascending order (irrespective of an order clause if present).
```php
Product::chunkById(1000, function ($products) {
foreach ($products as $product) {
// Example operation: Increase price by 10%
$product->price *= 1.1;
$product->save();
}
}, 'product_sku.keyword');
```
> Retrieves products in batches of 1000, using the `product_sku` field as the
> unique identifier. For each batch, it iterates over each product, applying a
> 10% price increase and then saving the changes.
---
---
## Dynamic Indices
In certain scenarios, it may be necessary to distribute a model's data across multiple Elasticsearch indices to optimize performance, manage data volume, or comply with specific data storage requirements. The Laravel-Elasticsearch integration accommodates this need through the support for dynamic indices, allowing a single model to interact with a range of indices following a consistent naming pattern.
***
## Implementing Dynamic Indices
To enable dynamic indexing, include the `DynamicIndex` trait in your model. This trait provides the necessary functionality to work with dynamic indices.
```php
namespace App\Models;
+use PDPhilip\Elasticsearch\Eloquent\DynamicIndex;
use PDPhilip\Elasticsearch\Eloquent\Model;
class PageHit extends Model
{
+ use DynamicIndex;
protected $connection = 'elasticsearch';
}
```
In this example, the PageHit model is configured to work with indices that match the pattern `page_hits*`, enabling operations across all indices starting with page_hits.
***
## Creating Records with Dynamic Indices
> **Caution:**
> When creating new records, **you must explicitly set the target index** using the `setSuffix` method to specify the exact index where the record should be stored.
```php
$pageHit = new PageHit;
$pageHit->page_id = 4;
$pageHit->ip = $someIp;
// Set the specific index for the new record
$pageHit->setSuffix('_' . date('Y-m-d'));
$pageHit->save();
```
This pattern ensures that new records are stored in the appropriate index based on your application's logic, such as using the current date to determine the index name.
***
## Retrieving the Current Record's Suffix
Each model instance associated with a dynamic index retains knowledge of the specific index it pertains to.
In some cases, you may need to know the suffix for a given record. To retrieve the current record's index suffix, use the `getRecordSuffix` method
```php
$indexSuffix = $pageHit->getRecordSuffix(); //returns "_2025-01-01"
```
## Querying Across Dynamic Indices
When querying a model with dynamic indices, the query will span all indices that match the defined pattern, allowing for aggregated data retrieval. This means that you can query your model without specifying the exact index, and the query will automatically search across all matching indices.
```php
// Queries all indices matching "page_hits*"
PageHit::where('page_id', 1)->get();
```
> Retrieve page hits for page_id 1 across all 'page_hits*' indices
```json
{
"index": "page_hits*",
"body": {
"query": {
"term": {
"page_id": {
"value": 1
}
}
},
"size": 10000
}
}
```
***
## Targeting a Specific Index
Use `withSuffix()` to limit queries to a single dynamic index:
```php
PageHit::withSuffix('_2023-01-01')->where('page_id', 3)->get();
```
> Retrieve page hits for page_id 3 from the `page_hits_2023-01-01` index only
```json
{
"index": "page_hits_2023-01-01",
"body": {
"query": {
"term": {
"page_id": {
"value": 3
}
}
},
"size": 10000
}
}
```
***
---
## Model Relationships in Elasticsearch
In Laravel applications, model relationships are crucial for structuring complex data and interactions between different entities. The Laravel-Elasticsearch integration supports a variety of relationship types, enabling Elasticsearch documents to relate to each other similarly to how models relate in traditional relational databases. This section provides a comprehensive guide on implementing Elasticsearch-to-Elasticsearch model relationships within a Laravel application.
***
## Defining Relationships
Just like in a traditional Laravel Eloquent model, you can define `belongsTo`, `hasMany`, `hasOne`, `morphOne`, and `morphMany` relationships in models that use Elasticsearch as their data storage. Here's a full example illustrating various relationship types in an Elasticsearch context:
***
## Common relationships example
Using `belongsTo`, `hasMany`, `hasOne`, `morphOne`, and `morphMany` relationships.
### Relationship Diagram
Illustrates how different models can interconnect within Elasticsearch, maintaining structured data interactions without the need for traditional joins.
[See diagram in the online documentation]
### Company Model
```php
/**
* App\Models\Company
*
******Fields*******
* @property string $_id
* @property string $name
* @property integer $status
* @property \Illuminate\Support\Carbon|null $created_at
* @property \Illuminate\Support\Carbon|null $updated_at
*
******Relationships*******
* @property-read CompanyLog $companyLogs
* @property-read CompanyProfile $companyProfile
* @property-read Avatar $avatar
* @property-read Photos $photos
*
* @mixin \Eloquent
*
*/
class Company extends Model
{
protected $connection = 'elasticsearch';
//Relationships =====================================
public function companyLogs()
{
return $this->hasMany(CompanyLog::class);
}
public function companyProfile()
{
return $this->hasOne(CompanyProfile::class);
}
public function avatar()
{
return $this->morphOne(Avatar::class, 'imageable');
}
public function photos()
{
return $this->morphMany(Photo::class, 'photoable');
}
}
```
### CompanyLog Model
```php
/**
* App\Models\CompanyLog
*
******Fields*******
* @property string $_id
* @property string $company_id
* @property string $title
* @property integer $code
* @property mixed $meta
* @property Carbon|null $created_at
* @property Carbon|null $updated_at
*
******Relationships*******
* @property-read Company $company
*
* @mixin \Eloquent
*
*/
class CompanyLog extends Model
{
protected $connection = 'elasticsearch';
//Relationships =====================================
public function company()
{
return $this->belongsTo(Company::class);
}
}
```
### Avatar Model
```php
/**
* App\Models\Avatar
*
******Fields*******
* @property string $_id
* @property string $url
* @property string $imageable_id
* @property string $imageable_type
* @property \Illuminate\Support\Carbon|null $created_at
* @property \Illuminate\Support\Carbon|null $updated_at
*
******Relationships*******
* @property-read Company $company
*
* @mixin \Eloquent
*
*/
class Avatar extends Model
{
protected $connection = 'elasticsearch';
//Relationships =====================================
public function imageable()
{
return $this->morphTo();
}
}
```
### Photo Model
```php
/**
* App\Models\Photo
*
******Fields*******
* @property string $_id
* @property string $url
* @property string $photoable_id
* @property string $photoable_type
* @property \Illuminate\Support\Carbon|null $created_at
* @property \Illuminate\Support\Carbon|null $updated_at
*
******Relationships*******
* @property-read Company $company
*
* @mixin \Eloquent
*
*/
class Photo extends Model
{
protected $connection = 'elasticsearch';
//Relationships =====================================
public function photoable()
{
return $this->morphTo();
}
}
```
### Example Usage
```php
$company = Company::first();
$companyLogs = $company->companyLogs->toArray(); //Shows all company logs (has many)
$companyProfile = $company->companyProfile->toArray(); //Shows the company profile (has one)
$companyAvatar = $company->avatar->toArray(); //Shows the company avatar (morph one)
$companyPhotos = $company->photos->toArray(); //Shows the company photos (morph many)
```
This comprehensive example demonstrates the application of various relationship types in a Laravel-Elasticsearch context, providing insights into effective data structuring for complex applications.
***
## Many to Many (belongsToMany)
### Relationship Diagram
The following diagram illustrates how a many-to-many (polymorphic) relationship functions in Elasticsearch:
[See diagram in the online documentation]
### Migration
```php
public function up(): void
{
Schema::create('users', function (Blueprint $index): void {
$index->text('name', hasKeyword: true);
});
Schema::create('roles', function (Blueprint $index): void {
$index->text('name', hasKeyword: true);
});
// Pivot index, automatically managed by belongsToMany from the User and Role models
Schema::create('role_user', function (Blueprint $index): void {
$index->keyword('role_id');
$index->keyword('user_id');
});
}
```
### User Model
```php
/**
* App\Models\User
*
******Fields*******
* @property string $_id
* @property string $name
*
******Relationships*******
* @property-read Role[] $roles
*/
class User extends Model
{
protected $connection = 'elasticsearch';
public function roles()
{
return $this->belongsToMany(Role::class);
}
}
```
### Role Model
```php
/**
* App\Models\Role
*
******Fields*******
* @property string $_id
* @property string $name
*
******Relationships*******
* @property-read User[] $users
*/
class Role extends Model
{
protected $connection = 'elasticsearch';
public function users()
{
return $this->belongsToMany(User::class);
}
}
```
### Example Usage
```php
// Returing relationships
$user = User::first();
$roles = $user->roles->toArray(); //Shows all roles (belongs to many)
$role = Role::first();
$users = $role->users->toArray(); //Shows all users (belongs to many)
// Creating relationships
$user = User::first();
$user->roles()->attach([$roleId1,$roleId2]); //Attach roles to a user
$user->roles()->create(['name' => 'Can sing']); //Create a role and attach to user
```
## One to Many (Polymorphic)
The one-to-many relationship typically involves a single parent model (e.g., a post) linked to multiple child models (e.g., comments). This relationship is crucial for scenarios where an entity can own or be associated with multiple other entities.
### Relationship Diagram
Illustrates a straightforward one-to-many relationship between posts and their comments in an Elasticsearch environment.
[See diagram in the online documentation]
### Index Structure
Documents in Elasticsearch are designed to support relationships directly within their structures:
```text
posts
_id - objectid
name - keyword / text
videos
_id - objectid
name - keyword / text
comments
_id - objectid
body - text
commentable_id - keyword
commentable_type - keyword
```
### Migration
The schema for posts and comments is defined to facilitate direct referencing between a post and its associated comments:
```php
public function up(): void
{
Schema::create('posts', function (Blueprint $index): void {
$index->text('name', hasKeyword: true);
});
Schema::create('videos', function (Blueprint $index): void {
$index->text('name', hasKeyword: true);
});
Schema::create('comments', function (Blueprint $index): void {
$index->text('body');
$index->keyword('commentable_id');
$index->keyword('commentable_type');
});
}
```
### Implementation Details
Managing one-to-many relationships in Laravel with Elasticsearch does not involve complex pivot table operations. Instead, the relationship is maintained through direct document references:
- **Creation**: When creating comments, the `post_id` field is used to link each comment to its respective post.
- **Querying**: To retrieve all comments for a post, Elasticsearch queries are designed to look up all comments where the `post_id` matches the post’s ID.
- **Updates and Deletes**: Modifying or deleting a post or its comments is handled through direct operations on the respective documents.
This approach enhances the performance and scalability of applications by leveraging Elasticsearch's powerful indexing and search capabilities.
## Many to Many (Polymorphic)
In a polymorphic many-to-many setup, each model type can relate to multiple instances of another model type through a shared identifier. This is managed in Elasticsearch by storing arrays of references directly on the model documents, avoiding the use of an additional table.
### Relationship Diagram
The following diagram illustrates how a many-to-many (polymorphic) relationship functions in Elasticsearch:
[See diagram in the online documentation]
### Index Structure
In Elasticsearch, the document structure reflects the flexibility needed to accommodate the polymorphic nature of relationships:
```text
posts
_id - objectid
name - keyword / text
tag_id - keyword
videos
_id - objectid
name - keyword / text
tag_id - keyword
tags
_id - objectid
name - keyword / text
taggables - nested
taggable_id - keyword
taggable_type - keyword
```
Each 'tag' can relate to 'posts' and 'videos' through the `taggables` nested field, which stores multiple relationships in the form of `taggable_id` and `taggable_type`.
### Migration
```php
public function up(): void
{
Schema::create('posts', function (Blueprint $index): void {
$index->text('name', hasKeyword: true);
$index->keyword('tag_id');
});
Schema::create('videos', function (Blueprint $index): void {
$index->text('name', hasKeyword: true);
$index->keyword('tag_id');
});
Schema::create('tags', function (Blueprint $index): void {
$index->text('name', hasKeyword: true);
$index->keyword('tag_id');
$index->nested('taggables');
});
}
```
### Implementation Details
In Laravel, managing these relationships involves standard methods like `attach()`, `detach()`, and `sync()`, which are adapted to work without pivot tables. For example, when you associate a post with a tag, the tag document's `taggables` field is updated to include the post's `_id` and a `taggable_type` of 'post'.
This Elasticsearch-driven approach provides a scalable and efficient way to manage polymorphic relationships, leveraging Elasticsearch's capabilities for handling nested structures and dynamic queries.
---
## Elasticsearch to MySQL Model Relationships
In many applications, Elasticsearch is used in conjunction with a relational database like MySQL. The Laravel-Elasticsearch integration allows you to define hybrid relationships between Elasticsearch and MySQL models seamlessly. This enables a flexible and powerful architecture for applications that need the speed and scalability of Elasticsearch along with the reliability and consistency of a relational database.
***
## Implementing Hybrid Relationships
Hybrid relationships are crucial for applications that leverage both the rapid search capabilities of Elasticsearch and the structured storage of MySQL. Here's how to implement these relationships using the `HybridRelations` trait.
```php
use PDPhilip\Elasticsearch\Eloquent\HybridRelations;
```
***
## Full example
### Relationship Diagram
This diagram visualizes the connections between Elasticsearch and MySQL models, illustrating the flexible data architecture enabled by hybrid relationships.
[See diagram in the online documentation]
### User model (SQL)
```php
use Illuminate\Foundation\Auth\User as Authenticatable;
use PDPhilip\Elasticsearch\Eloquent\HybridRelations;
/**
* App\Models\User
*
* *****Relationships*******
* @property-read UserLog $userLogs
* @property-read UserProfile $userProfile
* @property-read Company $company
* @property-read Avatar $avatar
* @property-read Photo $photos
*/
class User extends Authenticatable
{
use HybridRelations;
protected $connection = 'mysql';
public function userLogs()
{
return $this->hasMany(UserLog::class);
}
public function userProfile()
{
return $this->hasOne(UserProfile::class);
}
public function company()
{
return $this->belongsTo(Company::class);
}
public function avatar()
{
return $this->morphOne(Avatar::class, 'imageable');
}
public function photos()
{
return $this->morphMany(Photo::class, 'photoable');
}
}
```
### UserLog model (Elasticsearch)
```php
/**
* App\Models\UserLog
*
******Fields*******
* @property string $_id
* @property string $company_id
* @property string $title
* @property integer $code
* @property mixed $meta
* @property Carbon|null $created_at
* @property Carbon|null $updated_at
*
******Relationships*******
* @property-read User $user
*/
class UserLog extends Model
{
protected $connection = 'elasticsearch';
public function user()
{
return $this->belongsTo(User::class);
}
}
```
### UserProfile model (Elasticsearch)
```php
/**
* App\Models\UserProfile
*
******Fields*******
* @property string $_id
* @property string $user_id
* @property string $twitter
* @property string $facebook
* @property string $address
* @property \Illuminate\Support\Carbon|null $created_at
* @property \Illuminate\Support\Carbon|null $updated_at
*
******Relationships*******
* @property-read User $user
*
*/
class UserProfile extends Model
{
protected $connection = 'elasticsearch';
public function user()
{
return $this->belongsTo(User::class);
}
}
```
### Company Model (Elasticsearch)
```php
/**
* App\Models\CompanyLog
*
******Fields*******
* @property string $_id
* @property string $name
* @property integer $status
* @property \Illuminate\Support\Carbon|null $created_at
* @property \Illuminate\Support\Carbon|null $updated_at
*
******Relationships*******
* @property-read User $user
*/
class CompanyLog extends Model
{
protected $connection = 'elasticsearch';
public function company()
{
return $this->hasMany(User::class);
}
}
```
### Avatar Model (Elasticsearch)
```php
/**
* App\Models\Avatar
*
******Fields*******
* @property string $_id
* @property string $url
* @property string $imageable_id
* @property string $imageable_type
* @property \Illuminate\Support\Carbon|null $created_at
* @property \Illuminate\Support\Carbon|null $updated_at
*
******Relationships*******
* @property-read Company $company
*/
class Avatar extends Model
{
protected $connection = 'elasticsearch';
public function imageable()
{
return $this->morphTo();
}
}
```
### Photo Model (Elasticsearch)
```php
/**
* App\Models\Photo
*
******Fields*******
* @property string $_id
* @property string $url
* @property string $photoable_id
* @property string $photoable_type
* @property \Illuminate\Support\Carbon|null $created_at
* @property \Illuminate\Support\Carbon|null $updated_at
*
******Relationships*******
* @property-read Company $company
*/
class Photo extends Model
{
protected $connection = 'elasticsearch';
public function photoable()
{
return $this->morphTo();
}
}
```
### Example Usage
Illustrates how to retrieve and interact with data across Elasticsearch and MySQL.
```php
// Retrieve User and related Elasticsearch models
$user = User::first();
$userCompanyName = $user->company->name; //Company name for the user
$userTwitter = $user->userProfile->twitter;
$userAvatar = $user->avatar->url; //Link to Avatar
$userPhotos = $user->photos->toArray(); //Array of photos
// Retrieve UserLog and related MySQL User
$userLog = UserLog::first();
$userName = $userLog->user->name;
// Retrieve Company and related MySQL Users
$company = Company::first();
$companyUsers = $company->users->toArray(); //Array of users
```
This setup highlights the versatility of Laravel's Eloquent ORM in managing complex data relationships across different database systems, ensuring both rapid data access and reliable data integrity.
***
---
## Elasticsearch To Eloquent Mapping
Glossary of common Elasticsearch queries and their Eloquent equivalents, providing a quick reference for developers coming from Elasticsearch to Laravel's Eloquent.
***
## Full Text Queries
### Match query
- #### [whereMatch($field, $value, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-match)
```php
Person::whereMatch('description', 'John Smith')->get();
```
```json
{
"index": "people",
"body": {
"query": {
"match": {
"description": {
"query": "John Smith",
"operator": "and"
}
}
},
"size": 1000
}
}
```
### Match phrase query
- #### [wherePhrase($field, $value, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-phrase)
```php
Person::wherePhrase('description', 'loves espressos')->get();
```
```json
{
"index": "people",
"body": {
"query": {
"match_phrase": {
"description": {
"query": "loves espressos"
}
}
},
"size": 1000
}
}
```
### Match phrase prefix query
- #### [wherePhrasePrefix($field, $value, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-phrase-prefix)
```php
Person::wherePhrasePrefix('description', 'loves es')->get();
```
```json
{
"index": "people",
"body": {
"query": {
"match_phrase_prefix": {
"description": {
"query": "loves es"
}
}
},
"size": 1000
}
}
```
## Term level queries
### Term query
- #### [whereExact($field, $value, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-term)
```php
Person::whereExact('name', 'John Smith')->get();
```
> This will only return the documents where the name field is exactly 'John Smith'. 'john smith' or 'John' will not be returned.
```json
{
"index": "people",
"body": {
"query": {
"term": {
"name.keyword": {
"value": "John Smith"
}
}
},
"size": 1000
}
}
```
### Terms query
- #### [whereIn($field, $values, $options)](https://elasticsearch.pdphilip.com/eloquent/eloquent-queries#wherein)
```php
Product::whereIn('status', [1,5,11])->get();
```
```json
{
"index": "products",
"body": {
"query": {
"terms": {
"status": [
1,
5,
11
]
}
},
"size": 1000
}
}
```
### Range query
- #### [whereBetween($field, [$min,$max], $options)](https://elasticsearch.pdphilip.com/eloquent/eloquent-queries#wherebetween)
```php
Product::whereBetween('in_stock', [10, 100])->get();
```
> Find all products with an in_stock value between 10 and 100 (including 10 and 100)
```json
{
"index": "products",
"body": {
"query": {
"range": {
"status": {
"gte": 10,
"lte": 100
}
}
},
"size": 1000
}
}
```
- #### [where($field, '>', $value, $options)](https://elasticsearch.pdphilip.com/eloquent/eloquent-queries#where)
```php
Product::where('status','>=', 3)->take(10)->get();
```
```json
{
"index": "products",
"body": {
"query": {
"range": {
"status": {
"gte": 3
}
}
},
"size": 10
}
}
```
### Exists query
- #### [whereNotNull($field)](https://elasticsearch.pdphilip.com/eloquent/eloquent-queries#wherenotnull)
```php
Product::whereNotIn('color', ['red','green'])->whereNotNull('color')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must_not": [
{
"terms": {
"color.keyword": [
"red",
"green"
]
}
}
],
"must": [
{
"exists": {
"field": "color"
}
}
]
}
},
"size": 1000
}
}
```
- #### [whereNull($field)](https://elasticsearch.pdphilip.com/eloquent/eloquent-queries#wherenull)
```php
Product::whereNull('color')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "color"
}
}
]
}
},
"size": 1000
}
}
```
### Prefix query
- #### [wherePrefix($field,$value, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-prefix)
```php
Person::wherePrefix('name', 'John Sm')->get();
//Or
Person::whereStartsWith('name', 'John Sm')->get(); //Alias
```
```json
{
"index": "people",
"body": {
"query": {
"prefix": {
"name.keyword": {
"value": "John Sm"
}
}
},
"size": 1000
}
}
```
### Regex query
- #### [whereRegex($field, $pattern, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-regex)
```php
Product::whereRegex('color', 'bl(ue)?(ack)?')->get();
```
> The first example will return documents where the color field matches the pattern 'bl(ue)?(ack)?', which means it can be 'blue' or 'black'. The second example will return documents where the color field matches the pattern 'bl...\*', which means it starts with 'bl' and has at least three more characters. Both should return Blue or Black from the colors field.
```json
{
"index": "products",
"body": {
"query": {
"regexp": {
"color.keyword": {
"value": "bl(ue)?(ack)?"
}
}
},
"size": 1000
}
}
```
### Fuzzy query
- #### [whereFuzzy($field, $value, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-fuzzy)
```php
Product::whereFuzzy('description', 'qick brwn fx')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"fuzzy": {
"description": {
"value": "qick brwn fx"
}
}
},
"size": 1000
}
}
```
### Wildcard query
- #### [where($field, 'like', $value, $options)](https://elasticsearch.pdphilip.com/eloquent/eloquent-queries#where-using-like)
```php
Product::where('color', 'like', 'bl')->orderBy('color.keyword')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"wildcard": {
"color.keyword": {
"value": "*bl*"
}
}
},
"sort": [
{
"color.keyword": {
"order": "asc"
}
}
],
"size": 1000
}
}
```
## Multi-match query
### Best Fields
- #### [searchTerm($term, $options)](https://elasticsearch.pdphilip.com/eloquent/search-queries#search-term)
```php
Book::searchTerm('Eric')->orSearchTerm('Lean')->searchTerm('Startup')->get();
```
```json
{
"index": "books",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"multi_match": {
"query": "Eric",
"type": "best_fields"
}
}
]
}
},
{
"bool": {
"must": [
{
"multi_match": {
"query": "Lean",
"type": "best_fields"
}
},
{
"multi_match": {
"query": "Startup",
"type": "best_fields"
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
### Most Field
- #### [searchTermMost($term, $options)](https://elasticsearch.pdphilip.com/eloquent/search-queries#search-term-most)
```php
Book::searchTermMost('quick brown fox', [ "title", "title.original", "title.shingles" ])->get();
```
```json
{
"index": "books",
"body": {
"query": {
"multi_match": {
"query": "quick brown fox",
"type": "most_fields",
"fields": [
"title",
"title.original",
"title.shingles"
]
}
},
"size": 1000
}
}
```
### Phrase
- #### [searchPhrase($phrase, $options)](https://elasticsearch.pdphilip.com/eloquent/search-queries#search-phrase)
```php
Product::searchPhrase('United States')->orSearchPhrase('United Kingdom')->get();
```
```json
{
"index": "products",
"body": {
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"multi_match": {
"query": "United States",
"type": "phrase"
}
}
]
}
},
{
"bool": {
"must": [
{
"multi_match": {
"query": "United Kingdom",
"type": "phrase"
}
}
]
}
}
]
}
},
"size": 1000
}
}
```
### Phrase Prefix
- #### [searchPhrasePrefix($phrase, $options)](https://elasticsearch.pdphilip.com/eloquent/search-queries#search-phrase-prefix)
```php
Person::searchPhrasePrefix('loves espressos and te')->get();
```
```json
{
"index": "people",
"body": {
"query": {
"multi_match": {
"query": "loves espressos and te",
"type": "phrase_prefix"
}
},
"size": 1000
}
}
```
### Cross Fields
- #### [searchTermCross($term, $fields, $options)](https://elasticsearch.pdphilip.com/eloquent/search-queries#search-term-cross)
```php
Person::searchTermCross('Will Smith', [ 'first_name','last_name'],['operator' => 'and'])->get();
```
```json
{
"index": "people",
"body": {
"query": {
"multi_match": {
"query": "Will Smith",
"type": "cross_fields",
"fields": [
"first_name",
"last_name"
],
"operator": "and"
}
},
"_source": [
"*"
]
}
}
```
### Bool Prefix
- #### [searchBoolPrefix($phrase, $options)](https://elasticsearch.pdphilip.com/eloquent/search-queries#search-bool-prefix)
```php
Person::searchBoolPrefix('loves espressos and te')->get();
```
> Search for people who have the phrase 'loves espressos and' with a prefix of 'ru' in the next word in any field. Ex:
- 'loves espressos and tea'
- 'loves espressos and tennis'
- 'loves espressos and tequila'
```json
{
"index": "products",
"body": {
"query": {
"multi_match": {
"query": "loves espressos and te",
"type": "bool_prefix"
}
},
"size": 1000
}
}
```
## Geo Queries
### Geo-bounding box query
- #### [whereGeoBox($field, $topLeft, $bottomRight, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-geo-box)
```php
// Define the top-left and bottom-right coordinates of the box
$topLeft = [-10, 10]; // [latitude, longitude]
$bottomRight = [10, -10]; // [latitude, longitude]
// Retrieve UserLogs where 'agent.geo' falls within the defined box
UserLog::where('status', 7)->whereGeoBox('agent.geo', $topLeft, $bottomRight)->get();
```
```json
{
"index": "user_logs",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 7
}
}
},
{
"geo_bounding_box": {
"agent.geo": {
"top_left": [
-180,
90
],
"bottom_right": [
180,
-90
]
}
}
}
]
}
},
"size": 1000
}
}
```
### Geo-distance query
- #### [whereGeoDistance($field, $distance, $point, $options)](https://elasticsearch.pdphilip.com/eloquent/es-queries#where-geo-distance)
```php
// Specify the central point and radius
$point = [
'lat' => 51.509865,
'lon' => -0.118092,
]; //or:
$point = [51.509865, -0.118092]; // [latitude, longitude]
$distance = '20km';
// Retrieve UserLogs with Status 7 where 'agent.geo' is within 20km of center of London
UserLog::where('status', 7)->whereGeoDistance('agent.geo', $distance, $point)->get();
```
```json
{
"index": "user_logs",
"body": {
"query": {
"bool": {
"must": [
{
"term": {
"status": {
"value": 7
}
}
},
{
"geo_distance": {
"agent.geo": [
51.509865,
-0.118092
],
"distance": "20km"
}
}
]
}
},
"size": 1000
}
}
```
## Query String Query
### `query_string` query
- #### [searchQueryString($query, $fields, $options)](https://elasticsearch.pdphilip.com/eloquent/query-string-queries#search-query-string)
```php
Book::searchQueryString('Eric OR (Lean AND Startup)')->get();
```
```json
{
"index": "books",
"body": {
"query": {
"query_string": {
"query": "Eric OR (Lean AND Startup)"
}
},
"size": 1000
}
}
```
## Bucket Aggregations
### Term aggregation
- ##### [distinct(string|array $field, $includeDocCount = false](https://elasticsearch.pdphilip.com/eloquent/distinct/#distinct)
- ##### [bulkDistinct(string|array $field, $includeDocCount = false)](https://elasticsearch.pdphilip.com/eloquent/distinct/#bulk-distinct)
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->distinct('user_id');
```
> Retrieves All the unique user_ids of users logged in the last 30 days
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-21T18:41:18+00:00"
}
}
},
"size": 0,
"aggs": {
"by_user_id": {
"terms": {
"field": "user_id",
"size": 1000
}
}
}
},
"_source": [
"user_id"
]
}
```
### Composite aggregation
- #### [groupBy(string|array $field)](https://elasticsearch.pdphilip.com/eloquent/distinct/#groupby)
```php
UserLog::where('created_at', '>=', Carbon::now()->subDays(30))
->groupBy(['user_id'])
->get();
```
> Retrieves All the unique user_ids of users logged in the last 30 days
```json
{
"index": "user_logs",
"body": {
"query": {
"range": {
"created_at": {
"gte": "2025-02-22T09:20:46+00:00"
}
}
},
"size": 0,
"aggs": {
"group_by": {
"composite": {
"sources": [
{
"user_id": {
"terms": {
"field": "user_id.keyword"
}
}
}
]
}
}
}
},
"_source": [
"*"
]
}
```
### Range aggregation
- #### [groupByRanges(string $field, array $ranges)](https://elasticsearch.pdphilip.com/eloquent/distinct/#groupby-ranges)
```php
$ranges = [
[
'key' => 'low-stock',
'to' => 5,
],
[
'key' => 'medium-stock',
'from' => 5,
'to' => 15,
],
[
'key' => 'high-stock',
'from' => 15,
]
];
$groups = Item::groupByRanges('stock', $ranges)
->get();
```
```json
{
"index": "items",
"body": {
"size": 0,
"aggs": {
"stock_range": {
"range": {
"field": "stock",
"ranges": [
{
"key": "low-stock",
"to": 5
},
{
"key": "medium-stock",
"from": 5,
"to": 15
},
{
"key": "high-stock",
"from": 15
}
]
}
}
}
},
"_source": [
"*"
]
}
```
### Date Range aggregation
- #### [groupByDateRanges(string $field, array $ranges, $options = [])](https://elasticsearch.pdphilip.com/eloquent/distinct/#groupby-date-ranges)
```php
$ranges = [
['to' => 'now-10M/M'],
['from' => 'now-10M/M'],
];
$options = ['format' => 'MM-yyyy'];
UserSession::groupByDateRanges('created_at', $ranges, $options)->get();
```
```json
{
"index": "user_sessions",
"body": {
"size": 0,
"aggs": {
"created_at_range": {
"date_range": {
"field": "created_at",
"format": "MM-yyyy",
"ranges": [
{
"to": "now-10M/M"
},
{
"from": "now-10M/M"
}
]
}
}
}
}
}
```
---
## Handling Errors
This package returns errors that are more readable than Elasticsearch's default responses. And embeds helpful metadata to help elucidate the issue.
***
## QueryException
The `QueryException` class will be thrown when a query fails to be executed by the Elasticsearch client. This will account for most errors.
The message will be extracted from the verbose Elasticsearch response and the details will be stored in the Exception's `details` property.
### Example
Where the `manufacturer.location` field has not been mapped as a `geo` field:
```php
use PDPhilip\Elasticsearch\DSL\exceptions\QueryException;
try {
return Product::where('status', 2)->filterGeoPoint('manufacturer.location', '100km', [0, 0])->get();
} catch (QueryException $e) {
return $e->getDetails();
}
```
Error Message will be: `400 Bad Request: all shards failed - failed to find geo field [manufacturer.location]`
And `$e->getDetails()` returns:
```json
{
"error": "400 Bad Request: all shards failed - failed to find geo field [manufacturer.location]",
"details": {
"error": {
"root_cause": [
{
"type": "query_shard_exception",
"reason": "failed to find geo field [manufacturer.location]",
"index_uuid": "kJgCkVJ3RfSuF8G5OaROEg",
"index": "es11_products"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "es11_products",
"node": "FOKbXkxPT56cxxfvtbgV1g",
"reason": {
"type": "query_shard_exception",
"reason": "failed to find geo field [manufacturer.location]",
"index_uuid": "kJgCkVJ3RfSuF8G5OaROEg",
"index": "es11_products"
}
}
]
},
"status": 400
},
"code": 400,
"exception": "Elastic\\Elasticsearch\\Exception\\ClientResponseException",
"query": "returnSearch",
"params": {
"index": "es11_products",
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"status": 2
}
}
],
"filter": {
"geo_distance": {
"distance": "100km",
"manufacturer.location": {
"lat": 0,
"lon": 0
}
}
}
}
}
},
"size": 1000
},
"original": "400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"query_shard_exception\",\"reason\":\"failed to find geo field [manufacturer.location]\",\"index_uuid\":\"kJgCkVJ3RfSuF8G5OaROEg\",\"index\":\"es11_products\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"es11_products\",\"node\":\"FOKbXkxPT56cxxfvtbgV1g\",\"reason\":{\"type\":\"query_shard_exception\",\"reason\":\"failed to find geo field [manufacturer.location]\",\"index_uuid\":\"kJgCkVJ3RfSuF8G5OaROEg\",\"index\":\"es11_products\"}}]},\"status\":400}"
}
```
***
---
## Elasticsearch Quirks
Understanding Elasticsearch's unique behaviors can significantly improve your interaction with it, especially when using it through this Laravel plugin. Here are some common scenarios you might encounter
***
## Error: all shards failed
This error often points to an **index mapping issue**, such as:
* Attempting to sort by a field that is indexed as text without a keyword sub-field.
* Using a filter operation on a field that is not explicitly mapped to that filterable type.
**Solutions:**
* Ensure you are using the correct field type for your operations. For sorting, use keyword types or text fields with a keyword sub-field.
* Review your index mappings to ensure they align with your query requirements.
***
## Default Search Limit
Elasticsearch defaults to returning 10 results for search queries. This plugin extends that default to 1000 for more expansive data retrieval, but you can further adjust this with `$defaultLimit`
```php
class Product extends Model
{
+ protected $defaultLimit = 10000;
protected $connection = 'elasticsearch';
}
```
For processing large datasets, consider implementing [chunking](https://elasticsearch.pdphilip.com/eloquent/chunking/) to iterate over all records efficiently.
***
## Handling Empty Text Fields
By default, empty text fields are not indexed and thus not searchable. To facilitate searches for empty values, you have two main strategies:
See [Querying models with empty strings](https://elasticsearch.pdphilip.com/eloquent/querying-models#empty-strings-values)
* Omit the empty field during document saving/updating, then utilize [whereNull](https://elasticsearch.pdphilip.com/eloquent/querying-models#where-null) for searching.
* Define a default null value in your index schema, ex:
```php
Schema::create('products', function (Blueprint $index) {
$index->keyword('color')->nullValue('NA');
});
```
> The `nullValue` method sets a default value for the field when it is empty as a searchable string, 'NA' in this case.
***
## Sorting by Text Fields
Elasticsearch cannot sort by fields indexed as text due to their tokenized nature. If you try it will throw an 'All shards failed' error. To enable sorting:
* Use keyword type for fields where sorting is a priority and full-text search is not needed.
* For fields requiring both full-text search and sorting, define them with multi-fields in your schema, ensuring text and keyword types are included. Use the keyword type for sorting (`orderBy('field.keyword')`).
```php
Schema::create('contacts', function (Blueprint $index) {
$index->text('name');
$index->keyword('name');
});
```
***
## Save Operations and Refresh
In Elasticsearch, a refresh operation makes newly indexed documents searchable and is synchronous by default in this plugin, ensuring immediate data availability post-write. This could introduce latency in response times, not ideal in all situations.
To bypass this, use `withoutRefresh()->save()` when immediate searchability of the newly indexed document is not critical, reducing write latency.
***
---
---
# ElasticLens
> ElasticLens (pdphilip/elasticlens) creates and syncs searchable Elasticsearch indexes from your SQL models, providing full-text search using the Laravel-Elasticsearch package.
- Package: pdphilip/elasticlens v3 — requires pdphilip/elasticsearch ^5
- GitHub: https://github.com/pdphilip/elasticlens
---
## Getting Started
ElasticLens for Laravel is a package that uses Laravel-Elasticsearch to create and sync a searchable index of your SQL models.
ElasticLens, like scout, is a package that allows you to do full text search on your SQL models.
ElasticLens integrates directly with the Laravel-Elasticsearch package, creating a dedicated **Index Model** that is fully accessible and automatically synced with your SQL **Base Model**
> **Note:**
> **ElasticLens is a separate package from Laravel-Elasticsearch**. Laravel-Elasticsearch is Eloquent for Elasticsearch. ElasticLens is a package that uses Laravel-Elasticsearch to create and sync a searchable index of your SQL models.
## Installation
- Laravel 10/11/12
- Elasticsearch 8.x
**NB: Before you start, set the Laravel-Elasticsearch DB Config (click to expand)**
> See [Laravel-Elasticsearch](https://elasticsearch.pdphilip.com/getting-started/) for more details
>
> Update `.env`
```ini
ES_AUTH_TYPE=http
ES_HOSTS="http://localhost:9200"
ES_USERNAME=
ES_PASSWORD=
ES_CLOUD_ID=
ES_API_ID=
ES_API_KEY=
ES_SSL_CA=
ES_INDEX_PREFIX=my_app_
# prefix will be added to all indexes created by the package with an underscore
# ex: my_app_user_logs for UserLog model
ES_SSL_CERT=
ES_SSL_CERT_PASSWORD=
ES_SSL_KEY=
ES_SSL_KEY_PASSWORD=
# Options
ES_OPT_ID_SORTABLE=false
ES_OPT_VERIFY_SSL=true
ES_OPT_RETRIES=
ES_OPT_META_HEADERS=true
ES_ERROR_INDEX=
ES_OPT_BYPASS_MAP_VALIDATION=false
ES_OPT_DEFAULT_LIMIT=1000
```
> Update `config/database.php`
```php
'elasticsearch' => [
'driver' => 'elasticsearch',
'auth_type' => env('ES_AUTH_TYPE', 'http'), //http or cloud
'hosts' => explode(',', env('ES_HOSTS', 'http://localhost:9200')),
'username' => env('ES_USERNAME', ''),
'password' => env('ES_PASSWORD', ''),
'cloud_id' => env('ES_CLOUD_ID', ''),
'api_id' => env('ES_API_ID', ''),
'api_key' => env('ES_API_KEY', ''),
'ssl_cert' => env('ES_SSL_CA', ''),
'ssl' => [
'cert' => env('ES_SSL_CERT', ''),
'cert_password' => env('ES_SSL_CERT_PASSWORD', ''),
'key' => env('ES_SSL_KEY', ''),
'key_password' => env('ES_SSL_KEY_PASSWORD', ''),
],
'index_prefix' => env('ES_INDEX_PREFIX', false),
'options' => [
'bypass_map_validation' => env('ES_OPT_BYPASS_MAP_VALIDATION', false),
'logging' => env('ES_OPT_LOGGING', false),
'ssl_verification' => env('ES_OPT_VERIFY_SSL', true),
'retires' => env('ES_OPT_RETRIES', null),
'meta_header' => env('ES_OPT_META_HEADERS', true),
'default_limit' => env('ES_OPT_DEFAULT_LIMIT', 1000),
'allow_id_sort' => env('ES_OPT_ID_SORTABLE', false),
],
],
```
```bash
composer require pdphilip/elasticlens
```
Publish the config file and run the migrations with:
```bash
php artisan lens:install
```
Run the migrations to create the index build and migration logs indexes:
```bash
php artisan migrate
```
## Optional Configuration
`lens:install` will publish the config file to `config/elasticlens.php` and create the migration files build and migration logs.
You can customize the configuration in `config/elasticlens.php`
```php
// config/elasticlens.php
return [
'database' => 'elasticsearch',
'queue' => null, // Set queue to use for dispatching index builds, ex: default, high, low, etc.
// Watchers map changes to a given model to tigger an index build
// By default the base model is observed and will trigger an index build
// In some cases you may want to observe a different model
// For example when a relation is updated
'watchers' => [
// \App\Models\Profile::class => [
// \App\Models\Indexes\IndexedUser::class,
// ],
],
'index_build_state' => [
'enabled' => true, // Recommended to keep this enabled
'log_trim' => 2, // If null, the logs field will be empty
],
'index_migration_logs' => [
'enabled' => true, // Recommended to keep this enabled
],
'namespaces' => [
'App\Models' => 'App\Models\Indexes',
],
'index_paths' => [
'app/Models/Indexes/' => 'App\Models\Indexes',
],
];
```
## How it works
The **Index Model** acts as a separate Elasticsearch model managed by ElasticLens, yet you retain full control over it, just like any other Laravel model. In addition to working directly with the **Index Model**, ElasticLens offers tools for
- Full-text searching of your **Base Models** using the full feature set of the Larevel-Elasticsearch package
- Mapping fields (with embedded relationships) during the build process
- Define **Index Model** migrations.
- CLI tools to view sync status and manage your **Index Models**
For Example, a base **User** Model will sync with an Elasticsearch **IndexedUser** Model that provides all the features from **Laravel-Elasticsearch** to search your base **User** Models
---
## Indexing your Base-Model
Instant config. Add the `Indexable` trait to your **Base Model** then set your **Index Model** and you're good to go.
---
## Setup
### 1. Add the `Indexable` Trait to Your Base-Model
```php
//App\Models\Profile.php
+ use PDPhilip\ElasticLens\Indexable;
class Profile extends Model
{
+ use Indexable;
```
### 2 (a) Create an Index-Model for your Base-Model
- ElasticLens expects the **Index Model** to be named as `Indexed` + `BaseModelName` and located in the `App\Models\Indexes` directory.
```php
//App\Models\Indexes\IndexedProfile.php
namespace App\Models\Indexes;
use PDPhilip\ElasticLens\IndexModel;
class IndexedProfile extends IndexModel{}
```
### 2 (b) Or create with artisan
```bash
php artisan lens:make Profile
```
Generates a new index for the `Profile` model.
- That's it! Your `Profile` model will now automatically sync with the `IndexedProfile` model whenever changes occur. You can search your User model effortlessly, like:
```php
Profile::viaIndex()->searchTerm('running')->orSearchTerm('swimming')->get();
```
## Build Indexes for existing data
```bash
php artisan lens:build Profile
```
Build/Rebuilds all the `IndexedProfile` records for the `Profile` model.
---
## Full-text base-model search
Search through your SQL models with the power of Elasticsearch.
## Quick Search
Perform quick and easy full-text searches with `search()`:
```php
User::search('loves espressos');
```
> Search for the phrase `loves espressos` across all fields and return the base `User` models
## Advanced Search
Search with full control using the `viaIndex()` method:
```php
BaseModel::viaIndex()->{build your ES Eloquent query}->first();
BaseModel::viaIndex()->{build your ES Eloquent query}->get();
BaseModel::viaIndex()->{build your ES Eloquent query}->paginate();
BaseModel::viaIndex()->{build your ES Eloquent query}->avg('orders');
BaseModel::viaIndex()->{build your ES Eloquent query}->distinct();
BaseModel::viaIndex()->{build your ES Eloquent query}->{etc}
```
## Index-Model vs Base-Model Results
- Since the `viaIndex()` taps into the **Index Model**, the results will be instances of **Index Model**, not the **Base Model**.
- This can be useful for display purposes, such as highlighting embedded fields.
- However, in most cases you'll need to return and work with the **Base Model**
### To search and return results as `Base-Models`:
##### 1. Use `asBase()`
- Simply chain `->asBase()` at the end of your query:
```php
User::viaIndex()->searchTerm('david')->limit(3)->get()->asBase();
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')->get()->asBase();
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')->first()->asBase();
```
##### 2. Use `getBase()` instead of `get()->asBase()`
```php
User::viaIndex()->searchTerm('david')->orderByDesc('created_at')->getBase();
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')->getBase();
```
### To search and paginate results as `Base-Models`
##### Use: `paginateBase()`
- Complete the query string with `->paginateBase()`
```php
// Returns a pagination instance of Users ✔️:
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')
->paginateBase(10);
// Returns a pagination instance of IndexedUsers:
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')
->paginate(10);
// Will not paginate ❌ (but will at least return a collection of 10 Users):
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')
->paginate(10)->asBase();
```
---
## Examples:
### 1. Basic Term Search
**Return Base-Models:**
```php
User::viaIndex()->searchTerm('nara')
->where('state','active')
->limit(3)
->getBase();
```
**Return Index-Models:**
```php
User::viaIndex()->searchTerm('nara')
->where('state','active')
->limit(3)
->get();
```
> This searches all users who are `active` for the term 'nara' across all fields and return the top 3 results.
### 2. Phrase Search
**Return Base-Models:**
```php
User::viaIndex()->searchPhrase('Ice bathing')
->orderByDesc('created_at')
->limit(5)
->getBase();
```
**Return Index-Models:**
```php
User::viaIndex()->searchPhrase('Ice bathing')
->orderByDesc('created_at')
->limit(5)
->get();
```
> Searches all fields for the phrase 'Ice bathing' and returns the 3 newest results. Phrases match exact words in order.
### 3. Boosting Terms fields
**Return Base-Models:**
```php
User::viaIndex()->searchTerm('David',['first_name^3', 'last_name^2', 'bio'])
->first()->asBase();
```
**Return Index-Models:**
```php
User::viaIndex()->searchTerm('David',['first_name^3', 'last_name^2', 'bio'])
->first();
```
> Searches for the term 'David', boosts the first_name field by 3, last_name by 2, and also checks the bio field. Returns first result with the highest score.
### 4. Geolocation Filtering
**Return Base-Models:**
```php
User::viaIndex()->where('status', 'active')
->filterGeoPoint('home.location', '5km', [0, 0])
->orderByGeo('home.location',[0, 0])
->getBase();
```
**Return Index-Models:**
```php
User::viaIndex()->where('status', 'active')
->filterGeoPoint('home.location', '5km', [0, 0])
->orderByGeo('home.location',[0, 0])
->get();
```
> Finds all active users within a 5km radius from the coordinates [0, 0], ordering them from closest to farthest. Not kidding.
### 5. Regex Search
**Return Base-Models:**
```php
User::viaIndex()->whereRegex('favourite_color', 'bl(ue)?(ack)?')
->getBase();
```
**Return Index-Models:**
```php
User::viaIndex()->whereRegex('favourite_color', 'bl(ue)?(ack)?')
->get();
```
> Finds all users whose favourite color is blue or black.
### 6. Pagination
**Return Base-Models:**
```php
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')
->paginateBase(10);
```
**Return Index-Models:**
```php
User::viaIndex()->whereRegex('favorite_color', 'bl(ue)?(ack)?')
->paginate(10);
```
> Paginate search results.
### 7. Nested Object Search
**Return Base-Models:**
```php
User::viaIndex()->whereNestedObject('user_logs', function (Builder $query) {
$query->where('user_logs.country', 'Norway')
->where('user_logs.created_at', '>=',Carbon::now()->modify('-1 week'));
})->getBase();
```
**Return Index-Models:**
```php
User::viaIndex()->whereNestedObject('user_logs', function (Builder $query) {
$query->where('user_logs.country', 'Norway')
->where('user_logs.created_at', '>=',Carbon::now()->modify('-1 week'));
})->get();
```
> Searches nested user_logs for users who logged in from Norway within the last week. 🤯
### 8. Fuzzy Search
**Return Base-Models:**
```php
User::viaIndex()->searchFuzzy('quikc')
->orSearchFuzzy('brwn')
->orSearchFuzzy('foks')
->getBase();
```
**Return Index-Models:**
```php
User::viaIndex()->searchFuzzy('quikc')
->orSearchFuzzy('brwn')
->orSearchFuzzy('foks')
->get();
```
> No spell, no problem. Search Fuzzy.
### 9. Highlighting Search Results
```php
User::viaIndex()->searchTerm('espresso')->withHighlights()->get();
```
> Searches for 'espresso' across all fields and highlights where it was found.
> *Note:* Returning the base model will not highlight the results
### 10. Phrase prefix search
```php
User::viaIndex()->searchPhrasePrefix('loves espr')->withHighlights()->get();
```
> Searches for the phrase prefix 'loves espr' across all fields and highlights where it was found.
---
## Index-model field mapping
Define your index-model's field mapping and embedded relationships to fine-tune your indexed data
## Defining a field Map
By default, the **Index Model** will be built with all the fields it finds from the **Base Model** during synchronisation.
However, you can customize this by defining a `fieldMap()` method in your **Index Model**.
```php
+ use PDPhilip\ElasticLens\Builder\IndexBuilder;
+ use PDPhilip\ElasticLens\Builder\IndexField;
class IndexedUser extends IndexModel
{
protected $baseModel = User::class;
+ public function fieldMap(): IndexBuilder
+ {
+ return IndexBuilder::map(User::class, function (IndexField $field) {
+ $field->text('first_name');
+ $field->text('last_name');
+ $field->text('email');
+ $field->bool('is_active'); //See attributes as fields
+ $field->type('state', UserState::class); //Maps enum
+ $field->text('created_at');
+ $field->text('updated_at');
+ });
+ }
```
> **Note:**
> If the `fieldMap()` is defined then it will **only** build the fields defined within.
>
> The `id` can be excluded as the value of `$user->id` will correspond to `$indexedUser->id`.
>
> When mapping enums, ensure that you also cast them in the **Index Model**.
>
> If a value is not found during the build process, it will be stored as `null`.
## Attributes as fields
If your **Base Model** has attributes (calculated values) that you would like to have searchable, you can define them in the `fieldMap()` as if they were a regular field.
> For example, `$field->bool('is_active')` could be derived from a custom attribute
> in the **Base Model**:
```php
//App\Models\User.php
// @property-read bool is_active
public function getIsActiveAttribute(): bool
{
return $this->updated_at >= Carbon::now()->modify('-30 days');
}
```
Be mindful that these values are stored in Elasticsearch at their current state during synchronisation.
---
## Relationships as embedded fields
The stand-out feature of ElasticLens is the ability to embed relationships within your **Index Model**. This allows you to create a more structured and searchable index beyond the flat structure of your **Base Model**.
As an illustration, consider the following relationships around the `User` model which will be the base model for our `IndexedUser` model.
[See diagram in the online documentation]
### EmbedsMany
EX: `User` has many `Profiles`
```php
use PDPhilip\ElasticLens\Builder\IndexBuilder;
use PDPhilip\ElasticLens\Builder\IndexField;
class IndexedUser extends IndexModel
{
protected $baseModel = User::class;
public function fieldMap(): IndexBuilder
{
return IndexBuilder::map(User::class, function (IndexField $field) {
$field->text('first_name');
$field->text('last_name');
$field->text('email');
$field->bool('is_active');
$field->type('type', UserType::class);
$field->type('state', UserState::class);
$field->text('created_at');
$field->text('updated_at');
+ $field->embedsMany('profiles', Profile::class)->embedMap(function (IndexField $field) {
+ $field->text('profile_name');
+ $field->text('about');
+ $field->array('profile_tags');
+ });
});
}
```
### EmbedsOne
EX: `Profile` has one `ProfileStatus`
```php
use PDPhilip\ElasticLens\Builder\IndexBuilder;
use PDPhilip\ElasticLens\Builder\IndexField;
class IndexedUser extends IndexModel
{
protected $baseModel = User::class;
public function fieldMap(): IndexBuilder
{
return IndexBuilder::map(User::class, function (IndexField $field) {
$field->text('first_name');
$field->text('last_name');
$field->text('email');
$field->bool('is_active');
$field->type('type', UserType::class);
$field->type('state', UserState::class);
$field->text('created_at');
$field->text('updated_at');
$field->embedsMany('profiles', Profile::class)->embedMap(function (IndexField $field) {
$field->text('profile_name');
$field->text('about');
$field->array('profile_tags');
+ $field->embedsOne('status', ProfileStatus::class)->embedMap(function (IndexField $field) {
+ $field->text('id');
+ $field->text('status');
+ });
});
});
}
```
### EmbedsBelongTo
EX: `User` belongs to an `Account`
```php
use PDPhilip\ElasticLens\Builder\IndexBuilder;
use PDPhilip\ElasticLens\Builder\IndexField;
class IndexedUser extends IndexModel
{
protected $baseModel = User::class;
public function fieldMap(): IndexBuilder
{
return IndexBuilder::map(User::class, function (IndexField $field) {
$field->text('first_name');
$field->text('last_name');
$field->text('email');
$field->bool('is_active');
$field->type('type', UserType::class);
$field->type('state', UserState::class);
$field->text('created_at');
$field->text('updated_at');
$field->embedsMany('profiles', Profile::class)->embedMap(function (IndexField $field) {
$field->text('profile_name');
$field->text('about');
$field->array('profile_tags');
$field->embedsOne('status', ProfileStatus::class)->embedMap(function (IndexField $field) {
$field->text('id');
$field->text('status');
});
});
+ $field->embedsBelongTo('account', Account::class)->embedMap(function (IndexField $field) {
+ $field->text('name');
+ $field->text('url');
+ });
});
}
```
### Embedding without observing/watching
EX: `User` belongs to a `Country` and you don't need to observe the `Country` model to trigger a rebuild of the `User` model.
```php
use PDPhilip\ElasticLens\Builder\IndexBuilder;
use PDPhilip\ElasticLens\Builder\IndexField;
class IndexedUser extends IndexModel
{
protected $baseModel = User::class;
public function fieldMap(): IndexBuilder
{
return IndexBuilder::map(User::class, function (IndexField $field) {
$field->text('first_name');
$field->text('last_name');
$field->text('email');
$field->bool('is_active');
$field->type('type', UserType::class);
$field->type('state', UserState::class);
$field->text('created_at');
$field->text('updated_at');
$field->embedsMany('profiles', Profile::class)->embedMap(function (IndexField $field) {
$field->text('profile_name');
$field->text('about');
$field->array('profile_tags');
$field->embedsOne('status', ProfileStatus::class)->embedMap(function (IndexField $field) {
$field->text('id');
$field->text('status');
});
});
$field->embedsBelongTo('account', Account::class)->embedMap(function (IndexField $field) {
$field->text('name');
$field->text('url');
});
+ $field->embedsBelongTo('country', Country::class)->embedMap(function (IndexField $field) {
+ $field->text('country_code');
+ $field->text('name');
+ $field->text('currency');
+ })->dontObserve(); // Don't observe changes in the country model
});
}
```
### EmbedsMany with a query
EX: `User` has Many `UserLogs` and you only want to embed the last 10:
```php
use PDPhilip\ElasticLens\Builder\IndexBuilder;
use PDPhilip\ElasticLens\Builder\IndexField;
class IndexedUser extends IndexModel
{
protected $baseModel = User::class;
public function fieldMap(): IndexBuilder
{
return IndexBuilder::map(User::class, function (IndexField $field) {
$field->text('first_name');
$field->text('last_name');
$field->text('email');
$field->bool('is_active');
$field->type('type', UserType::class);
$field->type('state', UserState::class);
$field->text('created_at');
$field->text('updated_at');
$field->embedsMany('profiles', Profile::class)->embedMap(function (IndexField $field) {
$field->text('profile_name');
$field->text('about');
$field->array('profile_tags');
$field->embedsOne('status', ProfileStatus::class)->embedMap(function (IndexField $field) {
$field->text('id');
$field->text('status');
});
});
$field->embedsBelongTo('account', Account::class)->embedMap(function (IndexField $field) {
$field->text('name');
$field->text('url');
});
$field->embedsBelongTo('country', Country::class)->embedMap(function (IndexField $field) {
$field->text('country_code');
$field->text('name');
$field->text('currency');
})->dontObserve(); // Don't observe changes in the country model
+ $field->embedsMany('logs', UserLog::class, null, null, function ($query) {
+ $query->orderBy('created_at', 'desc')->limit(10); // Limit the logs to the 10 most recent
+ })->embedMap(function (IndexField $field) {
+ $field->text('title');
+ $field->text('ip');
+ $field->array('log_data');
+ });
});
}
```
## `IndexField` Methods
### Field types:
- `text($field)`
- `integer($field)`
- `array($field)`
- `bool($field)`
- `type($field, $type)` - Set own type (like Enums)
- `embedsMany($field, $relatedModelClass, $whereRelatedField, $equalsLocalField, $query)`
- `embedsBelongTo($field, $relatedModelClass, $whereRelatedField, $equalsLocalField, $query)`
- `embedsOne($field, $relatedModelClass, $whereRelatedField, $equalsLocalField, $query)`
> **Note:**
> For embeds the `$whereRelatedField`, `$equalsLocalField`, `$query` parameters are optional.
>
> `$whereRelatedField` is the `foreignKey` & `$equalsLocalField` is the `localKey` and they will be inferred from the relationship if not provided.
>
> `$query` is a closure that allows you to customize the query for the related model.
### Embedded field type methods:
- `embedMap(function (IndexField $field) {})` - Define the mapping for the embedded relationship
- `dontObserve()` - Don't observe changes in the `$relatedModelClass`
---
---
## Index-model migrations
Define your index-model's `migrationMap()` to take control of the index structure.
## Define the `migrationMap()`
Elasticsearch automatically indexes new fields it encounters, but it may not always index them in the way you need. To ensure the index is structured correctly, you can define a `migrationMap()` in your Index Model.
Since the **Index Model** utilizes the Laravel-Elasticsearch package, use [Index Blueprint](https://elasticsearch.pdphilip.com/schema/index-blueprint/) to define your index's field map in the `migrationMap()` method.
```php
use PDPhilip\Elasticsearch\Schema\Blueprint;
class IndexedUser extends IndexModel
{
//......
public function migrationMap(): callable
{
return function (Blueprint $index) {
$index->text('name');
$index->keyword('first_name');
$index->text('first_name');
$index->keyword('last_name');
$index->text('last_name');
$index->keyword('email');
$index->text('email');
$index->text('avatar')->indexField(false);
$index->keyword('type');
$index->text('type');
$index->keyword('state');
$index->text('state');
//...etc
};
}
```
## Run the Migration
```bash
php artisan lens:migrate User
```
This command will delete the existing index and records, run the migration, and rebuild all records.
---
---
## Base Model Observers
By default, the **Base Model** will be observed for changes (saves) and deletions. When the **Base Model** is deleted, the corresponding **Index Model** will also be deleted, even in cases of soft deletion.
### Handling Embedded Models
When you define a `fieldMap()` with embedded fields, the related models are also observed. For example:
> A save or delete action on `ProfileStatus` will trigger a chain reaction, fetching the related `Profile` and then `User`, which in turn initiates a rebuild of the index for that user record.
> However, to ensure these observers are loaded, you need to reference the User model explicitly:
```php
//This alone will not trigger a rebuild
$profileStatus->status = 'Unavailable';
$profileStatus->save();
//This will
new User::class
$profileStatus->status = 'Unavailable';
$profileStatus->save();
```
### `HasWatcher` trait
If you want ElasticLens to observe an embedded model independently, you can use the `HasWatcher` trait.
This allows you to define a watcher for a specific related model which will trigger a rebuild of a specific index model.
### 1. Add the `HasWatcher` Trait to `Embedded Model`:
```php
//App\Models\ProfileStatus.php
+use PDPhilip\ElasticLens\HasWatcher;
class ProfileStatus extends Eloquent
{
+ use HasWatcher;
```
### 2. Define the Watcher in the `elasticlens.php` Config File:
```php
// config/elasticlens.php
'watchers' => [
+ \App\Models\ProfileStatus::class => [
+ \App\Models\Indexes\IndexedUser::class,
+ ],
],
```
The watchers definition maps the watched model to trigger which index model to rebuild.
### Disabling Base Model Observation
If you want to disable the automatic observation of the **Base Model**, include the following in your **Index Model**:
```php
class IndexedUser extends IndexModel
{
protected $baseModel = User::class;
+ protected $observeBase = false;
```
---
---
## Artisan CLI Tools
ElasticLens provides a set of Artisan commands to manage and monitor your indexes.
## 1. Status Command
Overall Indexes Status
```bash
php artisan lens:status
```
Displays the overall status of all your indexes and the ElasticLens configuration.
## 2. Health Command
Index Health
```bash
php artisan lens:health User
```
Provides a comprehensive state of a specific index, in this case, for the `User` model.
## 3. Migrate Command
Migrate and Build/Rebuild an Index
```bash
php artisan lens:migrate User
```
Deletes the existing User index, runs the migration, and rebuilds all records.
## 4. Make Command
Create a new **Index Model**
```bash
php artisan lens:make Profile
```
Generates a new index for the `Profile` model.
## 5. Build Command
Bulk (re)build indexes:
```bash
php artisan lens:build Profile
```
Rebuilds all the `IndexedProfile` records for the `Profile` model.
---
---
## Build and Migration States
ElasticLens provides built-in Elasticsearch models to track the state of your index builds and migrations.
The `IndexableBuild` and `IndexableMigrationLog` models are Elasticsearch models that you can access directly using the Laravel-Elasticsearch package.
## Accessing `IndexableBuild` model
> ElasticLens includes a built-in `IndexableBuild` model that allows you to monitor and track the state of your index builds.
> This model records the status of each index build, providing you with insights into the indexing process.
**Fields**
```php
/**
* PDPhilip\ElasticLens\Models\IndexableBuild
******Fields*******
* @property string $model // The base model being indexed.
* @property string $model_id // The ID of the base model.
* @property string $index_model // The corresponding index model.
* @property string $last_source // The last source of the build state.
* @property IndexableStateType $state // The current state of the index build.
* @property array $state_data // Additional data related to the build state.
* @property array $logs // Logs of the indexing process.
* @property Carbon $created_at // Timestamp of when the build state was created.
* @property Carbon $updated_at // Timestamp of the last update to the build state.
******Attributes*******
* @property-read string $state_name // The name of the current state.
* @property-read string $state_color // The color associated with the current state.
**/
```
Built-in methods include:
```php
use PDPhilip\ElasticLens\Models\IndexableBuild;
IndexableBuild::returnState($model, $modelId, $indexModel);
IndexableBuild::countModelErrors($indexModel);
IndexableBuild::countModelRecords($indexModel);
```
> **Note:**
> While you can query the `IndexableBuild` model directly, avoid writing or deleting records within it manually, as this can interfere with the health checks and overall integrity of the indexing process.
>
> >The model should be used for reading purposes only to ensure accurate monitoring and reporting.
---
## Access `IndexableMigrationLog` model
> ElasticLens includes a built-in `IndexableMigrationLog` model for monitoring and tracking the state of index migrations.
> This model logs each migration related to an `Index Model`.
**Fields**
```php
/**
* PDPhilip\ElasticLens\Models\IndexableBuild
******Fields*******
* @property string $index_model // The migrated Index Model
* @property IndexableMigrationLogState $state // State of the migration
* @property array $map // Migration map that was passed to Elasticsearch.
* @property int $version_major // Major version of the indexing process.
* @property int $version_minor // Minor version of the indexing process.
* @property Carbon $created_at // Timestamp of when the migration was created.
******Attributes*******
* @property-read string $version // Parsed version ex v2.03
* @property-read string $state_name // Current state name.
* @property-read string $state_color // Color representing the current state..
**/
```
Built-in methods include:
```php
use PDPhilip\ElasticLens\Models\IndexableMigrationLog;
IndexableMigrationLog::getLatestVersion($indexModel);
IndexableMigrationLog::getLatestMigration($indexModel);
```
> **Note:**
> While you can query the `IndexableMigrationLog` model directly, avoid writing or deleting records within it manually, as this can interfere with versioning of the migrations.
>
> >The model should be used for reading purposes only to ensure accurate monitoring and reporting.
---