Elastic aggregation query (WIP)
Runs an aggregation of your own design against your own indices and maps the result onto assets. It is the escape hatch of the catalogue: when the data you need is already in Elasticsearch but no typed integration exposes it, this is how you get it into the inventory.
| Type name | elastic_aggregation_query_integration |
| Creates | Nothing: it only enriches existing endpoints |
| Runs on | A sensor with network reach to Elasticsearch |
| Fields written | Whichever you map |
How it works
Unlike every other integration, this one runs per asset rather than over a global list:
The query string supports {{field_name}} placeholders, replaced with that
asset's value for field_name. An asset missing any referenced field is skipped
rather than queried with a blank. Where the asset holds several values for a
field, only the first is used.
The aggregation is built from every Elasticsearch field named in either mapping,
Field mapping aggregation and Field mapping result: nested terms
aggregations, each ordered by descending document count and capped at 1,000
buckets. Leaf buckets are flattened into rows carrying each level's key plus the
document count. The search is limited to documents whose @timestamp falls in the
look-back window.
A row is always written back to the asset it was queried for, so no correlation rules apply. An asset whose aggregation returns no rows is left untouched, and nothing is ever created: the integration only updates assets that already exist, and its silence does not count towards an asset being decommissioned.
Requirements
The integration is outbound-only: it runs an operator-authored aggregation on a schedule, so nothing is installed in the Elastic stack and nothing has to reach into the deployment.
Network access
| From | To | Protocol | Port |
|---|---|---|---|
| The sensor running the integration | Elasticsearch, at the URL configured on the instance | HTTPS over TCP | Whatever the URL names — 9200 on a default self-managed install, 443 on Elastic Cloud |
One host, and it is Elasticsearch itself rather than Kibana: the run issues a
_search against the configured index pattern. This is usually an internal
destination rather than an internet one, so the rule belongs in the internal
firewall.
Access in Elastic
| Need | Why |
|---|---|
| An account that can create Elasticsearch users or roles | The credential is a user with read and view_index_metadata on the index pattern being queried — see Permissions in Elastic |
| A role scoped to exactly that pattern | The query is operator-authored, so a wide role here is a standing capability to read anything in the cluster |
If Elasticsearch presents a certificate the sensor host does not trust, install the CA on the host in preference to enabling Ignore SSL — the credential is sent on every request.
In this deployment
| Need | Why |
|---|---|
| A registered, healthy sensor with reach to Elasticsearch | The integration runs on a sensor, on the platform's schedule — see Sensors |
| Assets already in the inventory | This integration enriches rather than discovers: the sensor reads the existing assets of the configured type and runs the query once per asset |
| The asset fields the query references | Every {{field}} placeholder, and the Look back field if set, must be populated on an asset or that asset is skipped |
Configuration
| Field | Required | Notes |
|---|---|---|
| Name | Yes | Label for this instance |
| URL | Yes | Elasticsearch base URL |
| Username | Yes | Elasticsearch user |
| Password | Yes | Password for that user. Write-only: once saved it is never shown again, only replaced, and is sent only to the sensor |
| Cookies | No | Extra cookies added to requests. Write-only like Password, and can be removed |
| Ignore SSL | No, off by default | Skips certificate validation |
| Sensor | Yes | Sensor that will run the integration |
| Asset Type | No | Which assets the sensor reads and queries for. Only endpoints are written back, see the TODO below |
| Index Pattern | No | Index pattern to query, e.g. logs-*. Needed in practice: the connection test fails without it |
| Query | No | Elasticsearch query string, may contain {{field}} placeholders |
| Look back Time (seconds) | No | Size of the time window |
| Look back field | No | Asset field (a date/time) supplying the window's end; current time if empty |
| Field mapping aggregation | No | Fields to aggregate on, as a JSON array of {"elastic_field": "...", "lumenedge_field": "..."} pairs |
| Field mapping result | No | Bucket values to write, in the same JSON form |
| Processing Mode | No | Process single hit only or Process popular hit for all. Currently not applied: the first row is always the one written |
Although the console marks them optional, Asset Type, Index Pattern, Query, Look back Time (seconds) and both mappings are all read on every run, so fill them in. The console's hints for the two mapping fields describe the look-back field; ignore them.
Field mapping aggregation decides what the aggregation groups by, and its values
make up the record's identity within a run. Field mapping result decides which
values get written to which platform field. Both add their Elasticsearch fields to
the aggregation. For example, with [{"elastic_field": "source.ip", "lumenedge_field": "private_ip"}] as the aggregation mapping and
[{"elastic_field": "user.name", "lumenedge_field": "last_user"}] as the result
mapping, the most frequent user name seen from the asset's address is written to
a custom field last_user. A target field must already exist on the platform; a
key it does not know is not written.
Mapped values fill an empty field but never replace a populated one, regardless of the template's action settings. That makes this integration safe to point at an existing inventory, but it also means correcting an already-wrong value is not something this integration can do.
Permissions in Elastic
A user with read and view_index_metadata on the index pattern being queried.
Grant nothing wider — the query is operator-authored, so a broad role here is a
standing capability to read anything in the cluster.
Only read on the index pattern is checked by the connection test. Confirm whether
view_index_metadata is actually needed.
The identification stage takes the first row returned for an asset, the one with the most documents at each level, and ignores the rest. Two assets whose first rows carry the same Field mapping aggregation values in one run are treated as the same record, and only the first of them is updated. Design the aggregation so its values are unique per asset.
This type is currently wired for endpoints regardless of the Asset Type setting. Document the supported asset types once the other types are handled, and add a worked end-to-end example with a real index, query and both mappings.
See also
- Elastic agent — for Fleet-enrolled hosts, no query required
- Elastic asset owner — a purpose-built version of this pattern for ownership
- Integrations overview — the shared configuration model