Skip to main content
Version: 0.10

Elastic aggregation query (WIP)

Runs an aggregation of your own design against your own indices and maps the result onto assets. It is the escape hatch of the catalogue: when the data you need is already in Elasticsearch but no typed integration exposes it, this is how you get it into the inventory.

Type nameelastic_aggregation_query_integration
CreatesNothing: it only enriches existing endpoints
Runs onA sensor with network reach to Elasticsearch
Fields writtenWhichever you map

How it works​

Unlike every other integration, this one runs per asset rather than over a global list:

The query string supports {{field_name}} placeholders, replaced with that asset's value for field_name. An asset missing any referenced field is skipped rather than queried with a blank. Where the asset holds several values for a field, only the first is used.

The aggregation is built from every Elasticsearch field named in either mapping, Field mapping aggregation and Field mapping result: nested terms aggregations, each ordered by descending document count and capped at 1,000 buckets. Leaf buckets are flattened into rows carrying each level's key plus the document count. The search is limited to documents whose @timestamp falls in the look-back window.

A row is always written back to the asset it was queried for, so no correlation rules apply. An asset whose aggregation returns no rows is left untouched, and nothing is ever created: the integration only updates assets that already exist, and its silence does not count towards an asset being decommissioned.

Requirements​

The integration is outbound-only: it runs an operator-authored aggregation on a schedule, so nothing is installed in the Elastic stack and nothing has to reach into the deployment.

Network access​

FromToProtocolPort
The sensor running the integrationElasticsearch, at the URL configured on the instanceHTTPS over TCPWhatever the URL names — 9200 on a default self-managed install, 443 on Elastic Cloud

One host, and it is Elasticsearch itself rather than Kibana: the run issues a _search against the configured index pattern. This is usually an internal destination rather than an internet one, so the rule belongs in the internal firewall.

Access in Elastic​

NeedWhy
An account that can create Elasticsearch users or rolesThe credential is a user with read and view_index_metadata on the index pattern being queried — see Permissions in Elastic
A role scoped to exactly that patternThe query is operator-authored, so a wide role here is a standing capability to read anything in the cluster

If Elasticsearch presents a certificate the sensor host does not trust, install the CA on the host in preference to enabling Ignore SSL — the credential is sent on every request.

In this deployment​

NeedWhy
A registered, healthy sensor with reach to ElasticsearchThe integration runs on a sensor, on the platform's schedule — see Sensors
Assets already in the inventoryThis integration enriches rather than discovers: the sensor reads the existing assets of the configured type and runs the query once per asset
The asset fields the query referencesEvery {{field}} placeholder, and the Look back field if set, must be populated on an asset or that asset is skipped

Configuration​

FieldRequiredNotes
NameYesLabel for this instance
URLYesElasticsearch base URL
UsernameYesElasticsearch user
PasswordYesPassword for that user. Write-only: once saved it is never shown again, only replaced, and is sent only to the sensor
CookiesNoExtra cookies added to requests. Write-only like Password, and can be removed
Ignore SSLNo, off by defaultSkips certificate validation
SensorYesSensor that will run the integration
Asset TypeNoWhich assets the sensor reads and queries for. Only endpoints are written back, see the TODO below
Index PatternNoIndex pattern to query, e.g. logs-*. Needed in practice: the connection test fails without it
QueryNoElasticsearch query string, may contain {{field}} placeholders
Look back Time (seconds)NoSize of the time window
Look back fieldNoAsset field (a date/time) supplying the window's end; current time if empty
Field mapping aggregationNoFields to aggregate on, as a JSON array of {"elastic_field": "...", "lumenedge_field": "..."} pairs
Field mapping resultNoBucket values to write, in the same JSON form
Processing ModeNoProcess single hit only or Process popular hit for all. Currently not applied: the first row is always the one written

Although the console marks them optional, Asset Type, Index Pattern, Query, Look back Time (seconds) and both mappings are all read on every run, so fill them in. The console's hints for the two mapping fields describe the look-back field; ignore them.

Two mappings, two jobs

Field mapping aggregation decides what the aggregation groups by, and its values make up the record's identity within a run. Field mapping result decides which values get written to which platform field. Both add their Elasticsearch fields to the aggregation. For example, with [{"elastic_field": "source.ip", "lumenedge_field": "private_ip"}] as the aggregation mapping and [{"elastic_field": "user.name", "lumenedge_field": "last_user"}] as the result mapping, the most frequent user name seen from the asset's address is written to a custom field last_user. A target field must already exist on the platform; a key it does not know is not written.

Results are written in do-not-override mode

Mapped values fill an empty field but never replace a populated one, regardless of the template's action settings. That makes this integration safe to point at an existing inventory, but it also means correcting an already-wrong value is not something this integration can do.

Permissions in Elastic​

A user with read and view_index_metadata on the index pattern being queried. Grant nothing wider — the query is operator-authored, so a broad role here is a standing capability to read anything in the cluster.

TODO

Only read on the index pattern is checked by the connection test. Confirm whether view_index_metadata is actually needed.

Only one row per asset is consumed

The identification stage takes the first row returned for an asset, the one with the most documents at each level, and ignores the rest. Two assets whose first rows carry the same Field mapping aggregation values in one run are treated as the same record, and only the first of them is updated. Design the aggregation so its values are unique per asset.

TODO

This type is currently wired for endpoints regardless of the Asset Type setting. Document the supported asset types once the other types are handled, and add a worked end-to-end example with a real index, query and both mappings.

See also​