elasticsearch subscriber

Indexes enriched events into Elasticsearch datastreams.

Add to the subscribers block in config.yaml:

subscribers:
  elasticsearch:
    enabled: true
    addresses:
      - "https://es-host:9200"
    username: ""             # prefer env var ES_USERNAME
    password: ""             # prefer env var ES_PASSWORD
    datastream_prefix: "ziti"
    flush_interval: 5s
    flush_size: 100
    workers: 2               # bulk indexer workers
    buffer_size: 1000
    # Per-event filter applied before each bulk-indexer Add (drops never
    # touch the bulk API). include is any-of, exclude is none-of. See
    # ../ "Per-Subscriber Filtering" for the comparator reference.
    include: []
    exclude: []

Events are indexed into datastreams named <datastream_prefix>.<namespace> (e.g., ziti.usage, ziti.circuit). Uses Elasticsearch's bulk indexer with _op_type: create for datastream compatibility. The @timestamp field is set from the event timestamp. workers maps directly to the bulk indexer's NumWorkers — raise it when bulk index latency is holding up the pipeline.

Available fields and defaults

Field	Default	Description
addresses	—	Elasticsearch node URLs
username	—	Basic auth username (prefer ES_USERNAME env var)
password	—	Basic auth password (prefer ES_PASSWORD env var)
datastream_prefix	ziti	Prefix for datastream names
flush_interval	5s	Max time before flushing a partial batch
flush_size	100	Events per bulk request
workers	2	Bulk indexer worker count
buffer_size	1000	Subscriber channel capacity
include	[]	Per-event predicates against the enriched payload; any-of. Empty = pass everything. See Per-subscriber filtering.
exclude	[]	Per-event predicates; none-of — drop on match.

See Common tuning for workers and buffer_size semantics.