Use external Elasticsearch

If you already have an Elasticsearch cluster, you can connect the support stack to it instead of deploying one via ECK. This reduces resource usage and avoids running a second Elasticsearch instance.

warning

Kibana is automatically disabled in external mode because the ECK Kibana CR requires an ECK-managed Elasticsearch instance. Use your existing Kibana or access Elasticsearch directly.

Install with external Elasticsearch

1. Set the following environment variables:

   export EXTERNAL_ELASTICSEARCH=true
   export EXTERNAL_ES_URL="https://my-es.example.com:9200"
   export EXTERNAL_ES_USERNAME="elastic"
   export EXTERNAL_ES_PASSWORD="YOUR_PASSWORD"

   If your external Elasticsearch uses a publicly trusted certificate (e.g., Elastic Cloud), no additional configuration is needed. If it uses a private CA certificate, also set:

   export EXTERNAL_ES_CA_FILE="/path/to/ca.crt"

   The installer creates the CA secret automatically from this file.

2. Run the quickstart:

   nf-quickstart

   The installer automatically:

   • Skips the ECK operator and CRD installation
   • Creates a Kubernetes secret (external-es-creds) with your credentials
   • Generates a support-values.yml configured for external Elasticsearch
   • Sets up ILM policies and index templates on your external cluster

What changes in external mode

• The ECK operator and CRDs are not installed.
• No Elasticsearch or Kibana custom resources are created.
• nf-data-connector, Filebeat, Metricbeat, Grafana, and the ILM jobs all connect to your external Elasticsearch using the credentials in external-es-creds.
• Uninstall skips ECK cleanup automatically.

warning

Your cluster needs a node with the ingest role

nf-data-connector, Filebeat and Metricbeat write to Elasticsearch directly, and writes into the ziti.* data streams resolve an index-side default pipeline. If no node in your external cluster holds the ingest role, those bulk writes are rejected with There are no ingest nodes in this cluster and no telemetry is stored.

Ziti Console Enterprise needs inline scripts and expensive queries enabled

Ziti Console Enterprise 0.1.6 sums usage with a request-time runtime field, which is an inline painless script. If the console reads from your external cluster, the cluster must allow inline painless scripts and must not set search.allow_expensive_queries: false. Both are Elasticsearch defaults; otherwise the console's usage panels fail.

The installer does not point the console at an external cluster: it keeps the chart default metricsUrl, the in-cluster ECK address.

Switch to external Elasticsearch after installation

If you already have a default installation and want to switch to an external Elasticsearch cluster:

1. Update your support-values.yml:

   elasticsearch:
     enabled: false
     external:
       url: "https://my-es.example.com:9200"
       credentialSecret: "my-es-creds"
       usernameKey: "username"    # default
       passwordKey: "password"    # default
     # Set to "" for publicly trusted certificates
     # Set to a secret name if your ES uses a private CA (see below)
     tlsCaSecret: ""

2. Create the credential secret:

   kubectl create secret generic my-es-creds \
     --from-literal=username=elastic \
     --from-literal=password='YOUR_PASSWORD' \
     -n support

3. Upgrade the support Helm chart to apply the changes:

   helm upgrade --install support ./helm-charts/support/ --values support-values.yml -n support

If your external Elasticsearch uses a private CA, create a CA secret and set tlsCaSecret to its name:

kubectl create secret generic my-es-ca \
  --from-file=ca.crt=/path/to/ca.crt \
  -n support

elasticsearch:
  tlsCaSecret: "my-es-ca"

After upgrading, you can remove the ECK-managed Elasticsearch and Kibana resources and uninstall the ECK operator if it is no longer needed.

Index template for keyword fields

The Grafana dashboards and Ziti Console Enterprise rely on .keyword sub-fields for aggregations and filtering (e.g., tags.serviceId.keyword, service_name.keyword). Elasticsearch's built-in default index template normally creates these automatically by mapping every string field as both text and keyword. However, some managed or custom Elasticsearch clusters may not include this default dynamic mapping.

The support chart's ILM/index-template job automatically applies this mapping to all ziti* indices; the Beats agents manage their own filebeat-* and metricbeat-* templates. If you are managing index templates on your external cluster yourself, or if the job does not run against your cluster, ensure that your external Elasticsearch has an index template covering ziti* patterns with the following dynamic mapping:

PUT _index_template/ziti-template
{
  "index_patterns": ["ziti*"],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "default_policy"
        },
        "number_of_shards": 1,
        "number_of_replicas": 1
      }
    },
    "mappings": {
      "properties": {
        "initialState": { "type": "flattened", "ignore_above": 1024 },
        "finalState":   { "type": "flattened", "ignore_above": 1024 },
        "usage": {
          "type": "object",
          "properties": {
            "ingress": { "properties": { "rx": { "type": "long" }, "tx": { "type": "long" } } },
            "egress":  { "properties": { "rx": { "type": "long" }, "tx": { "type": "long" } } },
            "fabric":  { "properties": { "rx": { "type": "long" }, "tx": { "type": "long" } } }
          }
        }
      },
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ]
    }
  },
  "priority": 310,
  "data_stream": {
    "hidden": false
  }
}

The connector writes data streams, so this template declares data_stream. Container logs are the exception: they land in a regular rolling index behind the ziti.logs-v2 write alias, which the chart covers with a second, higher-priority template (ziti-logs-template, patterns ["ziti.logs-*"], priority 320) so it is not captured by the data-stream template above.

This ensures that every string field indexed under ziti* gets both a full-text text mapping and a .keyword sub-field suitable for terms aggregations, sorting, and filtering.

Three fields are mapped explicitly instead:

• initialState and finalState (ziti.entitychange) are flattened, because the shape of an entity's state differs between entity types. Query their sub-fields without .keyword, for example finalState.name.
• usage (ziti.usage) is an object holding ingress, egress and fabric byte counters. Without it, an index whose first usage document carries a scalar usage maps it as a number and rejects every later usage document.

tip

If your dashboards show "No results" for panels that use .keyword fields, a missing dynamic template is the most likely cause. You can verify by checking the mapping of an existing index:

curl -s -u elastic:PASSWORD "https://YOUR_ES:9200/ziti.circuit/_mapping" | jq '.[][].mappings.properties' | head -20

String fields should show "type": "text" with a "keyword" sub-field. If they only show "type": "text" or "type": "keyword" without the other, apply the index template above and reindex or wait for new data.