Troubleshooting

This page provides commands and procedures for diagnosing issues with your NetFoundry Self-Hosted environment.

Check deployment status

Run nf-status to get a quick overview of all NetFoundry deployments across the ziti, support, and cert-manager namespaces:

nf-status

All deployments should show the expected replica count in the READY column (e.g., 1/1). If any show 0/1, investigate further using the commands below.

When nf-diagnostics is deployed, nf-status also prints a one-line verdict for the last hour:

HEALTH (nf-diagnostics, 1h): WARNING - <summary>

Run a health report

A running pod is not proof that the network is healthy. nf-diagnostics report reads the telemetry in Elasticsearch and reports resource pressure, controller and router log spikes, offline routers, failing circuits, missing terminators, reconnect loops, link faults, and control-plane latency:

nf-diagnostics report               # last 24 hours
nf-diagnostics report --window 1h

Each finding names the pod, router, service, or identity involved, and most include a hint. The exit code is 0 (healthy), 1 (warning), 2 (critical), or 3 (could not analyze). See nf-diagnostics health report for what each section checks.

Check pod health

List pods in the ziti namespace:

kubectl get pods -n ziti

List pods in the support namespace:

kubectl get pods -n support

The Beats agents run in their own namespace:

kubectl get pods -n beats

Healthy pods show Running with all containers ready. Common unhealthy states:

Status	Meaning
Pending	Pod can't be scheduled—usually insufficient CPU, memory, or storage
CrashLoopBackOff	Container is crashing repeatedly—check logs for the root cause
ImagePullBackOff	Can't pull the container image—check registry credentials and network access
Error	Container exited with an error—check logs

To get details on why a pod is unhealthy:

kubectl describe pod <pod-name> -n ziti

Check Kubernetes events

Events reveal scheduling failures, volume issues, and other cluster-level problems. View recent events sorted by time:

kubectl get events -n ziti --sort-by='.metadata.creationTimestamp'
kubectl get events -n support --sort-by='.metadata.creationTimestamp'

To watch events in real time:

kubectl get events --watch -n ziti

Check services and external access

Verify that LoadBalancer services have been assigned external addresses:

kubectl get services -n ziti

If the EXTERNAL-IP column shows <pending>, the cluster's load balancer provisioner may not be configured or may be unable to allocate an address.

View logs

Controller logs

kubectl logs -f deployment/ziti-controller -n ziti

Edge router logs

kubectl logs -f deployment/ziti-router-1 -n ziti

Support stack component logs

kubectl logs -f deployment/grafana -n support
kubectl logs -f deployment/nf-data-connector -n support
kubectl logs -f deployment/nf-diagnostics -n support

Beats agent logs

kubectl logs -f daemonset/filebeat -n beats
kubectl logs -f daemonset/metricbeat -n beats

View previous container logs (after a crash)

If a container has restarted, view the logs from the previous instance:

kubectl logs <pod-name> -n ziti --previous

Restart a component

If a component is in a bad state after ruling out configuration issues, restart its deployment:

kubectl rollout restart deployment ziti-controller -n ziti
kubectl rollout restart deployment ziti-router-1 -n ziti

For support stack components:

kubectl rollout restart deployment grafana -n support
kubectl rollout restart deployment nf-data-connector -n support

No OpenZiti data in Grafana or Kibana

All OpenZiti events and metrics are collected by nf-data-connector. If the Grafana ziti.* dashboards are empty, start there.

1. Confirm the connector is running:

   kubectl get pods -n support -l app=nf-data-connector

   If the command returns no pods, check the replica count:

   kubectl get deployment nf-data-connector -n support

   A READY of 0/0 means the connector was rendered with replicas: 0 because dataConnector.controllerAddress in support-values.yml is still the default localhost:1280. Set it to your controller's client API address and port, then re-run the support helm upgrade.

2. Check that the connector has OpenZiti credentials:

   kubectl get secret data-connector-credentials -n support

   This secret is created by the data-connector-create-ziti-admin hook job during install and upgrade. If it is missing, inspect the job's logs:

   kubectl logs job/data-connector-create-ziti-admin -n support

   The job skips itself (without failing) when the controller is not yet reachable. Re-running the support helm upgrade fires it again.

3. Check the connector's own logs for controller or Elasticsearch connection errors:

   kubectl logs -f deployment/nf-data-connector -n support

   For more detail, set dataConnector.logLevel: debug in support-values.yml, or set dataConnector.stdout.enabled: true to see every event the connector receives, then re-run the support helm upgrade. Set them back when you're done, because both make the log much noisier.

4. If OpenZiti data is present but container logs are missing, the problem is on the Filebeat path instead. Filebeat writes to the ziti.logs-v2 index and needs the Elasticsearch credentials and CA replicated into the beats namespace:

   kubectl get pods -n beats
   kubectl get secret elasticsearch-es-elastic-user elasticsearch-es-http-certs-public -n beats

   On an external-Elasticsearch install, look for external-es-creds (and external-es-ca if you supplied a CA) instead. Pods stuck in ContainerCreating with a secret not found event mean those secrets were not replicated; re-running nf-upgrade --support replicates them.

Verify controller connectivity

Test that the controller API is reachable at its advertise address:

curl --insecure https://<controller-advertise-address>:<port>/version

If this fails, check DNS resolution and firewall rules:

nslookup <controller-advertise-address>

Log into the controller CLI

Use nf-login to authenticate with the controller and run Ziti CLI commands:

nf-login

Once logged in, you can inspect network state directly:

ziti edge list identities
ziti edge list services
ziti edge list edge-routers

Access the controller pod directly

For advanced debugging, you can exec into the controller pod:

kubectl exec -it deployment/ziti-controller -n ziti -- /bin/bash

Collect diagnostics for NetFoundry support

Run the support bundle command on the installer host:

nf-support-bundle

It writes support_bundle_<UTC timestamp>.zip to the current directory. Send that file to NetFoundry support with your ticket; it already contains the installer logs, so you don't need to attach the logs/ directory separately.

nf-support-bundle runs nf-diagnostics bundle, so the two commands take the same flags. It needs kubectl and helm on the PATH. If you have no kubeconfig of your own, it uses the K3s one (/etc/rancher/k3s/k3s.yaml).

On a terminal, the command first lists the Elasticsearch data sets it can include, with document counts, and asks which to add. ziti.logs is preselected; press Enter to accept it, or type numbers separated by commas, all, or none. Pass -y or --es to skip the question.

What the bundle contains

Path in the zip	Contents
report.txt, report.json	The nf-diagnostics health report for the same time window.
pods/<namespace>/	Logs from every container in the ziti, support, beats, cert-manager, and elastic-system namespaces, including the previous run of restarted containers.
ziti-agent/	ziti agent stack dumps from each controller and router pod, plus profiles when requested.
helm/	Values and rendered manifests for every Helm release in the ziti and support namespaces.
k8s/	Nodes, events, kubectl get all, pod descriptions, persistent volume claims, and versions.
host/	Operating system, CPU, memory, and disk details of the host.
installer/	Installer and upgrade logs from logs/, and the output of nf-status.
elasticsearch/<name>.ndjson.gz	The Elasticsearch data sets you selected, in time order. Above the --es-max cap, the newest documents are kept.
MANIFEST.json	The options used, component versions, and the result of each section.

Each section is collected independently. If one fails (for example, the cluster is unreachable, the support stack is not installed, or the nf-diagnostics service is down), the others are still collected and MANIFEST.json records what went wrong. Without the in-cluster service, the report and Elasticsearch exports are read from Elasticsearch directly; without a support stack, only those two sections fail. The command exits 0 whenever the zip is written, and 3 when it cannot write it.

What the bundle never contains

• Kubernetes Secret data. Secrets in the Helm manifests are reduced to their names and key names.
• INSTALL-NOTES.txt, which holds credentials. Only *.log files from logs/ are read from the install directory.
• Passwords, tokens, keys, JWTs, and PEM blocks in logs, Helm values, command output, and Elasticsearch documents. These are masked before anything is written to the zip.

You can review the zip before you send it.

Options

Flag	Description
--window W	Time window for logs, the report, and exports: 1h, 6h, 24h, 3d, or 7d (default 24h).
--es LIST	Elasticsearch data sets to include: a comma-separated list, all, or none (default ziti.logs).
--es-max N	Maximum documents per data set; the newest are kept (default 100000).
--list-es	List the available Elasticsearch data sets and exit.
--components LIST	Collect only these sections: report, pods, agent, helm, k8s, host, installer, es (default all).
--pprof	Also capture a 30-second CPU profile, a heap profile, and memory statistics from each controller and router.
--heap, --cpu, --mem	Capture only the heap profile, CPU profile, or memory statistics. These are the flags nf-support-bundle accepted before 2.0.2.
-o, --output DIR	Directory to write the zip to (default: the current directory).
-y, --yes	Don't ask any questions; use --es or its default.
--ziti-namespace NS	Ziti namespace (default ziti).
--support-namespace NS	Support stack namespace (default support).

Examples:

nf-support-bundle --list-es                                  # see what Elasticsearch data is available
nf-support-bundle -y --window 6h --es ziti.logs,ziti.connect  # no questions, last 6 hours
nf-support-bundle -y --pprof -o /tmp                          # add profiles, write to /tmp
nf-support-bundle -y --es none                                # no Elasticsearch data