nf-diagnostics health report

nf-diagnostics reads the data the support stack already collects in Elasticsearch and turns it into a health report for your deployment: which pods are short of CPU or memory, which routers went offline, which services are failing circuits, and whether the controller logs show an unusual burst of errors. It is new in 2.0.2.

Beta

nf-diagnostics is in beta. Its checks and report format may change in later releases.

It has two parts:

• The in-cluster service: a small deployment named nf-diagnostics in the support namespace. It serves the report as a web page and an HTTP API, refreshing it every few minutes.
• The host command: nf-diagnostics, installed with the netfoundry-self-hosted package. It prints the report in your terminal (nf-diagnostics report) and builds the support bundle (nf-diagnostics bundle, which nf-support-bundle runs). See Collect diagnostics for NetFoundry support.

Enable or disable nf-diagnostics

The service is part of the support stack and is on by default. It pulls a private image (quay.io/netfoundry/nf-diagnostics), so it needs the same NetFoundry container registry credentials as nf-data-connector and Ziti Console Enterprise.

• No pull secret: when the installer or nf-upgrade --support finds no netfoundry-pull-secret, it leaves nf-diagnostics disabled and prints a warning instead of deploying a pod that cannot pull its image. Offline installs are not affected: the image is preloaded and the installer creates a placeholder pull secret.

• Opt out: set NF_DIAGNOSTICS_DISABLED=true in the environment (or your env file) before running nf-quickstart or nf-upgrade, or set it in support-values.yml:

  nfDiagnostics:
    enabled: false

• Turn it on later: make sure netfoundry-pull-secret exists in the support namespace, set nfDiagnostics.enabled: true in support-values.yml (a fresh install that skipped it writes enabled: false there), then run nf-upgrade --support.

The nf-diagnostics host command works without the in-cluster service. See Run the report.

Configuration

Value	Default	Description
nfDiagnostics.enabled	true	Deploy the service.
nfDiagnostics.clusterLabel	""	Name shown at the top of the report. When empty, the controller host from dataConnector.controllerAddress is used.
nfDiagnostics.window	"24h"	Default time window for a report.
nfDiagnostics.serviceType	ClusterIP	Kubernetes service type. Leave this as ClusterIP; see Security.

What it checks

Each section of the report looks at one kind of signal over the chosen time window.

Section	Looks for	Data source
Controller	Whether the controller is running, from its pod metrics and its own events.	metricbeat-*, ziti.metrics*
Resource	Pods whose CPU or memory use is close to their limit (or to the node's capacity when the pod has no limit).	metricbeat-*
Logs	Sudden rises in controller and router error or warning lines, and known problems such as a controller still trying to send events to the RabbitMQ (AMQP) sink that 2.0 removed.	ziti.logs*
Routers	Routers that went offline or keep dropping and re-establishing their connection to the controller.	ziti.router*, ziti.metrics*
Terminators	Services left with no usable terminator, so dials to them do not complete.	ziti.terminator*
Circuits	Dials that did not complete, per service, with the cause. Services deleted since are informational.	ziti.circuit*
Faults	Link faults between routers and failed SDK connections per router.	ziti.metrics*, ziti.link*
Connects	Identities or routers reconnecting to the controller in a loop, and connect storms.	ziti.connect*
Latency	Control-plane latency between controller and routers: a rising average, and repeated slow outliers.	ziti.metrics*
Throughput	Unexpected spikes in network traffic.	ziti.usage*

A data source that does not exist yet (for example, no circuits on a new install) is reported as "no data", not as a problem. A section that could not be read is reported as Incomplete, never as healthy.

Findings include a hint where one applies, such as the steps to remove the retired AMQP settings from the controller.

Severities and exit codes

Most findings are notices: misconfigured services, offline hosting routers, log spikes and long-distance latency are routine, so the report points them out without raising an alarm. Only the controller not running, or no router online at all, is critical.

Shown as	JSON overall	Meaning	nf-diagnostics report exit code
Healthy	OK, INFO	Nothing needs a look.	0
Incomplete	UNKNOWN	A signal, or the whole report, could not be analyzed.	3
Notice	WARNING	Something is worth a look.	1
Critical	CRITICAL	The controller is not running, or no router is online.	2

Exit code 3 is also returned when the command itself fails, for example when it cannot reach the cluster. You can use the exit codes in a monitoring script or cron job.

Run the report

On the installer host:

nf-diagnostics report                            # last 24 hours
nf-diagnostics report --window 1h                # last hour
nf-diagnostics report --format json -o report.json

Flag	Description
--window W	Time window: 1h, 6h, 24h, 3d, or 7d (default 24h).
--format F	pretty, text, or json. Default: pretty on a terminal, text otherwise.
-o, --output F	Write the report to a file instead of the terminal.
--server URL	Use this nf-diagnostics URL instead of finding the service with kubectl.
--namespace NS	Support stack namespace (default support).

The command uses your kubectl configuration to reach the in-cluster service. If the service is not deployed, or is older than 0.2.0, it connects to the support-stack Elasticsearch directly (using the credentials stored in the cluster) and builds the same report locally. Without a support stack there is nothing to analyze, and the command exits with code 3.

nf-status runs a one-hour report and adds a single line to its output when the service is deployed:

HEALTH (nf-diagnostics, 1h): Notice - <summary>

If the line reads unavailable, run nf-diagnostics report to see why.

Web page and API

The service is reachable over OpenZiti at http://nf-diagnostics.ziti by any identity with the #support-users attribute, such as the support-user identity whose support-user.jwt the installer generates. The installer creates the OpenZiti service on a fresh install. On an upgrade from 2.0, nf-upgrade --support creates it when support access over OpenZiti was set up (the ziti-host deployment exists in the support namespace). nf-install-notes lists the address. See Access support tools.

The web page shows the report for a window you choose, lets you download it as text, and lets you download Elasticsearch data (see below).

The same data is available as an HTTP API. The full contract is served at /v1/openapi.yaml.

Endpoint	Returns
/healthz	Service liveness and version.
/v1/report.json	The report as JSON. ?window= accepts 1h, 6h, 24h, 3d, or 7d.
/v1/report.txt	The report as plain text.
/v1/indices	The Elasticsearch data sets available to export (for example ziti.logs, ziti.metrics, metricbeat), with document counts and time ranges.
/v1/export	One data set as compressed newline-delimited JSON: ?index=<name>&window=<window>&max=<documents>. Credentials found in documents are masked.

From a machine enrolled with the support-user identity, you can also run the host command against it:

nf-diagnostics report --server http://nf-diagnostics.ziti

Security

• Read-only access to Elasticsearch. With the bundled Elasticsearch, nf-diagnostics signs in as its own user, nf-diagnostics, which can only read the ziti.*, metricbeat*, filebeat*, zfw.*, and zeek.* data. With an external Elasticsearch, it uses the credentials you configured for the support stack.
• Verified TLS. Connections to Elasticsearch verify its certificate.
• No Kubernetes access. The pod runs as a non-root user with a read-only filesystem and has no Kubernetes API token.

The API has no login

Anyone who can reach the service can read the report and export log data. That is why it is a ClusterIP service reached only over OpenZiti, where the #support-users service policy controls who can dial it. Do not change nfDiagnostics.serviceType to LoadBalancer or expose it through an ingress.

Troubleshooting nf-diagnostics

Check the pod:

kubectl get pods -n support -l app=nf-diagnostics
kubectl logs deployment/nf-diagnostics -n support

Symptom	Cause and fix
No nf-diagnostics deployment	It was left disabled (no pull secret, or NF_DIAGNOSTICS_DISABLED=true). See Enable or disable.
Pod in ImagePullBackOff	The registry credentials in netfoundry-pull-secret are wrong or expired.
Report is UNKNOWN with an Elasticsearch error	Elasticsearch is down or still starting. Check kubectl get elasticsearch -n support.
http://nf-diagnostics.ziti does not resolve	The OpenZiti service is missing. Run nf-upgrade --support to create it, and check your tunneler has the support-user identity.
nf-diagnostics: command not found	The package is older than 2.0.2. Upgrade netfoundry-self-hosted.