Blixt Documentation v2.11

Metrics endpoints and the built-in dashboards require a license that includes Monitoring (Enterprise or High Performance).

Built-in Prometheus and Grafana

The blixtfs/enterprise image runs Prometheus and Grafana inside the container, already configured to scrape every BlixtFS component. The monitoring dashboards are available on port 3000:

docker run ... -p 3000:3000 blixtfs/enterprise:latest ...

Open Grafana at http://<host>:3000. It comes with these dashboards:

Dashboard Shows
Overview Health and throughput of the whole deployment
Cloud Requests, bytes, errors and retries per cloud provider
Database Query rates, latency and connection use
NFS NFS operations, latency and errors
SMB SMB operations, latency and errors
Client Operations as seen by the file-protocol clients

In Kubernetes deployments, the monitoring is accessed as a service.

Scraping with your own Prometheus instance

The Blixt Prometheus instance runs on port 9090.

Each component serves Prometheus metrics at /metrics on its own port:

Component Port
Config server 4721
File server 4722
Cache server 4723
Replication 4724
Write server 4725
Updater 4726
Protocol clients 4727
Indexing 4728
Notifier 4732
NFS gateway 9587
SMB gateway 9922
MCP server 4733

In Kubernetes, set serviceMonitor.enabled=true to create a Prometheus Operator ServiceMonitor.

Some BlixtFS metrics are native histograms. Enable scrape_native_histograms: true in your Prometheus scrape configuration to collect them.

Restrict who can reach the metrics endpoints with monitoring.allowed_cidrs.

Indexing progress

Each server that indexes buckets reports how far along it is, per cloud, in the cloud label:

Metric Meaning
bfs_indexing_objects_indexed Objects indexed by the current indexing passes
bfs_indexing_directories_loaded Directories loaded by lazy indexing
bfs_indexing_objects_per_second Objects indexed per second over the last minute
bfs_indexing_directories_per_second Directories loaded per second over the last minute

The totals are gauges, not counters: they start again from zero with each new pass. The rates drop to zero once indexing has finished.

Metrics worth alerting on

  • Cloud request errors and retries, by provider: credential or quota problems
  • Database connection use close to the limit: see Troubleshooting
  • Consistency checks finding issues repeatedly: bfs_fsck_issues_total
  • Local disk use close to cache.stop_new_files_percent (95% by default), at which point BlixtFS stops writing new files, and cache.stop_write_percent (98% by default), at which it stops accepting writes

Logs

Each server’s own messages go to standard output, so docker logs and kubectl logs show them. The daemons a server starts, such as nfs-ganesha, log to files instead, and the server names each file when it starts:

Daemon "nfsd" logs to /data/cache/log/nfsd.log

Those files are in /data/log in a single-node container, and in /data/cache/log in a scale-out deployment. The enterprise image’s embedded PostgreSQL logs to /data/log/postgres.log. Samba’s own log is /var/log/samba/smbd.log, and it limits that file itself. To read one on Kubernetes, run it in the pod:

kubectl exec <pod> -- tail -f /data/cache/log/nfsd.log

The NFS and SMB pods have no volume, so their log files go when the container does.

BlixtFS rotates the log files itself. Docker does not limit what it keeps of a container’s output unless asked to, so see the Docker installation page for that. Kubernetes limits it through the kubelet’s containerLogMaxSize and containerLogMaxFiles.

Setting Default Meaning
--log_json off Write JSON log lines for log collectors
--log_fsck off Log consistency-check details
--log_all off Log everything (very verbose; for troubleshooting only)
cleaner.log_max_size_mb 100 Rotate log files at this size
cleaner.log_max_files 5 Keep this many rotated files

Other logging.* settings turn on detail for one area each, such as cloud, database, nfs or pubsub. Only configuration logging is on by default.