Troubleshooting
Common problems and how to resolve them.
Start with the log: docker logs blixt or kubectl -n bfs logs deploy/<component>.
Most startup problems are reported there in plain language.
The container exits immediately
/data is not a mounted volume. BlixtFS refuses to run with its data
inside the container’s own filesystem, where it would be lost with the
container. Add -v blixt-data:/data for a Docker volume, or
-v /some/host/path:/data for a host directory.
The database directory “is still owned by 0:0 after chown”. /data is a
host directory on a filesystem that ignores changes of owner, and the embedded
database has to own its data directory.
Use a Docker volume: -v blixt-data:/data. See
Known issues.
The configuration is invalid. The log names the setting. Check YAML indentation, and that bucket entries sit under the right provider.
The auth token was refused. The log says “refused this process’s auth
token (AUTH_TOKEN)”. Every BlixtFS server in a deployment needs the same
AUTH_TOKEN.
The write tier is off in a scale-out deployment. The log says “a scaleout
deployment requires the write tier”. Remove write.enabled: false or
--write=false.
A bucket doesn’t appear
- Check the log for the bucket’s name. Credential and permission errors are reported per bucket.
- Check that the bucket is under the right provider directory: an S3 bucket
appears in
aws/, a GCS bucket ingcp/. - For S3, check that the bucket’s
regionis correct. - Large buckets may still be indexing. Mark them
lazyto serve them at once.
Mounting fails
Connection refused or timed out. Check that the port is published
(-p 2049:2049) and not blocked by a firewall. For a Compose deployment,
remember the example binds ports to 127.0.0.1.
MacOS: “Operation not permitted”. Add noresvport to the mount options.
SMB on MacOS: port 445 is in use. MacOS File Sharing already uses port 445. Use NFS instead, or turn off File Sharing.
NFSv3 doesn’t work. NFSv3 needs a license and, in Docker, host networking. Use NFSv4 otherwise.
Changes made in the bucket don’t show up
-
Check that notifications are working. The log reports whether notifications were set up for each bucket, and why not if they weren’t. For Google Cloud Storage, the storage service agent needs the
roles/pubsub.publisherrole. On any directory in the bucket,getfattr -n bfs.notificationssays when an event was last applied and how many are waiting:bfs.notifications="last applied 12s ago; 2 waiting, the oldest for 40s"Events waiting a long time mean they are arriving faster than they are applied, or the queue is shared with other traffic. Start BlixtFS with
--log_pubsubto log a line for every event applied, saying how long after the event it took effect, and every 30 seconds while events are waiting. Those lines start withpubsub:. -
Check the provider. CoreWeave has no notifications, and Cloudflare R2 needs an API token for them. For these, changes arrive with the next consistency check or rescan.
-
Check the client’s cache. NFS clients cache attributes and directory lookups. See Connecting clients for mount options that reduce caching.
-
Ask for a check.
setfattr -n bfs.fsck_request -v 1 <directory>brings that directory up to date. See Consistency checks.
File operations fail with I/O errors
The cloud provider is refusing requests. The Cloud dashboard and the log show provider errors: expired credentials, missing permissions or quota limits.
Local disk is full. Once disk use reaches cache.stop_new_files_percent,
no new files are written, and the log says “Refusing new files”. Files already
being written can finish. Once it reaches cache.stop_write_percent, every
write is refused, and the log says “Refusing writes”. Either way, clients see
“No space left on device” until disk space is freed up. Check upload errors in
the log, and give /data more space.
With write servers, a full write server gets no new files. A file whose write server fills up while it is being written moves to a write server with room for all of it, and the write goes on there. Clients see “No space left on device” only when too few write servers have room.
Getting help
Collect the BlixtFS version (from the startup banner), your configuration with secrets removed, and the log around the time of the problem. Then contact us.