Blixt Documentation v2.11

Naming buckets

BlixtFS serves the buckets you name when it starts. Name them as positional arguments, in the BUCKETS environment variable, or in the configuration file.

Naming a cloud without naming any of its buckets — an account or project plus credentials — serves every bucket those credentials can see.

Naming nothing at all is not an error. BlixtFS starts and serves a single file in the mount root:

/no_buckets_specified.txt

The file explains how to name buckets and links back to this page. It is read-only and disappears as soon as BlixtFS is restarted with at least one bucket, so an empty mount always explains why it is empty.

Bucket specs

On the command line and in environment variables, a bucket is named with a bucket spec: a scheme, the bucket name, and optional attributes after a colon.

scheme://bucket-name[:attribute,attribute=value,...]
docker run ... blixtfs/standard:2.11.0 \
  s3://my-bucket \
  gs://training-data:readonly,lazy \
  r2://media-archive:rescan=6h

Schemes

Scheme Provider Directory under the mount root
s3:// Amazon S3 aws/
gs:// Google Cloud Storage gcp/
azure:// Azure Blob Storage azure/
oci:// Oracle OCI Object Storage oracle/
minio:// MinIO minio/
r2:// Cloudflare R2 cloudflare/
cw:// CoreWeave AI Object Storage coreweave/

A bucket name without a scheme belongs to the cloud named by --default_cloud. The directory root names can be changed.

Attributes

Attribute Meaning Available
readonly Never write to this bucket. See Read-only buckets. 2.8 or earlier
ro Short for readonly. 2.9
anonymous Read without credentials, for public datasets. Implies readonly. 2.9
lazy Serve immediately; index each directory on first access. 2.8 or earlier
pubsub, nopubsub Turn change notifications on or off for this bucket. 2.8 or earlier
topic=NAME Subscribe to an existing notification topic instead of creating one (GCS and S3). 2.9
rescan=DURATION Relist the bucket on a schedule, for example rescan=1h. rescan=never turns it off. 2.9
read_replication=N Number of cache copies of each chunk (scale-out). 2.8 or earlier
write_replication=N Number of write-tier copies of unuploaded data (scale-out). 2.8 or earlier
uid=N, gid=N Default owner and group of files in this bucket. See Default owner and permissions. 2.11
file_mode=OCTAL, dir_mode=OCTAL Default permissions of files and directories, for example file_mode=0640. 2.11

The readonly attribute has existed for several releases, but the guarantee that BlixtFS never writes to the bucket in any way is complete from 2.9. See Read-only buckets.

If a bucket spec contains an attribute BlixtFS doesn’t recognise, that bucket is not served, and the log explains why. The other buckets are served as usual. A typo can never leave a bucket served writable or unindexed: the bucket is left out entirely rather than served with the attribute ignored. (Releases before 2.9 ignored the attribute, and 2.9.0 refused to start.)

BlixtFS serves whole buckets. A spec that names a path inside a bucket, such as r2://media-archive/2024, is skipped the same way, and the log names the bucket to configure instead. This applies in the configuration file and environment variables too.

Buckets in the configuration file

In YAML, each bucket is an entry under its provider, and attributes become keys:

cloud:
  aws:
    enabled: true
    buckets:
      - bucket: s3://my-bucket
        region: us-east-1
      - bucket: s3://shared-data
        region: eu-west-1
        read_only: true
        rescan: 1h
        topic: arn:aws:sns:eu-west-1:123456789012:owner-events
  gcp:
    enabled: true
    project: my-project
    buckets:
      - bucket: gs://training-data
        lazy: true
      - bucket: gs://public-dataset
        anonymous: true
      - bucket: gs://team-share
        uid: 1000
        gid: 1000
        file_mode: 0660
        dir_mode: 0770

S3 bucket entries need the bucket’s actual region. To serve every bucket the credentials can see, set all_buckets: true on the provider instead of listing them.

Default owner and permissions

Each bucket has a default owner, group, file mode and directory mode. A new bucket takes its defaults from the server-wide filesystem settings:

filesystem:
  default_uid: 1000
  default_gid: 1000
  default_file_mode: 0644
  default_directory_mode: 0755

The uid, gid, file_mode and dir_mode attributes override them for one bucket. Unlike the server-wide settings, they also apply to a bucket that already exists: change one, and every file that had the old default changes with it. Files with any other owner or mode are not affected. Removing an attribute leaves the bucket’s current default in place.

The files that change get a new change time (ctime), as they would with chown or chmod. BlixtFS serves the new owner and modes within a second or two of updating them. Mounted clients cache attributes, though, and can show the old values until their cache expires. A Linux NFS client keeps them for up to 60 seconds by default. Mount with a smaller actimeo to see changes sooner.

Modes are always octal, so 644 and 0644 are the same. A bad value means the bucket is not served, as with other attributes.

Environment variables

Buckets can also be listed in BUCKETS, or per provider in AWS_BUCKETS, GCP_BUCKETS, AZURE_BUCKETS, ORACLE_BUCKETS, MINIO_BUCKETS, CLOUDFLARE_BUCKETS and COREWEAVE_BUCKETS. Separate several buckets with spaces or semicolons. Attributes work the same way as on the command line:

BUCKETS="gs://training-data:readonly,lazy gs://team-share:uid=1000,file_mode=0660"

A comma separates a bucket’s attributes, never buckets. A spec with a comma in its bucket name, such as gs://a,gs://b, is not served, and the log explains why.

Provider directories

The “global” mount root has a directory for every cloud provider, as described in “Directory under the mount root” above. By default, only configured providers are listed. Any other provider’s directory is there too, and works by name. Once a client names it – looks it up (stat, cd, a path through it) or makes it with mkdir – the “global” mount root lists it.

ls /mnt/blixt/oracle

It stays listed, on every server and across restarts. A mkdir of a name that isn’t a provider is refused.

A provider you have enabled is always listed. Showing a provider doesn’t enable it: its buckets still need credentials. See Cloud credentials.

SMB clients may not list a provider that another client named until they reconnect: they keep serving the “global” root’s listing they already have.

Creating buckets from the filesystem

A mkdir directly under a cloud’s directory in the “global” root means make this bucket available:

mkdir /mnt/blixt/minio/new-bucket
  • If the bucket already exists — it is loaded: probed and served exactly as it is when listed, read-only included.
  • If the bucket does not exist — it is created, if supported by the object store API. Some cloud providers (AWS, GCP, Cloudflare and CoreWeave) have no creation path at all: their buckets are made in the provider’s console or with its CLI. See LINK for details.

Serving buckets from the filesystem

Accessing a bucket under a cloud’s directory in the global root makes it available, even if the bucket wasn’t explicitly configured.

$ ls /mnt/blixt/aws/
configured-bucket
$ ls /mnt/blixt/aws/other-bucket/
data/  index.json
$ ls /mnt/blixt/aws/
configured-bucket  other-bucket

BlixtFS probes the bucket first: it has to exist and its contents have to be listable. A bucket that fails either is ENOENT, with the reason in the server log.

There is one refusal that is not ENOENT. If the cloud will not say whether the bucket exists – S3 answers 403 for a bucket in an account your credentials cannot see, deliberately, so that a bucket name cannot be tested for – then BlixtFS does not know whether it is there, and says so: the listing fails with EPERM (EACCES over FUSE) rather than claiming the name is free. A mkdir of that name fails the same way instead of trying to create a bucket whose name is taken. Credentials the cloud rejects outright are reported the same way, which is the case worth watching for in the log: it is a configuration problem, and every bucket on that cloud is about to fail with it. It is then served in lazy mode, so the listing that found it is answered immediately while the rest of the namespace fills in behind it.

A bucket BlixtFS cannot write is served read-only, which is how a public dataset is mounted: nothing is written to it, not even the directory markers and metadata stamps a writable bucket gets, and every write from a client is refused with EROFS. The decision is stored with the bucket, so it holds after a restart and on every other server — the same guarantee a bucket configured with :ro gets.

The bucket now has an entry in the index, and every bucket in the metadata index is served. Nothing marks it as special: it is indexed, swept and watched exactly like a bucket named in the configuration, and so is a bucket created with mkdir.

In a scale-out deployments new bucket discovery is managed centrally. New buckets are made available to all servers at once.

Things worth knowing:

  • Disable bucket discovery by setting indexing.bucket_discovery to false.
  • Any client with a mount can discover buckets. Discovery costs cloud API calls and metadata rows, and the write probe puts one small object into a bucket nobody configured — under BlixtFS’s internal prefix, and deleted again.
  • Live change notifications need write access, so a bucket served read-only this way has none: configuring them is itself a write. Its changes are picked up by rescans instead.
  • A public bucket the configured credentials cannot see at all is still ENOENT. Discovery probes with the credentials the server already has; reading a bucket with no credentials at all is the anonymous attribute, and that has to be configured.
  • To stop serving a bucket, remove it. See Removing buckets.

This is the per-name counterpart of all_buckets, which enumerates every bucket the credentials can list at startup.

Removing buckets from the filesystem

Setting the extended attribute bfs.remove_bucket on a bucket’s directory stops serving the bucket. Available from 2.11.

setfattr -n user.bfs.remove_bucket -v 1 /mnt/blixt/aws/other-bucket

The bucket’s entry in the index is deleted and it leaves the provider directory’s listing, on every server. The bucket itself is never touched: its objects stay where they are. Looking the bucket up again serves it again and indexes it from scratch, like any other discovered bucket. The value is ignored.

  • Only a bucket’s own directory. The attribute is refused with EINVAL on anything else, such as a file or directory inside a bucket, a provider directory or the mount root. It is refused on its own too, if the same call sets other attributes.
  • Not while the bucket is in use. While a file in the bucket is open, a write is still uploading, or a consistency check of the bucket is running, the attribute is refused with EINVAL, and the server log says why. Close the files, wait for the uploads to finish, and try again.
  • A read-only bucket can be removed too. Removing writes only to the index, never to the bucket.
  • A configured bucket comes back. A bucket named on the command line, in an environment variable or in the configuration file is served again when the server that reads that configuration restarts, and so is every bucket of a provider with all_buckets. To remove one for good, take it out of the configuration too.
  • Change notifications stop, but what was set up in the provider for them stays, and is used again if the bucket comes back.
  • Any client with a mount can remove a bucket, for every other client. The file protocols do not say who is asking. Turn the attribute off by setting indexing.bucket_removal to false. In a scale-out deployment the config server’s setting decides.

Which clients can set the attribute:

  • Linux clients over NFSv4 and SMB, and FUSE mounts. NFSv3, SFTP and WebDAV have no extended attributes, and macOS has no user. namespace.
  • FUSE takes the name without the user. prefix: setfattr -n bfs.remove_bucket -v 1 /mnt/blixt/aws/other-bucket.

To delete an empty bucket from the provider as well, rmdir its directory instead, where the provider allows it.

Indexing large buckets

The first time BlixtFS serves a bucket, it lists every object into the index before serving it. For buckets with millions of objects that can take a long time. Two ways to start faster:

  • Mark the bucket lazy. It is served immediately. Each directory is listed the first time someone opens it, and a background crawler indexes the rest.
  • Set indexing.lazy: true to make every bucket lazy.

Some cloud providers (GCP) provide API calls to list objects faster with metadata attributes. See LINK for details.

Keeping up with outside changes

By default BlixtFS subscribes to the bucket’s change notifications (see Change notifications). For buckets that can’t deliver them, or where you prefer not to, use nopubsub together with rescan=. The periodic consistency check also catches anything missed.