Architecture
How a BlixtFS deployment is put together, and how a file request travels from a client to the bucket and back.
Overview
A BlixtFS deployment sits between your clients and your object store. Clients connect to the protocol gateway using standard file protocols (NFS, SMB, etc). The gateways call the file servers over a common protocol. The file servers answer metadata requests from an index held in memory and persisted to disk, and manage data flows to and from the bucket.
See Components.
Life of a request
Listing a directory. The file server answers from the metadata index without calling the object store. This is much faster and cheaper in operations than listing objects in the object store.
Reading a file. The file server uses the combined state of all cached reads and writes to return a correct read, as expected or specified by each file protocol. Reads from object stores are efficient: tuned for each cloud provider; Parallel reads may be used for increased throughput.
Writing a file. Written data is persisted to local disk or replicated to multiple disks. BlixtFS will acknowledge the write as soon as the data has been safely persisted. BlixtFS uploads objects using one or more streams depending on configuration and object size.
Changing the bucket directly. When another program uploads or deletes an object, the provider publishes a change notification. The updater receives it and tells the file server that owns the file, which updates the index. Clients see the change on their next lookup.
Deployment shapes
| Single node | Scale-out | |
|---|---|---|
| Processes | One container runs every service | One container or pod per service |
| File servers | One | Multiple |
| Write storage | Local disk | Replicated across multiple servers |
| Failover | Transparent restart (Enterprise) | Hot standby servers (High Performance) |
| Typical platform | Docker | Kubernetes |
In this section
- Components Every BlixtFS service, what it does and the port it listens on.
- Data model How files map to objects, and what the metadata index holds.
- Reads and writes The cache, the write log and background uploads.
- Change notifications How changes made outside BlixtFS reach the filesystem.
- Scale-out Sharding, partitions, replication and standby servers.
- Security model Credentials, authentication between services, and client access control.