Serve buckets from two clouds

Configuration

One BlixtFS deployment can serve any number of providers at once. Give it credentials for each:

cloud:
  aws:
    enabled: true
    credentials: /credentials/aws
    buckets:
      - bucket: s3://acme-raw
        region: us-east-1
  gcp:
    enabled: true
    project: acme-ml
    credentials: /credentials/gcp.json
    buckets:
      - bucket: gs://acme-training

docker run -d --name blixt --privileged -p 2049:2049 \
  -v /srv/blixt:/data \
  -v /srv/blixt-secrets:/credentials:ro \
  blixtfs/standard:2.9.0 --config /data/config.yaml

Result

sudo mount -t nfs -o vers=4 blixt-server:/ /mnt/blixt
ls /mnt/blixt

aws  gcp

Both buckets are now ordinary directories. Copying between clouds is a file copy:

rsync -a --progress /mnt/blixt/aws/acme-raw/2026/ /mnt/blixt/gcp/acme-training/2026/

Data copied between clouds passes through the BlixtFS server, and your cloud providers charge their usual egress fees. For very large one-off transfers, a provider’s own transfer service may be cheaper.

Where to run it

Every read that misses the cache goes to the bucket’s cloud, so run BlixtFS where most of the reads happen. That is usually next to the compute that uses the data.