Hugging Face Storage Buckets

Hugging Face Storage Buckets are mutable object storage on the Hugging Face Hub, built on the Xet storage backend. Arraylake reaches them through the Hugging Face S3-compatible API. You can use a bucket as the storage location for your organization.

For general information on configuring storage and managing repositories, see the Manage Storage and Manage Repositories guides. This page covers the specific settings that Hugging Face requires.

Prerequisites

1. Create a bucket

Create a bucket under your username or under one of your organizations. You can use the Hub UI at huggingface.co/new-bucket, the hf CLI, or the Python API. See the Hugging Face bucket documentation for all three. This example uses the bucket name arraylake-demo.

2. Note your namespace

The namespace owns the bucket. It is your Hugging Face username, or the name of the organization you created the bucket under. Arraylake builds the gateway endpoint from it. In this example the namespace is my-org.

3. Generate S3 credentials

The gateway authenticates with AWS-style keys derived from a Hugging Face User Access Token. The token itself is not an S3 credential.

Open your Access Tokens settings and create a token with Write permission. Then open that token's menu and choose Generate S3 credentials. Copy both values. The access key ID starts with HFAK.

Scope the token to the one bucket you registered. The S3 credentials inherit the token's permissions, so a fine-grained token limits what Arraylake can reach. See Hugging Face security and compliance for the access control model.

warning

Hugging Face shows the secret access key once. Save it to a secure location such as a password manager before you close the dialog.

warning

HMAC credentials (access key and secret access key) are long-lived and grant direct access to your bucket. Handle them carefully. Avoid embedding them in notebooks or scripts that may be shared. For guidance on securely managing credentials and understanding the risks, see Manage Storage.

Configure Arraylake

Create a new BucketConfig for the bucket, using either the Arraylake Python client or the web app.

The web app has a Hugging Face platform option. Pick it, and the form fills in the gateway endpoint, the region, and path-style URLs for you. You supply the bucket name, the namespace, and the credentials.

The Python client takes those settings explicitly. Use the following values:

• Platform: S3 compatible, which an s3:// URI plus an endpoint URL selects for you
• Bucket: the bucket you created, without the namespace (e.g. arraylake-demo)
• Credentials: HMAC
• Access Key / Secret Access Key: the keys you generated above
• Endpoint URL: https://s3.hf.co/<namespace>/ (e.g. https://s3.hf.co/my-org/)
• Region: us-east-1
• Force path style: enabled

The gateway serves one region, so us-east-1 is the only valid region. The gateway also rejects virtual-hosted style addresses, so you must enable path style.

warning

Keep the trailing slash on the endpoint URL. The gateway scopes every request to the namespace in the endpoint path. Without the slash, the S3 client joins the namespace and the bucket name into one word. Every request then fails with NoSuchBucket. The web app builds this URL for you, so the slash matters only when you pass endpoint_url yourself.

Web App

Select Hugging Face from the platform dropdown. Enter the bucket name, then the namespace. The form shows the endpoint it builds from that namespace. Paste the access key ID and the secret access key. HMAC is the only credential type the gateway accepts, so the form preselects it.

Python

from arraylake import Client

credentials = ...  # e.g. load from a json file or env variables

client = Client()

client.create_bucket_config(
  org="my-arraylake-org",
  nickname="my-hf-bucket",
  uri="s3://arraylake-demo",
  endpoint_url="https://s3.hf.co/my-org/",
  region_name="us-east-1",
  force_path_style=True,
  auth_config={
      'access_key_id': credentials['access_key_id'],
      'secret_access_key': credentials['secret_access_key'],
      'method': 'hmac',
  }
)

Limitations

A lost response can look like a conflict. The gateway discards user metadata. Icechunk writes a unique id into that metadata on every conditional write. It reads the id back after a suspicious failure. The id separates a lost success response from a real conflict. On Hugging Face that read always fails. A commit whose acknowledgement is lost in transit can therefore report a conflict that never happened. Your data stays consistent. Retry the commit when this happens.

One key pair serves every user. Hugging Face issues long-lived keys and offers no short-lived credentials. Arraylake therefore stores the key pair, and gives it to every principal with repository access. Two consequences follow. Repositories that share a bucket cannot use different credentials. Hugging Face Audit Logs name the bucket, not the person. Issue one token per bucket, owned by a dedicated Hugging Face account rather than a person. You can then rotate it without disrupting anyone.

That's it! You're ready to create a Repository! 🎉