Iceberg Tables

Arraylake stores two kinds of data side by side:

• Repos hold multidimensional array data — Zarr groups and arrays, versioned with Icechunk.
• Iceberg namespaces and tables hold tabular data — Apache Iceberg tables, served by an Iceberg REST catalog built directly into Arraylake.

Iceberg tables live in namespaces, and every namespace belongs to your organization. Any engine that speaks the Iceberg REST catalog protocol — PyIceberg, DuckDB, Spark, Trino, and more — can create and query them. Arraylake provides the catalog, storage, credentials, and access control; you never run your own catalog service.

How Arraylake maps onto Iceberg

Iceberg concept	Arraylake
Warehouse	your organization, addressed as my-org
Namespace	an Iceberg namespace in that org, bound to one of the org's buckets
Table	a table in a namespace, addressed as <namespace>.<table>

There is exactly one catalog (warehouse) per organization, and it always exists — nothing to create, and no bucket in the identifier. Everything you can reach in the org is in that one catalog.

Namespaces share one set of names with the org's repos: a namespace cannot be named like an existing repo, and vice versa (case-insensitively). Every name in an org points at exactly one thing, whether you meet it in the web app, the Python client, or a SQL tool that lists both kinds side by side.

Namespaces are flat: identifiers carry exactly one namespace level, so table identifiers always have the form <namespace>.<table>. See Namespaces for the full semantics (including what CREATE NAMESPACE and DROP NAMESPACE do).

Avoid dots in namespace names

Namespace names may contain ., but engines split table identifiers on dots — a namespace named raw.obs would be read as a two-level namespace and rejected. Use - or _ instead.

Use Icechunk repos for array data (model output, satellite imagery, sensor grids) and Iceberg tables for tabular data (observations, events, features, business records). They live side by side in the same org, share the same buckets, and use the same tokens.

Early access

Iceberg is gated by an organization feature flag. If creating a namespace returns a 403 saying Iceberg is not enabled, contact support@earthmover.io to enable it for your org.

How it works

• Arraylake implements the Apache Iceberg REST Catalog spec at https://api.earthmover.io/iceberg. The warehouse identifier is your org name: warehouse="my-org".
• Engines authenticate with an ordinary Arraylake bearer token and receive vended storage credentials scoped to the table they touch — users never handle cloud credentials directly. See Connecting Query Engines.
• Each namespace is bound to one of your org's bucket configurations (S3, S3-compatible, and GCS are supported; Azure is not yet supported for Iceberg) and owns a server-generated prefix in it. Table data and metadata live under that prefix, unless the table binds a bucket of its own.
• Engines write data files directly to object storage; the Arraylake server validates and applies each table commit atomically, so concurrent writers are safe. See the catalog reference for details.
• Access is granted at the namespace or the individual table grain, to users, teams, and api-clients. See Access control.

Quickstart

1. Install the client

pip install 'arraylake[iceberg]'

The iceberg extra installs PyIceberg, which the client uses to talk to the catalog.

2. Create a namespace

You'll need an org with at least one bucket configuration. We'll use the org my-org, the bucket configuration nicknamed my-bucket, and create a namespace called my-tables.

Web App

Navigate to your org's Iceberg page (https://app.earthmover.io/my-org), click New Namespace, pick the bucket configuration to store it on, name it my-tables, and hit Create.

Once created, the namespace page shows a table and schema browser, an access-control tab for granting namespace- and table-level access, and a Connect tab with copy-paste connection snippets for each engine.

Python

from arraylake import Client

client = Client()
client.create_iceberg_namespace(
    "my-org",
    "my-tables",
    bucket_config_nickname="my-bucket",  # omit for the org's default bucket
)

Any engine

Engines can create namespaces themselves. A plain CREATE NAMESPACE lands on the org's default bucket; pass the arraylake.bucket property to choose another:

catalog.create_namespace("my-tables")
catalog.create_namespace("my-other-tables", properties={"arraylake.bucket": "my-bucket"})

If the bucket is omitted, the org's default bucket is used. Arraylake picks the storage prefix for you — prefixes are never caller-supplied.

3. Open the catalog

client.get_iceberg returns a ready-configured pyiceberg.catalog.rest.RestCatalog on your org's warehouse:

from arraylake import Client

client = Client()
catalog = client.get_iceberg("my-org")   # a pyiceberg RestCatalog
catalog.properties["warehouse"]          # 'my-org'
catalog.list_namespaces()                # [('my-tables',), ...]

The catalog sees every namespace in the org that you have access to, plus any namespace containing a table you were granted individually.

4. Write your first table

Tables in my-tables are addressed as my-tables.<table>:

import pyarrow as pa

data = pa.table(
    {
        "station": pa.array(["KSEA", "KPDX", "KSFO"], pa.string()),
        "temp_c": pa.array([11.5, 13.1, 14.2], pa.float64()),
    }
)

table = catalog.create_table("my-tables.observations", schema=data.schema)
table.append(data)

Namespaces are flat — catalog.create_table("my-tables.raw.observations", ...) is rejected with a 400. Group tables by creating more namespaces rather than by nesting:

catalog.create_namespace("my-raw-tables")   # on the org's default bucket
catalog.list_namespaces()                    # [('my-tables',), ('my-raw-tables',)]

Each append is one Iceberg commit, recorded in the table's snapshot history.

5. Read it back

table = catalog.load_table("my-tables.observations")
df = table.scan().to_pandas()

# filtered scans push down predicates
hot = table.scan(row_filter="temp_c > 13").to_pandas()

That's it! The same tables can now be queried from DuckDB, Spark, polars, and any other Iceberg REST client.

Where is the data stored?

Every namespace is bound to one bucket configuration and gets its own prefix inside it (the namespace name prefixed with 8 random characters, under the bucket configuration's own prefix). All of its tables' data files, manifests, and metadata files live under that prefix by default, and vended credentials are scoped to the table's location inside it.

A single table can be stored somewhere else by naming a bucket at creation time with the arraylake.bucket table property:

catalog.create_table(
    "my-tables.big_events",
    schema=data.schema,
    properties={"arraylake.bucket": "my-other-bucket"},
)

That table claims its own generated prefix on my-other-bucket; everything else in my-tables stays put. As with namespaces, the prefix itself is always server-generated. The override cannot be combined with a staged create (create_table_transaction).

Namespace and table creation refuses a prefix that overlaps another namespace's, table's, or repo's prefix (equal, parent, or child) on the same bucket, and requires the target prefix to be empty: Arraylake treats everything under it as its own. At creation Arraylake writes a small .arraylake-warehouse.json marker at the prefix; query engines never see it.

No garbage collection yet

Arraylake does not currently reclaim Iceberg storage: files left behind by expired snapshots, rewritten manifests, or dropped tables stay in the bucket. DROP TABLE ... PURGE is accepted but does not delete data. The one exception is deleting a namespace (see below): once its recovery period elapses, everything under its prefix is removed.

Describing namespaces

Like a repo, a namespace carries an Arraylake description and metadata (typed key/value pairs) that the web app shows on the org home and the namespace page. Edit them under the namespace's Settings → General tab.

These are distinct from the Iceberg namespace properties an engine sets with ALTER NAMESPACE ... SET PROPERTIES or pyiceberg's update_namespace_properties. Properties are the engines' configuration surface (location, and whatever your engine writes); Arraylake shows them read-only and never edits them. The one crossover is the comment property engines write for COMMENT ON SCHEMA: the app falls back to it as the description until you set one in Arraylake.

Deleting and restoring namespaces

Deleting a namespace follows the same lifecycle as deleting a repo. By default it is a soft delete: the name and its storage prefix stay claimed for a 7-day recovery period, during which the namespace can be restored along with its grants.

client.delete_iceberg_namespace("my-org", "my-tables", imsure=True)
client.list_iceberg_namespaces("my-org")   # 'my-tables' is gone from the listing
client.restore_iceberg_namespace("my-org", "my-tables")

When the recovery period elapses, Arraylake empties everything under the namespace's storage prefix (the root marker and any files left behind by dropped tables) and removes the namespace. Its name becomes reusable as soon as that cleanup starts, and its prefix once the cleanup finishes.

Two options change this:

• immediate=True skips the recovery period. The namespace can no longer be restored and its name is free to reuse right away; the storage cleanup runs on the next hourly cleanup pass.
• retain_data=True keeps the bytes under the prefix in your bucket. Only the Arraylake namespace is removed, so use this when you intend to keep or hand off the files.

# Gone for good, name reusable now, storage cleaned up within the hour
client.delete_iceberg_namespace("my-org", "scratch", imsure=True, immediate=True)
# Keep the files in the bucket, drop only the catalog entry
client.delete_iceberg_namespace("my-org", "archive", imsure=True, retain_data=True)

A namespace that still holds tables cannot be deleted — drop its tables first. Engines see the same rule as the spec's 409 NamespaceNotEmptyException on DROP NAMESPACE, and DROP NAMESPACE itself is always the default soft delete.

Learn More

📄️Connecting Query Engines

Connect PyIceberg, DuckDB, Spark, polars, and pandas to Arraylake Iceberg tables

📄️Geospatial Data

Store and query GeoParquet-style geospatial data in Arraylake Iceberg tables

📄️REST Catalog Reference

Reference for the Arraylake Iceberg REST catalog API

Manage namespaces with Python

Namespace administration does not require the iceberg extra. These methods are available on both Client and AsyncClient:

from arraylake import Client
from arraylake.types import RepoVisibility

client = Client()
namespace = client.create_iceberg_namespace(
    "my-org", "observations",
    bucket_config_nickname="my-bucket",
    description="Weather station observations",
    metadata={"source": "stations", "quality_checked": True},
    properties={"owner": "science"},
)
namespace = client.get_iceberg_namespace("my-org", "observations")
namespace = client.modify_iceberg_namespace(
    "my-org", "observations",
    description="Quality-controlled weather observations",
    add_metadata={"region": "US"},
    update_metadata={"quality_checked": False},
    visibility=RepoVisibility.PRIVATE,
)

Creation accepts visibility as well. Supported values are PRIVATE and AUTHENTICATED_PUBLIC; public visibility requires a supported bucket. For updates, omitted or None fields remain unchanged. Pass description="" to clear the description and remove_metadata=["key"] to remove metadata keys. Keys in add_metadata, update_metadata, and remove_metadata must be disjoint. Edit the engine-owned Iceberg properties with catalog.update_namespace_properties(...) after creation.

Search and filter namespaces, or iterate large results lazily:

namespaces = client.list_iceberg_namespaces(
    "my-org", search="weather", filter_metadata={"source": "stations"},
)
for namespace in client.list_iceberg_namespaces_paginated(
    "my-org", sort="name", direction="asc", page_size=50,
):
    print(namespace.name, namespace.table_count, namespace.effective_actions)

for table in client.list_iceberg_tables_paginated(
    "my-org", "observations", search="weather", sort="updated",
):
    print(table.name, table.updated)

list_iceberg_tables returns the same table summaries as a list. Table summaries come from catalog rows; listing does not fetch table metadata or record table accesses. Use catalog.load_table(...) for schemas, snapshots, and data access.

Both namespace and table listings support updated, created, and name sorting; namespaces also support tables (table count). With a search query, sorting defaults to relevance and name is unavailable. Without search, sorting defaults to updated and relevance is unavailable. Page sizes range from 1 to 100. Namespace listings accept include_ghosts=True to include deleted namespaces still available for restoration.

# A namespace must be empty before deletion.
client.delete_iceberg_namespace("my-org", "observations", imsure=True)
client.restore_iceberg_namespace("my-org", "observations")

Permission management remains in the web app.

Command-line interface

Use al iceberg namespace for namespace administration and al iceberg table for table discovery:

al iceberg namespace create my-org observations --description "Weather observations"
al iceberg namespace get my-org observations --output json
al iceberg namespace update my-org observations --add-metadata '{"region": "US"}'
al iceberg namespace list my-org --search weather --sort relevance --output json
al iceberg namespace list my-org --filter-metadata '{"region": "US"}' --include-ghosts
al iceberg table list my-org observations --sort name --direction asc
al iceberg table describe my-org observations.weather --output json
# Quote individual components when names contain dots; preserve quotes in the shell.
al iceberg table describe my-org '"observations.v2"."weather.daily"' --output json
al iceberg namespace delete my-org observations --confirm
al iceberg namespace restore my-org observations

table describe takes the org and one NAMESPACE.TABLE identifier. Names retain their case. Double-quote components containing dots or special characters, and escape a double quote within a name by doubling it ("a""b".weather). Wrap quoted identifiers in shell single quotes to preserve the double quotes. Table listings show qualified identifiers; JSON output includes identifier alongside name and namespace. The web app's table copy button uses the same identifier format.

Deletion prompts for confirmation unless --confirm is supplied, and refuses nonempty namespaces. table describe requires arraylake[iceberg]; the other commands use the management API without that extra. List, get, create, update, and describe commands support --output json for scripting. Run any command with --help for its options.

Async use and login refresh

AsyncClient.get_iceberg constructs the catalog in a worker thread. The returned PyIceberg catalog and table objects are synchronous, so offload their operations when using an event loop:

import asyncio
from arraylake import AsyncClient

client = AsyncClient()
catalog = await client.get_iceberg("my-org")
try:
    table = await asyncio.to_thread(catalog.load_table, "observations.weather")
    frame = await asyncio.to_thread(lambda: table.scan().to_pandas())
finally:
    await asyncio.to_thread(catalog.close)

async for namespace in client.list_iceberg_namespaces_paginated("my-org"):
    print(namespace.name)

Catalog requests reload the cached user login and refresh it shortly before the ID token expires. Refresh failures require logging in again. Explicit API-client tokens are used as supplied; their expiration and replacement remain the caller's responsibility. This refresh integration applies to catalogs opened through get_iceberg, not tokens copied into other engines.