Skip to main content

Geospatial Data in Iceberg Tables

Iceberg and Parquet are rapidly becoming the standard substrate for large-scale geospatial analytics: GeoParquet brought geometry to Parquet files, and Iceberg v3 adds native geometry/geography column types on top. Arraylake Iceberg tables work well for this data today using the same conventions GeoParquet uses internally — this page shows the pattern.

The pattern: WKB geometry + bbox covering​

Arraylake tables use Iceberg format v2, which has no native geometry type (that arrives with v3 — see below). The interoperable v2 approach, identical to what a GeoParquet file does inside plain Parquet, is:

  • Store each geometry column as WKB in a binary column. Every spatial library (Shapely, GeoPandas, DuckDB spatial, Sedona, PostGIS) reads and writes WKB.
  • Record the CRS and encoding in table properties, since v2 has no spec'd home for them.
  • Add a bbox covering struct (xmin/ymin/xmax/ymax) alongside each geometry column, following GeoParquet 1.1's covering convention. Iceberg keeps min/max column statistics for nested struct fields, so predicates on the bbox fields prune manifests and data files — spatial pushdown without a spatial index.

The Arraylake web app recognizes these geo.* properties: columns declared in geo.geometry-columns render in the schema browser as geometry columns (with their geometry type, encoding, and CRS) rather than opaque binary fields.

Writing geospatial data​

Using Shapely for the WKB encoding:

import pyarrow as pa
import shapely
from arraylake import Client

client = Client()
catalog = client.get_iceberg("my-org")

bbox = pa.struct(
    [("xmin", pa.float64()), ("ymin", pa.float64()), ("xmax", pa.float64()), ("ymax", pa.float64())]
)
schema = pa.schema(
    [
        ("event_id", pa.int64()),
        ("event_type", pa.string()),
        ("magnitude", pa.float64()),
        ("geometry", pa.binary()),   # WKB Point: event location
        ("bbox", bbox),              # covering for `geometry`
    ]
)

table = catalog.create_table(
    "my-tables.storm_events",
    schema=schema,
    properties={
        "comment": "Severe-weather events with GeoParquet-style WKB geometry",
        "geo.encoding": "WKB",
        "geo.crs": "OGC:CRS84 (lon-lat, WGS 84)",
        "geo.geometry-columns": "geometry (Point)",
        "geo.covering": "bbox covers geometry (GeoParquet 1.1 style)",
    },
)

events = [
    (1, "tornado", 3.0, shapely.Point(-97.5, 35.5)),
    (2, "hail", 2.25, shapely.Point(-98.1, 37.7)),
    (3, "flash flood", 0.8, shapely.Point(-95.4, 29.8)),
]
table.append(
    pa.Table.from_pylist(
        [
            {
                "event_id": eid,
                "event_type": etype,
                "magnitude": mag,
                "geometry": shapely.to_wkb(geom),
                "bbox": dict(zip(("xmin", "ymin", "xmax", "ymax"), geom.bounds)),
            }
            for eid, etype, mag, geom in events
        ],
        schema=schema,
    )
)

For point geometries the bbox is degenerate (xmin == xmax), which is fine — it still drives file pruning. For lines and polygons, geom.bounds gives the true bounding box.

Partition spatially-queried tables by time or region

An identity partition on an event date (or a coarse region key) composes with bbox pruning: the partition eliminates whole directories, then the bbox statistics prune the surviving files.

Spatial filtering with predicate pushdown​

A bounding-box query is an ordinary Iceberg filter on the bbox fields, so it pushes down to file statistics:

table = catalog.load_table("my-tables.storm_events")

# events inside a lon/lat window (here, Tornado Alley)
window = "bbox.xmin >= -103 and bbox.xmax <= -94 and bbox.ymin >= 33 and bbox.ymax <= 39"
df = table.scan(row_filter=window).to_pandas()

# decode WKB for use with GeoPandas
import geopandas as gpd
gdf = gpd.GeoDataFrame(
    df.drop(columns=["geometry"]),
    geometry=shapely.from_wkb(df["geometry"]),
    crs="OGC:CRS84",
)

For exact (non-rectangular) predicates, filter on the bbox first — that's the part Iceberg can prune on — then refine with Shapely or GeoPandas on the much smaller result.

Querying from DuckDB​

DuckDB's spatial extension decodes the WKB columns directly, and the bbox predicate pushes down through the Iceberg scan just as in Python:

INSTALL iceberg; LOAD iceberg;
INSTALL spatial; LOAD spatial;

CREATE SECRET al (TYPE ICEBERG, TOKEN 'ema_...', ENDPOINT 'https://api.earthmover.io/iceberg');
ATTACH 'my-org' AS warehouse (TYPE iceberg, SECRET al);

SELECT event_type,
       count(*) AS n,
       ST_AsText(ST_GeomFromWKB(min(geometry))) AS sample_point
FROM warehouse."my-tables".storm_events
WHERE bbox.xmin >= -103 AND bbox.xmax <= -94
  AND bbox.ymin >= 33 AND bbox.ymax <= 39
GROUP BY 1 ORDER BY n DESC;

The same WKB columns work in Spark (via Apache Sedona's ST_GeomFromWKB), Trino, and any engine with WKB functions — see Connecting Query Engines for catalog configuration.

Iceberg v3 and native geometry types​

Iceberg format v3 defines first-class geometry and geography column types, aligned with GeoParquet and with Parquet's new GEOMETRY/GEOGRAPHY logical types. With v3, the CRS lives on the column type and geometry bounds are tracked natively in column statistics, so engines get spatial pushdown without a hand-maintained bbox.

Arraylake currently creates and accepts v2 tables while the engine ecosystem (PyIceberg included) completes v3 write support; v3 will be supported once that lands. The pattern on this page is forward-compatible: WKB is exactly the encoding engines use to transport v3 geometry values, so migrating is an ALTER TABLE-style schema evolution — add a geometry-typed column, rewrite the WKB into it, and drop the bbox struct.