Skip to main content

openEO: Server-Side Processing

openEO is an open API standard for describing geospatial analyses as process graphs and executing them on the backend where the data lives. Where the other Compute protocols serve slices of your data for you to process locally, openEO inverts this: you describe the analysis with the standard openEO Python client, Compute runs it next to your Arraylake data, and you download only the result.

References​

These are links to the openEO standard and websites:

Arraylake implements a subset of the openEO API (v1.2.0): collection discovery, synchronous processing, and batch jobs. Its processes cover sampling points and averaging over polygons from gridded datasets, band math, reducing dimensions, aggregating time into calendar periods, climatologies and anomalies, and building multiscale overviews. What's Not Supported lists the standard openEO processes it doesn't run yet.

Activating openEO for Arraylake Datasets​

openEO can be activated using the Arraylake command line interface (CLI)

al compute enable {org} openeo --sku-code {sku}

The service is enabled on an organization-wide basis; every Repo the caller has access to is served as a collection.

Connecting​

The easiest way to connect is with the arraylake Python client, which resolves the service URL and authenticates the connection for you:

from arraylake import Client

client = Client()
connection = client.get_openeo_connection("{org}")
note

get_openeo_connection requires arraylake>=1.2.0 and the optional openeo dependency, installed with pip install 'arraylake[openeo]'.

Any stock openEO client can also connect directly using the service URL:

import openeo

connection = openeo.connect("https://compute.earthmover.io/v1/services/openeo/{org}")
connection.authenticate_bearer_token("{api_token}")

Unlike the other Compute protocols, openEO clients authenticate with a bearer token — use an Arraylake API token — rather than HTTP Basic auth. Public services require no authentication.

Collections​

Datasets are addressed as openEO collections using an Arraylake path as the collection id:

{org}/{repo}
{org}/{repo}/{ref}
{org}/{repo}/{ref}/{path/to/group}
{org}/table/{namespace}.{table}
{org}/table/{namespace}.{table}/{ref}
{org}/table/{namespace}.{level}.{table}/{ref}

The ref defaults to main and the group defaults to the root group. In a table's id every . separates levels: the first is the namespace, the last is the table, and any between are nested namespaces (nnja-ai.tankA.adpupa.NC002001). A namespace or table whose own name contains a . can't be addressed by an id. A ref can name its kind, as branch.{name}, tag.{name}, or snapshot.{id}; a bare name is tried as a branch, then a tag, then (for a Repo) a snapshot ID. An Iceberg table loads as the points or polygons to aggregate over; describe_collection describes it as a vector cube along a geometry dimension.

connection.list_collections()
connection.describe_collection("{org}/{repo}")

describe_collection returns a STAC collection describing the datacube's dimensions (cube:dimensions) and spatial and temporal extents. As with the other protocols, this works best when datasets are compliant with CF Conventions, particularly around coordinates and CRS metadata. Coordinates named latitude/lat and longitude/lon with no CF attributes are still recognized as WGS84, as long as their values are valid latitudes and longitudes.

Supported Processes​

The service advertises its process set at GET /processes. Submitted graphs are validated against this set, so graphs using unsupported processes are rejected before anything runs.

ProcessNotes
load_collectionOpen a collection as a datacube. Supports bands, temporal_extent, and spatial_extent (a bounding box), which apply the three filters below. A table ({org}/table/{namespace}.{table}) loads as a vector cube of its points or polygons, for aggregate_spatial only, and supports only spatial_extent (for points).
filter_bandsKeep only the named bands.
filter_temporalKeep the left-closed interval [start, end) of a temporal dimension. Either bound may be null for an open interval.
filter_bboxKeep a bounding box, given in WGS84 unless its crs says otherwise. GeoJSON extents and boxes that cross the antimeridian are not supported.
filter_labelsKeep the labels of a dimension that a condition accepts, e.g. one forecast lead time or a few ensemble members. Compare the label value with constants: use eq or array_contains (joined with or) for exact labels, or gt, gte, lt, lte or between (joined with and) for a range. Lead times are written as ISO 8601 durations such as "PT6H".
load_geojsonLoad GeoJSON geometries as a vector datacube for aggregate_spatial, as inline geometries are. properties is not supported.
load_uploaded_filesLoad point geometries, or the polygons of GeoParquet files (a list or a folder, in one bucket), from files previously uploaded to one of your organization's buckets, referenced by arraylake:// paths. See Large Geometry Sets.
aggregate_spatialSample the datacube at Point geometries, or average it over Polygon and MultiPolygon geometries — inline GeoJSON, a vector cube from load_uploaded_files, or a table's load_collection — returning a vector datacube. See Averaging Over Polygons.
applyApply elementwise math (see the callback math below) to every value of every band.
apply_dimensionOver bands: append computed bands, or replace the bands with them. To append, the callback returns its data extended with array_append. The openEO client's append_index and compute_indices helpers also work. See Derived Bands.
rename_labelsRename labels: source[i] becomes target[i], for the bands or any non-spatial dimension. A time dimension relabelled with anything but dates becomes a text dimension.
rename_dimensionRename a dimension, such as time to t. It reads the data once more, integer bands become 64-bit, and other coordinates along the dimension are left out. bands can't be renamed.
drop_dimensionDrop a dimension that has one label left, such as time after filtering to one day, or bands of a one-band cube.
merge_cubesCombine the bands of two datacubes, such as two collections on the same grid. The cubes must share their grid already: load them with the same spatial_extent and temporal_extent, and apply the same filters to both, because merging doesn't resample. Bands in both cubes are combined by an overlap_resolver callback over x (from cube1) and y (from cube2), with the callback math and reducers below: for example lambda x, y: x + y, or the openEO client's shortcut overlap_resolver="max", which takes the greater value, or the one that isn't missing. The cubes must be merged before aggregate_spatial.
add_dimensionadd_dimension(name="bands", label=..., type="bands") names the band that a band reduction (reduce_dimension over bands, or ndvi without target_band) left. Until then, processes that work on bands (such as filter_bands) fail with DimensionNotAvailable. Type temporal (a date label) or other gives every band a new dimension of one label (integer bands become 64-bit). Type spatial is not supported.
reduce_dimensionReduce a dimension of the datacube. bands reduces with band math (array_element and the callback math and comparisons below) into one band named value (name it with add_dimension); any other dimension (an ensemble, a lead time, time) reduces with mean, sum, sd, variance, first, or last, each band keeping its name. See Reducers and Ensemble Statistics. A vector cube's geometry dimension can't be reduced.
aggregate_spatial_windowReduce the x and y dimensions in non-overlapping windows of size pixels with mean, sum, sd, variance, first, or last, such as a 4x coarser overview. Except with first and last, the windows must fit the data's storage chunks along x and y: size divides the chunk length or is a multiple of it, and a crop (spatial_extent or filter_bbox) starts on a chunk boundary, unless the axis is a single chunk. With boundary: "pad", a partial window must fall at the axis's last stored pixel: on a south-up grid, align to the bottom (align="lower-left") or use a size that divides the axis. An axis can be windowed only once per graph, before any process that relabels it. Layouts that don't fit are refused before the job runs. See Building Multiscale Overviews.
reduce_spatialReduce both spatial dimensions with mean, sum, sd, variance, first, or last, such as the mean of a whole region over time.
aggregate_temporal_periodAggregate the temporal dimension into calendar periods (hour, day, week, dekad, month, season, tropical-season, year, decade, decade-ad) with mean, sum, sd, variance, first, or last. Each period is labelled by its start time. See Monthly Means and Climatologies.
aggregate_temporalAggregate the temporal dimension into the given [start, end) intervals with mean, sum, sd, variance, first, or last, each labelled by its start or by the matching labels.
climatological_normalAverage each day of the year, month, season, tropical season, or the whole period over the years of climatology_period (1981-2010 by default).
anomalySubtract the matching normal (from climatological_normal) from each time step, by day of the year, month, season, tropical season, or one normal for the whole period (climatology-period or single-period). The periods hour, week, dekad, year, decade and decade-ad are not supported.
resample_cube_temporalMove a datacube onto another's time steps, each taking the nearest one (the earlier on a tie), whatever its value. valid_within is not supported.
ndviCompute (nir - red) / (nir + red) from the named bands: one band named ndvi, or a new band named by target_band beside the others. Without nir and red, the bands must be named nir and red; bands aren't matched by their eo:common_name.
mean, sum, sd, variance, min, max, median, first, last, productReducers. Which process takes which is in Reducers.
array_element, array_append, array_create, array_modify, linear_scale_range, clip, add, subtract, multiply, divide, power, normalized_difference, absolute, sqrt, exp, ln, log, cos, sin, tan, arccos, arcsin, arctan, sinh, cosh, tanh, arcosh, arsinh, artanh, floor, ceil, int, round, sgn, e, pi, constantCallback math for apply, apply_dimension, reduce_dimension over bands, and merge_cubes' overlap_resolver. array_element picks a band by index or label, array_append adds a computed band, clip limits values to a range, and linear_scale_range clips and rescales them. round rounds half to even, and int rounds towards zero. Integer bands are promoted to float64. A computed band gets a units attribute when it follows from its inputs' units; adding, subtracting, multiplying or dividing by a constant, which may be a unit conversion, leaves it unset.
eq, neq, gt, gte, lt, lte, between, and, or, xor, not, is_nan, is_nodata, is_valid, ifComparisons and logic in the same callbacks as the math above, such as lambda x: if_(x > 280, x) to keep values over a threshold. A comparison with a missing value is null, as openEO defines; if takes a null condition as false, and gives a missing value where it rejects unless given reject. A comparison used as a number is 1 or 0, or missing where it is null, so x * (x > 280) masks with zeros. eq and neq take a numeric delta.
eq, array_contains, between, gt, gte, lt, lte, and, orAlso the conditions of filter_labels, comparing the label value with constants.
save_resultSerialize the result to parquet, csv, geojson, or netcdf. A vector result (from aggregate_spatial) saved as parquet is GeoParquet, its points or polygons in a WKB geom column, so geopandas.read_parquet reads it as a GeoDataFrame; geojson makes them the features' geometries. A datacube can also be saved without aggregating it, subject to the same result size limit; netcdf preserves its dimensions. Batch jobs can instead commit the result to a Repo with format icechunk, to a new group or into an existing one (see Saving Results to a Repo), or to an Iceberg table with format iceberg (see Saving Results to a Table).

Temporal intervals follow the openEO standard: the end is excluded, so ["2024-09-01", "2024-10-01"] selects all of September.

Reducers​

A reducer folds many values into one. Each process takes these:

Reduceraggregate_spatial at pointsaggregate_spatial over polygonsreduce_dimension over bands, and overlap_resolverAlong a dimension: reduce_dimension, reduce_spatial, aggregate_temporal, aggregate_temporal_periodaggregate_spatial_window
mean✓✓✓✓✓
sum✓✓✓✓
sd, variance✓✓✓
min, max✓✓
median✓
first, last✓✓✓✓
product✓
  • A point samples one grid cell, so every reducer at a point gives that cell's value.
  • Everywhere else, reducers skip missing values unless they pass ignore_nodata=False, except over polygons and with first and last along a dimension; see Missing Values.
  • sd and variance are the sample statistics. Along a dimension or over windows they are computed in one pass, so they lose precision on float32 data whose values are large relative to their spread (temperatures in kelvin, for example).
  • Along a dimension and over windows, first and last take the value at the first or last label, missing or not; in aggregate_spatial_window, each window's upper-left or lower-right pixel on a north-up grid. Over bands, they take the first or last band that has a value, or with ignore_nodata=False the first or last band.
  • climatological_normal always takes the mean.
  • count, extrema, quantiles, and min, max or median along a dimension are not supported.

Missing Values​

openEO reducers skip missing values (ignore_nodata) by default, and so does this service: a mean is over the values that are present, and a result with no value to fold, such as a period with every time step missing, is missing. Pass ignore_nodata=False to make any missing value make its result missing instead, as in reducer=lambda data: data.mean(ignore_nodata=False).

Two reducers can't skip missing values yet, so they must pass ignore_nodata=False; leaving it at the default is refused:

  • mean over polygons in aggregate_spatial: a missing value in any cell a polygon covers makes its mean missing.
  • first and last along a dimension (reduce_dimension other than over bands, reduce_spatial, the temporal aggregations and aggregate_spatial_window), which take their value missing or not.

Example: Sampling Points from a Datacube​

The canonical openEO workflow on Arraylake extracts values at many point locations from a large gridded dataset, returning a small table instead of the underlying chunks.

Build the process graph by chaining datacube operations:

import pandas as pd

points = pd.read_csv("points.csv") # columns: id, longitude, latitude

geometries = {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {"id": row.id},
"geometry": {
"type": "Point",
"coordinates": [row.longitude, row.latitude],
},
}
for row in points.itertuples(index=False)
],
}

cube = connection.load_collection(
"{org}/{repo}",
bands=["t2m"],
temporal_extent=["2024-09-01", "2024-10-01"],
)

result = cube.aggregate_spatial(geometries=geometries, reducer="mean").save_result(format="parquet")

Nothing has executed yet — result is a process graph that can be run synchronously or as a batch job.

note

Inline GeoJSON works up to a few thousand points; beyond that the request exceeds the graph size cap and is rejected. For larger point sets, see Large Geometry Sets.

Synchronous Execution​

Small requests can be executed synchronously, streaming the result back within the request:

result.download("samples.parquet")

Requests that take too long to run synchronously are rejected with an error directing you to submit a batch job instead.

Batch Jobs​

Larger requests should run as batch jobs, which execute durably in the background and write their results to your organization's bucket:

job = result.create_job(title="t2m point sampling")
job.start_and_wait()

Result assets are presigned object-storage URLs, so fetch them with a plain HTTP GET. Don't use results.download_files() or asset.download(): the openEO client sends its bearer token to the presigned URL, and the storage service rejects the request with a 403.

from pathlib import Path

import requests

out = Path("output")
out.mkdir(exist_ok=True)
for name, asset in job.get_results().get_metadata()["assets"].items():
response = requests.get(asset["href"], timeout=600) # no Authorization header
response.raise_for_status()
(out / name).write_bytes(response.content)

table = pd.concat(pd.read_parquet(path) for path in sorted(out.glob("*.parquet")))
note

A large job may be fanned out across many workers. When that happens, the results are split into one asset per part rather than a single file. Fetch every asset, as above, instead of assuming a single result file.

Jobs can be listed with connection.list_jobs(), and a finished job's results remain downloadable until the job is deleted.

Example: Averaging Over Polygons​

aggregate_spatial over polygons gives each polygon the mean of the grid cells it covers, such as a monthly mean temperature per county:

import json

import geopandas as gpd

counties = gpd.read_file("counties.geojson").to_crs("EPSG:4326")
geometries = json.loads(counties.geometry.to_json())

cube = connection.load_collection(
"{org}/{repo}",
bands=["t2m"],
temporal_extent=["2024-09-01", "2024-10-01"],
)
result = cube.aggregate_spatial(
geometries=geometries,
reducer=lambda data: data.mean(ignore_nodata=False),
).save_result(format="parquet")
result.download("county_means.parquet")

The result has a row per polygon (and per remaining label of the other dimensions, such as each time step). Each row is labelled by the polygon's position in geometries and carries the polygon itself: as a GeoParquet or GeoJSON geometry, or as WKT in a geometry_wkt column in CSV. NetCDF, and a result saved to a Repo, keep each polygon once, as a CF geometry container that xvec reads and the Tiles service renders with style=polygons. Polygons from an uploaded GeoParquet file or an Iceberg table work the same way, and each row also carries the file's or table's other columns.

A few rules apply over polygons:

  • The reducer is mean, and must pass ignore_nodata=False: a missing value in any cell a polygon covers makes its mean missing.
  • A cell counts towards a polygon when the polygon covers the cell's centre. A polygon that covers no cell centre, such as one smaller than a cell, gets a missing value.
  • Polygons that overlap don't share cells: a cell counts towards the first polygon that covers it. Aggregate overlapping polygons, such as nested regions, in separate requests.
  • The datacube's grid must be regular: evenly spaced x and y coordinates. Polygons are reprojected to the datacube's CRS; inline GeoJSON is longitude/latitude, and a GeoParquet file or table declares its CRS.
  • geometries must be all polygons or all points, not a mix.

Example: Ensemble Statistics​

reduce_dimension computes statistics along any dimension of a datacube, such as the members of an ensemble forecast. This example derives 10 m wind speed from its components, then takes its ensemble spread:

cube = connection.load_collection(
"{org}/{repo}",
bands=["wind_u_10m", "wind_v_10m"],
temporal_extent=["2026-10-01", "2026-10-02"],
)
speed = cube.reduce_dimension(
dimension="bands",
reducer=lambda data: (
data.array_element(label="wind_u_10m") ** 2
+ data.array_element(label="wind_v_10m") ** 2
).sqrt(),
)
spread = speed.reduce_dimension(
dimension="ensemble_member",
reducer="sd",
)
result = spread.aggregate_spatial(geometries=geometries, reducer="mean").save_result(format="parquet")

A few rules apply when reducing a dimension other than bands:

  • The reducer is one of mean, sum, sd, variance, first, or last, applied to data directly. It skips missing values, unless it passes ignore_nodata=False; first and last must pass it, and take the value at the first or last label whatever it is.
  • Every band must have the dimension, so select the bands to reduce with bands or filter_bands first.
  • A reduction reads every value it folds: an ensemble mean over 51 members reads 51 times its output. The result size limit counts only the result, but the extra reading takes time, so keep the other dimensions (time, area, points) bounded.
  • sd and variance are computed in one pass, so they lose precision on float32 data whose values are large relative to their spread (temperatures in kelvin, for example).

Example: Monthly Means and Climatologies​

aggregate_temporal_period aggregates the time dimension into calendar periods, such as monthly means from hourly data:

cube = connection.load_collection(
"{org}/{repo}",
bands=["t2m"],
temporal_extent=["2020-01-01", "2025-01-01"],
)
monthly = cube.aggregate_temporal_period(period="month", reducer="mean")
result = monthly.aggregate_spatial(geometries=geometries, reducer="mean").save_result(format="parquet")

climatological_normal averages each month (or day of the year, or season) over a span of years, as a baseline to compare against:

normals = cube.process(
"climatological_normal",
data=cube,
period="month",
climatology_period=["1991", "2020"],
)

anomaly then subtracts the matching normal from each time step, such as how much warmer each month was than usual:

anomalies = monthly.process("anomaly", data=monthly, normals=normals, period="month")

A few rules apply to temporal aggregations:

  • Each period is labelled by its start time (2020-01-01T00:00:00 for January 2020) rather than openEO's 2020-01-style text, so you can still filter the result by time and save it to a Repo.
  • aggregate_temporal_period returns every period from the first time step's to the last's, at most 1,000,000 of them. A period with no time step holds no data.
  • The reducer is one of mean, sum, sd, variance, first, or last, applied to data directly. It skips missing values unless it passes ignore_nodata=False; first and last must pass it, and take the period's first or last time step whatever its value.
  • Weeks are ISO weeks, starting on Monday. Seasons run December-February, March-May, June-August and September-November.
  • aggregate_temporal takes your own [start, end) intervals. They may overlap only where the data has no time step.
  • anomaly needs normals for the same bands, on the same grid, and works on daily, monthly, seasonal, or whole-period normals. Compute anomalies before aggregate_spatial.
  • climatological_normal labels its result as openEO defines (01-12 for months, djf, mam, jja and son for seasons), so the result has no time axis to filter. Winters count towards the year they end in, and February 29 is left out of daily normals.
  • sd and variance are computed in one pass, as in reduce_dimension.

Example: Derived Bands​

apply_dimension over bands adds a computed band beside the bands it was computed from. This example appends 10 m wind speed to its components, keeping the 6-hour lead time and three ensemble members:

cube = connection.load_collection(
"{org}/{repo}",
bands=["wind_u_10m", "wind_v_10m"],
temporal_extent=["2026-10-01", "2026-10-02"],
)
cube = cube.filter_labels(condition=lambda lead: lead == "PT6H", dimension="lead_time")
cube = cube.filter_labels(
condition=lambda member: openeo.processes.array_contains([0, 1, 2], member),
dimension="ensemble_member",
)
cube = cube.apply_dimension(
dimension="bands",
process=lambda data: data.array_append(
(
data.array_element(label="wind_u_10m") ** 2
+ data.array_element(label="wind_v_10m") ** 2
).sqrt(),
label="wind_speed_10m",
),
)

The result has three bands: wind_u_10m, wind_v_10m, and wind_speed_10m. The new band carries the units of its inputs (m s-1) when the dataset records them. To keep only the new band, follow with filter_bands(["wind_speed_10m"]).

What's Not Supported​

These standard openEO processes aren't available yet. A graph that uses one is rejected before anything runs. GET /processes lists exactly what is.

ProcessesWhy
if, neq, not, xor, all, any, is_nan, is_nodata, is_valid, and comparisons (eq, gt, ...) outside filter_labelsNo conditional values yet: a computed band can't depend on a comparison.
mask, mask_polygon, filter_spatial, trim_cubeMasking needs conditional values too.
arctan2, modNot available as array functions yet.
count, extrema, quantiles, order, sort, rearrangeNot available as reductions yet; see Reducers.
resample_spatial, resample_cube_spatialNot supported yet. To coarsen a grid by a whole factor, use aggregate_spatial_window (with mean or sum, a factor that divides the storage chunk length or is a multiple of it). Reprojecting to another CRS isn't supported.
apply_kernel, apply_neighborhoodNot supported.
array_apply, array_concat, array_filter, array_find, array_labels, array_interpolate_linearNot supported. Over bands, apply_dimension with array_append, array_modify and array_create covers most band math.
dimension_labels, create_data_cube, run_udf, text_begins, text_concat, text_contains, text_endsNot supported.

Other limits to plan around:

  • Missing values can't be skipped; see Missing Values.
  • Datacubes are never resampled or reprojected to match each other: merge_cubes and anomaly need cubes on the same grid, and aggregate_spatial reprojects the geometries instead of the data.
  • Spatial extents are bounding boxes: GeoJSON extents and boxes crossing the antimeridian aren't supported.
  • The vector datacube aggregate_spatial returns can be reduced, aggregated in time, and saved, but not merged, compared with anomaly, or reduced along its geometry dimension.
  • Text values can't be computed.

Saving Results to a Repo​

A batch job can commit its result to a group of an Arraylake Repo instead of writing a file, with save_result format icechunk: a new group, or an existing one (see Writing into an Existing Group). The result is written straight from the datacube, chunk by chunk, so it isn't subject to the result size limit, and it's committed once, whole: if anything fails, nothing is committed.

result = cube.save_result(
format="icechunk",
options={
"repo": "{org}/{results-repo}", # required
"branch": "main", # default: main
"group": "wind/speed-10m", # the group to write; default: the root group
"mode": "create", # default: create, a new group (or add, overwrite)
"message": "10 m wind speed", # the commit message
},
)
job = result.create_job(title="10 m wind speed")
job.start_and_wait()
  • You need write access to the target Repo, as well as read access to the Repos the graph reads. On a public deployment, authenticate with your own token to save.
  • With the default mode create, the group must not exist yet, and saving to the root group requires an empty Repo.
  • Saving to a Repo runs only as a batch job; synchronous execution refuses it.
  • The result is chunked along each dimension as its source was, and every variable is compressed with Zstandard.
  • The commit message names the job and run that made the commit. A job re-delivered after it committed reuses that commit rather than writing a second one.
  • chunks sets the result's storage chunk length along dimensions you name, such as {"lat_4x": 1800, "lon_4x": 1800, "time": 1}, overriding the source's.
  • multiscales records the new group as a level of a multiscale layout, on the data's group or the level's parent (true for the defaults); see Building Multiscale Overviews.
  • Text variables are stored too, such as the text columns that rows averaged over a table's or GeoParquet file's polygons carry (a county's name, say), or a climatology's season labels.

The finished job's results have one asset, the collection it committed, pinned to the commit's snapshot. Its arraylake:collection_id loads exactly what the job wrote:

asset = job.get_results().get_metadata()["assets"]["result"]
saved = connection.load_collection(asset["arraylake:collection_id"])

The job itself reports where it committed under arraylake:result_commit: the Repo, branch, group and snapshot id.

Writing into an Existing Group​

options.mode says what happens to the group:

modeWhat it does
create (default)Writes a new group. Refused if the group exists (or, for the root group, holds anything).
addWrites the result's bands beside the group's variables. Refused if any band shares a name with a variable already there.
overwriteWrites the result over bands the group already holds, in place. Refused if the group has no variable of a band's name, or if a band's data type or dimension order differs from the stored one. Each band keeps its stored chunking and encoding, so chunks isn't allowed.

add and overwrite change only the bands in the result: the group keeps its attributes, its coordinates, and every variable the result doesn't name. So the result has to lie on the group's grid: every dimension it shares with the group must have the same length, and every one-dimensional coordinate both hold (the dimension labels, such as time, lat and lon) must have the same values. A graph that crops, windows or aggregates its data is refused, so save that result to a new group. multiscales is only for create.

For example, to derive 10 m wind speed from the u10 and v10 bands and save it beside them in the same group:

wind = connection.load_collection("{org}/{repo}/main/era5", bands=["u10", "v10"])
speed = wind.reduce_dimension(
dimension="bands",
reducer=lambda b: (b.array_element(label="u10") ** 2 + b.array_element(label="v10") ** 2).sqrt(),
)
speed = speed.add_dimension(name="bands", label="wind_speed", type="bands") # name the new band
result = speed.save_result(
format="icechunk",
options={"repo": "{org}/{repo}", "group": "era5", "mode": "add", "message": "Add 10 m wind speed"},
)

Every mode commits once: if anything fails, or the branch moves while the job writes, nothing is committed.

Building Multiscale Overviews​

Batch jobs can build the overview levels of a GeoZarr multiscale pyramid, which the Tiles service uses to serve zoomed-out maps cheaply. The Tiles service finds levels below the group it's pointed at: every group under it with data variables and a GeoZarr spatial:transform is a level, picked by pixel size. So save the levels below your dataset, and the dataset itself serves zoomed-out tiles from them, even when it's the Repo's root group.

Each level is one job: it averages the level before it over windows with aggregate_spatial_window, renames its spatial dimensions, and saves to a new group below the dataset with options.multiscales. Here the dataset is the Repo's root group, and the first level is built from it:

level = connection.load_collection("{org}/{repo}/main") # the dataset, at the root
level = level.aggregate_spatial_window(
size=[2, 2],
reducer=lambda data: data.mean(ignore_nodata=False),
)
level = level.rename_dimension(source="lat", target="lat_2x")
level = level.rename_dimension(source="lon", target="lon_2x")
result = level.save_result(
format="icechunk",
options={
"repo": "{org}/{repo}",
"group": "overviews/2x",
"multiscales": {"group": "", "derived_from": "", "resampling_method": "average"},
},
)
result.create_job(title="2x overview").start_and_wait()

The next level loads {org}/{repo}/main/overviews/2x and does the same into overviews/4x, with "multiscales": {"group": "", "derived_from": "overviews/2x", "resampling_method": "average"}. The root's layout then lists overviews/2x and overviews/4x. For a dataset in a group, name it instead: levels such as t2m/overviews/2x, with "multiscales": {"group": "t2m", "derived_from": "t2m"}.

Alternatively, keep the native resolution and its levels side by side in a pyramid container, such as pyramid/0, pyramid/1, pyramid/2. Leave out multiscales.group, and the layout goes on the level's parent, pyramid:

result = level.save_result(
format="icechunk",
options={
"repo": "{org}/{repo}",
"group": "pyramid/1",
"multiscales": {"derived_from": "pyramid/0", "resampling_method": "average"},
},
)

Point the Tiles service at pyramid, the group the levels are below, to use them.

  • The source must have GeoZarr proj: and spatial: metadata, such as spatial:transform. Each level's spatial:transform, spatial:shape, spatial:bbox and spatial:dimensions are updated to its grid, which is what the Tiles service picks levels by.
  • A window's value is labelled at the centre of its block, so a level starts at the same corner as its source. Windows at the far edges that are short of size pixels are averaged over the pixels they have (boundary="pad", the default), or left out with boundary="trim".
  • A missing value makes its window's mean missing. first takes each block's upper-left pixel on a north-up grid, for categorical data.
  • Rename each level's spatial dimensions, as above: levels that share dimension names with different sizes can't be opened together as an xarray DataTree.
  • multiscales writes the multiscales layout of multiscales.group (by default the level's parent), following the multiscales convention (v0.1): each level's group relative to the layout's group, the level it was derived_from with the scale and translation between them, its spatial:transform and spatial:shape, and its resampling_method. The layout is updated in a second commit, right after the level's.
  • The level's group must be below multiscales.group, and derived_from must be another group below it or multiscales.group itself. The convention can't list the layout's own group as a level, so a level built from it names no derived_from (nor scale and translation) and is placed by its spatial:transform alone.
  • A level keeps its source's chunk lengths along x and y, even with renamed dimensions; set chunks to choose others.
  • The windows must fit those chunks: with size=[2, 2], every level's x and y chunk lengths must be even (or each axis a single chunk), so the next level can be built from it. A graph whose windows don't fit is refused with the reason. On a south-up grid (latitude increasing), pass align="lower-left" when size doesn't divide the latitude axis.

Saving Results to a Table​

A batch job can instead commit its result's rows to one of your organization's Iceberg tables, with save_result format iceberg. Point samples land in a table you can query with DuckDB, Spark or pyiceberg:

result = cube.aggregate_spatial(geometries=stations, reducer="mean").save_result(
format="iceberg",
options={
"table": "arraylake://{org}/table/{namespace}.{table}", # required
"mode": "append", # append to an existing table, or "create" it; default: append
"branch": "main", # the branch to append to; default: main
},
)
job = result.create_job(title="t2m at stations")
job.start_and_wait()
  • The rows are the result flattened as parquet flattens it: one row per cell, a point-sampled result's points in a WKB geom column. A created table gets the result's schema, a geo property naming geom, and an openeo.created-by-run property naming the run; appending requires the table's schema.
  • Appending needs write access to the table (or its namespace); creating one needs write access to the namespace. Tables in a subscribed namespace can't be written.
  • The job commits one snapshot. Its summary records the job (openeo.job, openeo.run), the snapshot of every Repo and table the graph read (arraylake.source.{org}.{repo}, arraylake.source.{org}.{namespace}.{table}), and the run's provenance digest. A job re-delivered after it committed reuses that snapshot while the table keeps it, and one re-delivered after creating its table appends to it.
  • The result is read in full before it's written, so it's subject to the result size limit, and nanosecond timestamps are stored as microseconds. A column with no Iceberg type (a duration, an unsigned 64-bit integer) is refused.

The job reports the table, branch and snapshot id under arraylake:result_commit.

Large Geometry Sets​

Process graphs travel inside a size-capped request, which limits inline GeoJSON geometries to a few thousand points, or fewer detailed polygons. For larger geometry sets, upload the geometry file to one of your organization's buckets and reference it with the load_uploaded_files process instead of inlining it.

Upload the file with the Arraylake client's obstore helper, which scopes writes to the bucket and its configured prefix:

import obstore

store = await client.aclient.get_obstore_for_bucket(
org="{org}", nickname="{bucket_nickname}", access="write"
)
with open("points.csv", "rb") as f:
await obstore.put_async(store, "openeo-uploads/points.csv", f.read())

Then reference it as an arraylake://{org}/bucket/{bucket_nickname}/{key} path, where org is the organization that owns the compute service you submit to and key is relative to the bucket's configured prefix:

points = connection.datacube_from_process(
"load_uploaded_files",
paths=["arraylake://{org}/bucket/{bucket_nickname}/openeo-uploads/points.csv"],
format="csv",
)

result = cube.aggregate_spatial(geometries=points, reducer="mean").save_result(format="parquet")
job = result.create_job(title="large point sampling")

Supported formats are geojson, csv, and parquet. A GeoParquet file's Point geometries are read from its primary geometry column, and reprojected to longitude/latitude if the file declares another CRS. A GeoParquet file can instead hold Polygon and MultiPolygon geometries, which aggregate_spatial averages over; polygons can come from several paths or a folder in one bucket, as long as every file has the same columns and geo metadata (such as its CRS), and none holds points. For CSV, and Parquet without GeoParquet metadata, coordinate columns named lon/longitude/x and lat/latitude/y are detected case-insensitively; other names can be given explicitly with options={"x_column": "...", "y_column": "..."} (which also overrides a GeoParquet geometry column). Multiple paths are concatenated in order. A parquet path may name a folder, as the Buckets browser copies a partitioned dataset's URI: every Parquet file in it is read, in key order, skipping writer bookkeeping such as _SUCCESS.

Using load_uploaded_files requires the org-wide CAN_READ_REPOS permission, since the referenced files are read with the organization's bucket credentials.

Geometries from an Iceberg Table​

Points or polygons kept in one of your organization's Iceberg tables can be loaded with load_collection and passed as the geometries to aggregate over, without exporting them to a file:

stations = connection.datacube_from_process("load_collection", id="{org}/table/{namespace}.{table}/tag.v1")

result = cube.aggregate_spatial(geometries=stations, reducer="mean").save_result(format="parquet")

The table is read at the snapshot its ref resolves to when the job starts, so every part of a large job reads the same rows. Its geometries come from its geometry column when the table has a geo property, whose value is GeoParquet's geo metadata as JSON, naming its primary column, encoding (WKB or point), and CRS. Otherwise they are points from coordinate columns named lon/longitude/x and lat/latitude/y. A geometry column holds all Points, or all Polygons and MultiPolygons, which aggregate_spatial averages over.

spatial_extent keeps the points inside a bounding box (in WGS84, or the crs it names), each keeping its geometry id; it isn't supported for a table of polygons, and bands and temporal_extent don't apply to a table's geometries:

stations = connection.datacube_from_process(
"load_collection",
id="{org}/table/{namespace}.{table}",
spatial_extent={"west": -105, "south": 37, "east": -102, "north": 41},
)

Reading a table requires read access to it (a grant on the table or its namespace). POST /validation and POST /preview check it and read the table's geometries, as a job does.

Result Artifacts​

Every execution writes its result as an object, an artifact, in one of your organization's bucket configurations, under a reserved prefix:

{bucket_prefix}/__earthmover_artifacts__/openeo_artifacts/{execution_id}/

By default, artifacts are written to the bucket where the Repo referenced by the graph's load_collection lives. Batch jobs can target a different bucket in your organization by passing its bucket config nickname in the bucket_nickname field (a non-standard extension) when creating the job:

job = result.create_job(
title="t2m point sampling",
additional={"bucket_nickname": "scratch"},
)

Synchronous executions stage their result the same way before streaming it back, always using the default bucket.

The asset hrefs in a finished job's results are presigned object-storage URLs, so any HTTP client can download them directly without a token. Deleting a job removes it from the API but does not remove its artifacts from the bucket; they can be managed like any other objects there.