openEO: Server-Side Processing
openEO is an open API standard for describing geospatial analyses as process graphs and executing them on the backend where the data lives. Where the other Compute protocols serve slices of your data for you to process locally, openEO inverts this: you describe the analysis with the standard openEO Python client, Compute runs it next to your Arraylake data, and you download only the result.
References
These are links to the openEO standard and websites:
Arraylake implements a subset of the openEO API (v1.2.0): collection discovery, synchronous processing, and batch jobs. Its processes cover sampling points and averaging over polygons from gridded datasets, band math, reducing dimensions, aggregating time into calendar periods, climatologies and anomalies, and building multiscale overviews. What's Not Supported lists the standard openEO processes it doesn't run yet.
Activating openEO for Arraylake Datasets
openEO can be activated using the Arraylake command line interface (CLI)
al compute enable {org} openeo --sku-code {sku}
The service is enabled on an organization-wide basis; every Repo the caller has access to is served as a collection.
Connecting
The easiest way to connect is with the arraylake Python client, which resolves the service URL and authenticates the connection for you:
from arraylake import Client
client = Client()
connection = client.get_openeo_connection("{org}")
get_openeo_connection requires arraylake>=1.2.0 and the optional openeo dependency, installed with pip install 'arraylake[openeo]'.
Any stock openEO client can also connect directly using the service URL:
import openeo
connection = openeo.connect("https://compute.earthmover.io/v1/services/openeo/{org}")
connection.authenticate_bearer_token("{api_token}")
Unlike the other Compute protocols, openEO clients authenticate with a bearer token — use an Arraylake API token — rather than HTTP Basic auth. Public services require no authentication.
Collections
Datasets are addressed as openEO collections using an Arraylake path as the collection id:
{org}/{repo}
{org}/{repo}/{ref}
{org}/{repo}/{ref}/{path/to/group}
{org}/table/{namespace}.{table}
{org}/table/{namespace}.{table}/{ref}
{org}/table/{namespace}.{level}.{table}/{ref}
The ref defaults to main and the group defaults to the root group. In a table's id every . separates levels: the first is the namespace, the last is the table, and any between are nested namespaces (nnja-ai.tankA.adpupa.NC002001). A namespace or table whose own name contains a . can't be addressed by an id. A ref can name its kind, as branch.{name}, tag.{name}, or snapshot.{id}; a bare name is tried as a branch, then a tag, then (for a Repo) a snapshot ID. An Iceberg table loads as the points or polygons to aggregate over; describe_collection describes it as a vector cube along a geometry dimension.
connection.list_collections()
connection.describe_collection("{org}/{repo}")
describe_collection returns a STAC collection describing the datacube's dimensions (cube:dimensions) and spatial and temporal extents.
As with the other protocols, this works best when datasets are compliant with CF Conventions, particularly around coordinates and CRS metadata.
Coordinates named latitude/lat and longitude/lon with no CF attributes are still recognized as WGS84, as long as their values are valid latitudes and longitudes.
Supported Processes
The service advertises its process set at GET /processes. Submitted graphs are validated against this set, so graphs using unsupported processes are rejected before anything runs.
| Process | Notes |
|---|---|
load_collection | Open a collection as a datacube. Supports bands, temporal_extent, and spatial_extent (a bounding box), which apply the three filters below. A table ({org}/table/{namespace}.{table}) loads as a vector cube of its points or polygons, for aggregate_spatial only, and supports only spatial_extent (for points). |
filter_bands | Keep only the named bands. |
filter_temporal | Keep the left-closed interval [start, end) of a temporal dimension. Either bound may be null for an open interval. |
filter_bbox | Keep a bounding box, given in WGS84 unless its crs says otherwise. GeoJSON extents and boxes that cross the antimeridian are not supported. |
filter_labels | Keep the labels of a dimension that a condition accepts, e.g. one forecast lead time or a few ensemble members. Compare the label value with constants: use eq or array_contains (joined with or) for exact labels, or gt, gte, lt, lte or between (joined with and) for a range. Lead times are written as ISO 8601 durations such as "PT6H". |
load_geojson | Load GeoJSON geometries as a vector datacube for aggregate_spatial, as inline geometries are. properties is not supported. |
load_uploaded_files | Load point geometries, or the polygons of GeoParquet files (a list or a folder, in one bucket), from files previously uploaded to one of your organization's buckets, referenced by arraylake:// paths. See Large Geometry Sets. |
aggregate_spatial | Sample the datacube at Point geometries, or average it over Polygon and MultiPolygon geometries — inline GeoJSON, a vector cube from load_uploaded_files, or a table's load_collection — returning a vector datacube. See Averaging Over Polygons. |
apply | Apply elementwise math (see the callback math below) to every value of every band. |
apply_dimension | Over bands: append computed bands, or replace the bands with them. To append, the callback returns its data extended with array_append. The openEO client's append_index and compute_indices helpers also work. See Derived Bands. |
rename_labels | Rename labels: source[i] becomes target[i], for the bands or any non-spatial dimension. A time dimension relabelled with anything but dates becomes a text dimension. |
rename_dimension | Rename a dimension, such as time to t. It reads the data once more, integer bands become 64-bit, and other coordinates along the dimension are left out. bands can't be renamed. |
drop_dimension | Drop a dimension that has one label left, such as time after filtering to one day, or bands of a one-band cube. |
merge_cubes | Combine the bands of two datacubes, such as two collections on the same grid. The cubes must share their grid already: load them with the same spatial_extent and temporal_extent, and apply the same filters to both, because merging doesn't resample. Bands in both cubes are combined by an overlap_resolver callback over x (from cube1) and y (from cube2), with the callback math and reducers below: for example lambda x, y: x + y, or the openEO client's shortcut overlap_resolver="max", which takes the greater value, or the one that isn't missing. The cubes must be merged before aggregate_spatial. |
add_dimension | add_dimension(name="bands", label=..., type="bands") names the band that a band reduction (reduce_dimension over bands, or ndvi without target_band) left. Until then, processes that work on bands (such as filter_bands) fail with DimensionNotAvailable. Type temporal (a date label) or other gives every band a new dimension of one label (integer bands become 64-bit). Type spatial is not supported. |
reduce_dimension | Reduce a dimension of the datacube. bands reduces with band math (array_element and the callback math and comparisons below) into one band named value (name it with add_dimension); any other dimension (an ensemble, a lead time, time) reduces with mean, sum, sd, variance, first, or last, each band keeping its name. See Reducers and Ensemble Statistics. A vector cube's geometry dimension can't be reduced. |
aggregate_spatial_window | Reduce the x and y dimensions in non-overlapping windows of size pixels with mean, sum, sd, variance, first, or last, such as a 4x coarser overview. Except with first and last, the windows must fit the data's storage chunks along x and y: size divides the chunk length or is a multiple of it, and a crop (spatial_extent or filter_bbox) starts on a chunk boundary, unless the axis is a single chunk. With boundary: "pad", a partial window must fall at the axis's last stored pixel: on a south-up grid, align to the bottom (align="lower-left") or use a size that divides the axis. An axis can be windowed only once per graph, before any process that relabels it. Layouts that don't fit are refused before the job runs. See Building Multiscale Overviews. |
reduce_spatial | Reduce both spatial dimensions with mean, sum, sd, variance, first, or last, such as the mean of a whole region over time. |
aggregate_temporal_period | Aggregate the temporal dimension into calendar periods (hour, day, week, dekad, month, season, tropical-season, year, decade, decade-ad) with mean, sum, sd, variance, first, or last. Each period is labelled by its start time. See Monthly Means and Climatologies. |
aggregate_temporal | Aggregate the temporal dimension into the given [start, end) intervals with mean, sum, sd, variance, first, or last, each labelled by its start or by the matching labels. |
climatological_normal | Average each day of the year, month, season, tropical season, or the whole period over the years of climatology_period (1981-2010 by default). |
anomaly | Subtract the matching normal (from climatological_normal) from each time step, by day of the year, month, season, tropical season, or one normal for the whole period (climatology-period or single-period). The periods hour, week, dekad, year, decade and decade-ad are not supported. |
resample_cube_temporal | Move a datacube onto another's time steps, each taking the nearest one (the earlier on a tie), whatever its value. valid_within is not supported. |
ndvi | Compute (nir - red) / (nir + red) from the named bands: one band named ndvi, or a new band named by target_band beside the others. Without nir and red, the bands must be named nir and red; bands aren't matched by their eo:common_name. |
mean, sum, sd, variance, min, max, median, first, last, product | Reducers. Which process takes which is in Reducers. |
array_element, array_append, array_create, array_modify, linear_scale_range, clip, add, subtract, multiply, divide, power, normalized_difference, absolute, sqrt, exp, ln, log, cos, sin, tan, arccos, arcsin, arctan, sinh, cosh, tanh, arcosh, arsinh, artanh, floor, ceil, int, round, sgn, e, pi, constant | Callback math for apply, apply_dimension, reduce_dimension over bands, and merge_cubes' overlap_resolver. array_element picks a band by index or label, array_append adds a computed band, clip limits values to a range, and linear_scale_range clips and rescales them. round rounds half to even, and int rounds towards zero. Integer bands are promoted to float64. A computed band gets a units attribute when it follows from its inputs' units; adding, subtracting, multiplying or dividing by a constant, which may be a unit conversion, leaves it unset. |
eq, neq, gt, gte, lt, lte, between, and, or, xor, not, is_nan, is_nodata, is_valid, if | Comparisons and logic in the same callbacks as the math above, such as lambda x: if_(x > 280, x) to keep values over a threshold. A comparison with a missing value is null, as openEO defines; if takes a null condition as false, and gives a missing value where it rejects unless given reject. A comparison used as a number is 1 or 0, or missing where it is null, so x * (x > 280) masks with zeros. eq and neq take a numeric delta. |
eq, array_contains, between, gt, gte, lt, lte, and, or | Also the conditions of filter_labels, comparing the label value with constants. |
save_result | Serialize the result to parquet, csv, geojson, or netcdf. A vector result (from aggregate_spatial) saved as parquet is GeoParquet, its points or polygons in a WKB geom column, so geopandas.read_parquet reads it as a GeoDataFrame; geojson makes them the features' geometries. A datacube can also be saved without aggregating it, subject to the same result size limit; netcdf preserves its dimensions. Batch jobs can instead commit the result to a Repo with format icechunk, to a new group or into an existing one (see Saving Results to a Repo), or to an Iceberg table with format iceberg (see Saving Results to a Table). |
Temporal intervals follow the openEO standard: the end is excluded, so ["2024-09-01", "2024-10-01"] selects all of September.
Reducers
A reducer folds many values into one. Each process takes these:
| Reducer | aggregate_spatial at points | aggregate_spatial over polygons | reduce_dimension over bands, and overlap_resolver | Along a dimension: reduce_dimension, reduce_spatial, aggregate_temporal, aggregate_temporal_period | aggregate_spatial_window |
|---|---|---|---|---|---|
mean | ✓ | ✓ | ✓ | ✓ | ✓ |
sum | ✓ | ✓ | ✓ | ✓ | |
sd, variance | ✓ | ✓ | ✓ | ||
min, max | ✓ | ✓ | |||
median | ✓ | ||||
first, last | ✓ | ✓ | ✓ | ✓ | |
product | ✓ |
- A point samples one grid cell, so every reducer at a point gives that cell's value.
- Everywhere else, reducers skip missing values unless they pass
ignore_nodata=False, except over polygons and withfirstandlastalong a dimension; see Missing Values. sdandvarianceare the sample statistics. Along a dimension or over windows they are computed in one pass, so they lose precision onfloat32data whose values are large relative to their spread (temperatures in kelvin, for example).- Along a dimension and over windows,
firstandlasttake the value at the first or last label, missing or not; inaggregate_spatial_window, each window's upper-left or lower-right pixel on a north-up grid. Overbands, they take the first or last band that has a value, or withignore_nodata=Falsethe first or last band. climatological_normalalways takes themean.count,extrema,quantiles, andmin,maxormedianalong a dimension are not supported.
Missing Values
openEO reducers skip missing values (ignore_nodata) by default, and so does this service: a mean is over the values that are present, and a result with no value to fold, such as a period with every time step missing, is missing. Pass ignore_nodata=False to make any missing value make its result missing instead, as in reducer=lambda data: data.mean(ignore_nodata=False).
Two reducers can't skip missing values yet, so they must pass ignore_nodata=False; leaving it at the default is refused:
meanover polygons inaggregate_spatial: a missing value in any cell a polygon covers makes its mean missing.firstandlastalong a dimension (reduce_dimensionother than overbands,reduce_spatial, the temporal aggregations andaggregate_spatial_window), which take their value missing or not.
Example: Sampling Points from a Datacube
The canonical openEO workflow on Arraylake extracts values at many point locations from a large gridded dataset, returning a small table instead of the underlying chunks.
Build the process graph by chaining datacube operations:
import pandas as pd
points = pd.read_csv("points.csv") # columns: id, longitude, latitude
geometries = {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {"id": row.id},
"geometry": {
"type": "Point",
"coordinates": [row.longitude, row.latitude],
},
}
for row in points.itertuples(index=False)
],
}
cube = connection.load_collection(
"{org}/{repo}",
bands=["t2m"],
temporal_extent=["2024-09-01", "2024-10-01"],
)
result = cube.aggregate_spatial(geometries=geometries, reducer="mean").save_result(format="parquet")
Nothing has executed yet — result is a process graph that can be run synchronously or as a batch job.
Inline GeoJSON works up to a few thousand points; beyond that the request exceeds the graph size cap and is rejected. For larger point sets, see Large Geometry Sets.
Synchronous Execution
Small requests can be executed synchronously, streaming the result back within the request:
result.download("samples.parquet")
Requests that take too long to run synchronously are rejected with an error directing you to submit a batch job instead.
Batch Jobs
Larger requests should run as batch jobs, which execute durably in the background and write their results to your organization's bucket:
job = result.create_job(title="t2m point sampling")
job.start_and_wait()
Result assets are presigned object-storage URLs, so fetch them with a plain HTTP GET.
Don't use results.download_files() or asset.download(): the openEO client sends its
bearer token to the presigned URL, and the storage service rejects the request with a 403.
from pathlib import Path
import requests
out = Path("output")
out.mkdir(exist_ok=True)
for name, asset in job.get_results().get_metadata()["assets"].items():
response = requests.get(asset["href"], timeout=600) # no Authorization header
response.raise_for_status()
(out / name).write_bytes(response.content)
table = pd.concat(pd.read_parquet(path) for path in sorted(out.glob("*.parquet")))
A large job may be fanned out across many workers. When that happens, the results are split into one asset per part rather than a single file. Fetch every asset, as above, instead of assuming a single result file.
Jobs can be listed with connection.list_jobs(), and a finished job's results remain downloadable until the job is deleted.
Example: Averaging Over Polygons
aggregate_spatial over polygons gives each polygon the mean of the grid cells it covers, such as a monthly mean temperature per county:
import json
import geopandas as gpd
counties = gpd.read_file("counties.geojson").to_crs("EPSG:4326")
geometries = json.loads(counties.geometry.to_json())
cube = connection.load_collection(
"{org}/{repo}",
bands=["t2m"],
temporal_extent=["2024-09-01", "2024-10-01"],
)
result = cube.aggregate_spatial(
geometries=geometries,
reducer=lambda data: data.mean(ignore_nodata=False),
).save_result(format="parquet")
result.download("county_means.parquet")
The result has a row per polygon (and per remaining label of the other dimensions, such as each time step). Each row is labelled by the polygon's position in geometries and carries the polygon itself: as a GeoParquet or GeoJSON geometry, or as WKT in a geometry_wkt column in CSV. NetCDF, and a result saved to a Repo, keep each polygon once, as a CF geometry container that xvec reads and the Tiles service renders with style=polygons. Polygons from an uploaded GeoParquet file or an Iceberg table work the same way, and each row also carries the file's or table's other columns.
A few rules apply over polygons:
- The reducer is
mean, and must passignore_nodata=False: a missing value in any cell a polygon covers makes its mean missing. - A cell counts towards a polygon when the polygon covers the cell's centre. A polygon that covers no cell centre, such as one smaller than a cell, gets a missing value.
- Polygons that overlap don't share cells: a cell counts towards the first polygon that covers it. Aggregate overlapping polygons, such as nested regions, in separate requests.
- The datacube's grid must be regular: evenly spaced x and y coordinates. Polygons are reprojected to the datacube's CRS; inline GeoJSON is longitude/latitude, and a GeoParquet file or table declares its CRS.
geometriesmust be all polygons or all points, not a mix.
Example: Ensemble Statistics
reduce_dimension computes statistics along any dimension of a datacube, such as the members of an ensemble forecast.
This example derives 10 m wind speed from its components, then takes its ensemble spread:
cube = connection.load_collection(
"{org}/{repo}",
bands=["wind_u_10m", "wind_v_10m"],
temporal_extent=["2026-10-01", "2026-10-02"],
)
speed = cube.reduce_dimension(
dimension="bands",
reducer=lambda data: (
data.array_element(label="wind_u_10m") ** 2
+ data.array_element(label="wind_v_10m") ** 2
).sqrt(),
)
spread = speed.reduce_dimension(
dimension="ensemble_member",
reducer="sd",
)
result = spread.aggregate_spatial(geometries=geometries, reducer="mean").save_result(format="parquet")
A few rules apply when reducing a dimension other than bands:
- The reducer is one of
mean,sum,sd,variance,first, orlast, applied todatadirectly. It skips missing values, unless it passesignore_nodata=False;firstandlastmust pass it, and take the value at the first or last label whatever it is. - Every band must have the dimension, so select the bands to reduce with
bandsorfilter_bandsfirst. - A reduction reads every value it folds: an ensemble mean over 51 members reads 51 times its output. The result size limit counts only the result, but the extra reading takes time, so keep the other dimensions (time, area, points) bounded.
sdandvarianceare computed in one pass, so they lose precision onfloat32data whose values are large relative to their spread (temperatures in kelvin, for example).
Example: Monthly Means and Climatologies
aggregate_temporal_period aggregates the time dimension into calendar periods, such as monthly means from hourly data:
cube = connection.load_collection(
"{org}/{repo}",
bands=["t2m"],
temporal_extent=["2020-01-01", "2025-01-01"],
)
monthly = cube.aggregate_temporal_period(period="month", reducer="mean")
result = monthly.aggregate_spatial(geometries=geometries, reducer="mean").save_result(format="parquet")
climatological_normal averages each month (or day of the year, or season) over a span of years, as a baseline to compare against:
normals = cube.process(
"climatological_normal",
data=cube,
period="month",
climatology_period=["1991", "2020"],
)
anomaly then subtracts the matching normal from each time step, such as how much warmer each month was than usual:
anomalies = monthly.process("anomaly", data=monthly, normals=normals, period="month")
A few rules apply to temporal aggregations:
- Each period is labelled by its start time (
2020-01-01T00:00:00for January 2020) rather than openEO's2020-01-style text, so you can still filter the result by time and save it to a Repo. aggregate_temporal_periodreturns every period from the first time step's to the last's, at most 1,000,000 of them. A period with no time step holds no data.- The reducer is one of
mean,sum,sd,variance,first, orlast, applied todatadirectly. It skips missing values unless it passesignore_nodata=False;firstandlastmust pass it, and take the period's first or last time step whatever its value. - Weeks are ISO weeks, starting on Monday. Seasons run December-February, March-May, June-August and September-November.
aggregate_temporaltakes your own[start, end)intervals. They may overlap only where the data has no time step.anomalyneeds normals for the same bands, on the same grid, and works on daily, monthly, seasonal, or whole-period normals. Compute anomalies beforeaggregate_spatial.climatological_normallabels its result as openEO defines (01-12for months,djf,mam,jjaandsonfor seasons), so the result has no time axis to filter. Winters count towards the year they end in, and February 29 is left out of daily normals.sdandvarianceare computed in one pass, as inreduce_dimension.
Example: Derived Bands
apply_dimension over bands adds a computed band beside the bands it was computed from.
This example appends 10 m wind speed to its components, keeping the 6-hour lead time and three ensemble members:
cube = connection.load_collection(
"{org}/{repo}",
bands=["wind_u_10m", "wind_v_10m"],
temporal_extent=["2026-10-01", "2026-10-02"],
)
cube = cube.filter_labels(condition=lambda lead: lead == "PT6H", dimension="lead_time")
cube = cube.filter_labels(
condition=lambda member: openeo.processes.array_contains([0, 1, 2], member),
dimension="ensemble_member",
)
cube = cube.apply_dimension(
dimension="bands",
process=lambda data: data.array_append(
(
data.array_element(label="wind_u_10m") ** 2
+ data.array_element(label="wind_v_10m") ** 2
).sqrt(),
label="wind_speed_10m",
),
)
The result has three bands: wind_u_10m, wind_v_10m, and wind_speed_10m.
The new band carries the units of its inputs (m s-1) when the dataset records them.
To keep only the new band, follow with filter_bands(["wind_speed_10m"]).
What's Not Supported
These standard openEO processes aren't available yet. A graph that uses one is rejected before anything runs. GET /processes lists exactly what is.
| Processes | Why |
|---|---|
if, neq, not, xor, all, any, is_nan, is_nodata, is_valid, and comparisons (eq, gt, ...) outside filter_labels | No conditional values yet: a computed band can't depend on a comparison. |
mask, mask_polygon, filter_spatial, trim_cube | Masking needs conditional values too. |
arctan2, mod | Not available as array functions yet. |
count, extrema, quantiles, order, sort, rearrange | Not available as reductions yet; see Reducers. |
resample_spatial, resample_cube_spatial | Not supported yet. To coarsen a grid by a whole factor, use aggregate_spatial_window (with mean or sum, a factor that divides the storage chunk length or is a multiple of it). Reprojecting to another CRS isn't supported. |
apply_kernel, apply_neighborhood | Not supported. |
array_apply, array_concat, array_filter, array_find, array_labels, array_interpolate_linear | Not supported. Over bands, apply_dimension with array_append, array_modify and array_create covers most band math. |
dimension_labels, create_data_cube, run_udf, text_begins, text_concat, text_contains, text_ends | Not supported. |
Other limits to plan around:
- Missing values can't be skipped; see Missing Values.
- Datacubes are never resampled or reprojected to match each other:
merge_cubesandanomalyneed cubes on the same grid, andaggregate_spatialreprojects the geometries instead of the data. - Spatial extents are bounding boxes: GeoJSON extents and boxes crossing the antimeridian aren't supported.
- The vector datacube
aggregate_spatialreturns can be reduced, aggregated in time, and saved, but not merged, compared withanomaly, or reduced along itsgeometrydimension. - Text values can't be computed.
Saving Results to a Repo
A batch job can commit its result to a group of an Arraylake Repo instead of writing a file, with save_result format icechunk: a new group, or an existing one (see Writing into an Existing Group).
The result is written straight from the datacube, chunk by chunk, so it isn't subject to the result size limit, and it's committed once, whole: if anything fails, nothing is committed.
result = cube.save_result(
format="icechunk",
options={
"repo": "{org}/{results-repo}", # required
"branch": "main", # default: main
"group": "wind/speed-10m", # the group to write; default: the root group
"mode": "create", # default: create, a new group (or add, overwrite)
"message": "10 m wind speed", # the commit message
},
)
job = result.create_job(title="10 m wind speed")
job.start_and_wait()
- You need write access to the target Repo, as well as read access to the Repos the graph reads. On a public deployment, authenticate with your own token to save.
- With the default
modecreate, the group must not exist yet, and saving to the root group requires an empty Repo. - Saving to a Repo runs only as a batch job; synchronous execution refuses it.
- The result is chunked along each dimension as its source was, and every variable is compressed with Zstandard.
- The commit message names the job and run that made the commit. A job re-delivered after it committed reuses that commit rather than writing a second one.
chunkssets the result's storage chunk length along dimensions you name, such as{"lat_4x": 1800, "lon_4x": 1800, "time": 1}, overriding the source's.multiscalesrecords the new group as a level of a multiscale layout, on the data's group or the level's parent (truefor the defaults); see Building Multiscale Overviews.- Text variables are stored too, such as the text columns that rows averaged over a table's or GeoParquet file's polygons carry (a county's name, say), or a climatology's season labels.
The finished job's results have one asset, the collection it committed, pinned to the commit's snapshot.
Its arraylake:collection_id loads exactly what the job wrote:
asset = job.get_results().get_metadata()["assets"]["result"]
saved = connection.load_collection(asset["arraylake:collection_id"])
The job itself reports where it committed under arraylake:result_commit: the Repo, branch, group and snapshot id.
Writing into an Existing Group
options.mode says what happens to the group:
mode | What it does |
|---|---|
create (default) | Writes a new group. Refused if the group exists (or, for the root group, holds anything). |
add | Writes the result's bands beside the group's variables. Refused if any band shares a name with a variable already there. |
overwrite | Writes the result over bands the group already holds, in place. Refused if the group has no variable of a band's name, or if a band's data type or dimension order differs from the stored one. Each band keeps its stored chunking and encoding, so chunks isn't allowed. |
add and overwrite change only the bands in the result: the group keeps its attributes, its coordinates, and every variable the result doesn't name.
So the result has to lie on the group's grid: every dimension it shares with the group must have the same length, and every one-dimensional coordinate both hold (the dimension labels, such as time, lat and lon) must have the same values.
A graph that crops, windows or aggregates its data is refused, so save that result to a new group.
multiscales is only for create.
For example, to derive 10 m wind speed from the u10 and v10 bands and save it beside them in the same group:
wind = connection.load_collection("{org}/{repo}/main/era5", bands=["u10", "v10"])
speed = wind.reduce_dimension(
dimension="bands",
reducer=lambda b: (b.array_element(label="u10") ** 2 + b.array_element(label="v10") ** 2).sqrt(),
)
speed = speed.add_dimension(name="bands", label="wind_speed", type="bands") # name the new band
result = speed.save_result(
format="icechunk",
options={"repo": "{org}/{repo}", "group": "era5", "mode": "add", "message": "Add 10 m wind speed"},
)
Every mode commits once: if anything fails, or the branch moves while the job writes, nothing is committed.
Building Multiscale Overviews
Batch jobs can build the overview levels of a GeoZarr multiscale pyramid, which the Tiles service uses to serve zoomed-out maps cheaply.
The Tiles service finds levels below the group it's pointed at: every group under it with data variables and a GeoZarr spatial:transform is a level, picked by pixel size.
So save the levels below your dataset, and the dataset itself serves zoomed-out tiles from them, even when it's the Repo's root group.
Each level is one job: it averages the level before it over windows with aggregate_spatial_window, renames its spatial dimensions, and saves to a new group below the dataset with options.multiscales.
Here the dataset is the Repo's root group, and the first level is built from it:
level = connection.load_collection("{org}/{repo}/main") # the dataset, at the root
level = level.aggregate_spatial_window(
size=[2, 2],
reducer=lambda data: data.mean(ignore_nodata=False),
)
level = level.rename_dimension(source="lat", target="lat_2x")
level = level.rename_dimension(source="lon", target="lon_2x")
result = level.save_result(
format="icechunk",
options={
"repo": "{org}/{repo}",
"group": "overviews/2x",
"multiscales": {"group": "", "derived_from": "", "resampling_method": "average"},
},
)
result.create_job(title="2x overview").start_and_wait()
The next level loads {org}/{repo}/main/overviews/2x and does the same into overviews/4x, with "multiscales": {"group": "", "derived_from": "overviews/2x", "resampling_method": "average"}.
The root's layout then lists overviews/2x and overviews/4x.
For a dataset in a group, name it instead: levels such as t2m/overviews/2x, with "multiscales": {"group": "t2m", "derived_from": "t2m"}.
Alternatively, keep the native resolution and its levels side by side in a pyramid container, such as pyramid/0, pyramid/1, pyramid/2.
Leave out multiscales.group, and the layout goes on the level's parent, pyramid:
result = level.save_result(
format="icechunk",
options={
"repo": "{org}/{repo}",
"group": "pyramid/1",
"multiscales": {"derived_from": "pyramid/0", "resampling_method": "average"},
},
)
Point the Tiles service at pyramid, the group the levels are below, to use them.
- The source must have GeoZarr
proj:andspatial:metadata, such asspatial:transform. Each level'sspatial:transform,spatial:shape,spatial:bboxandspatial:dimensionsare updated to its grid, which is what the Tiles service picks levels by. - A window's value is labelled at the centre of its block, so a level starts at the same corner as its source. Windows at the far edges that are short of
sizepixels are averaged over the pixels they have (boundary="pad", the default), or left out withboundary="trim". - A missing value makes its window's
meanmissing.firsttakes each block's upper-left pixel on a north-up grid, for categorical data. - Rename each level's spatial dimensions, as above: levels that share dimension names with different sizes can't be opened together as an xarray
DataTree. multiscaleswrites themultiscaleslayout ofmultiscales.group(by default the level's parent), following the multiscales convention (v0.1): each level's group relative to the layout's group, the level it wasderived_fromwith thescaleandtranslationbetween them, itsspatial:transformandspatial:shape, and itsresampling_method. The layout is updated in a second commit, right after the level's.- The level's group must be below
multiscales.group, andderived_frommust be another group below it ormultiscales.groupitself. The convention can't list the layout's own group as a level, so a level built from it names noderived_from(norscaleandtranslation) and is placed by itsspatial:transformalone. - A level keeps its source's chunk lengths along x and y, even with renamed dimensions; set
chunksto choose others. - The windows must fit those chunks: with
size=[2, 2], every level's x and y chunk lengths must be even (or each axis a single chunk), so the next level can be built from it. A graph whose windows don't fit is refused with the reason. On a south-up grid (latitude increasing), passalign="lower-left"whensizedoesn't divide the latitude axis.
Saving Results to a Table
A batch job can instead commit its result's rows to one of your organization's Iceberg tables, with save_result format iceberg. Point samples land in a table you can query with DuckDB, Spark or pyiceberg:
result = cube.aggregate_spatial(geometries=stations, reducer="mean").save_result(
format="iceberg",
options={
"table": "arraylake://{org}/table/{namespace}.{table}", # required
"mode": "append", # append to an existing table, or "create" it; default: append
"branch": "main", # the branch to append to; default: main
},
)
job = result.create_job(title="t2m at stations")
job.start_and_wait()
- The rows are the result flattened as
parquetflattens it: one row per cell, a point-sampled result's points in a WKBgeomcolumn. A created table gets the result's schema, ageoproperty naminggeom, and anopeneo.created-by-runproperty naming the run; appending requires the table's schema. - Appending needs write access to the table (or its namespace); creating one needs write access to the namespace. Tables in a subscribed namespace can't be written.
- The job commits one snapshot. Its summary records the job (
openeo.job,openeo.run), the snapshot of every Repo and table the graph read (arraylake.source.{org}.{repo},arraylake.source.{org}.{namespace}.{table}), and the run's provenance digest. A job re-delivered after it committed reuses that snapshot while the table keeps it, and one re-delivered after creating its table appends to it. - The result is read in full before it's written, so it's subject to the result size limit, and nanosecond timestamps are stored as microseconds. A column with no Iceberg type (a duration, an unsigned 64-bit integer) is refused.
The job reports the table, branch and snapshot id under arraylake:result_commit.
Large Geometry Sets
Process graphs travel inside a size-capped request, which limits inline GeoJSON geometries to a few thousand points, or fewer detailed polygons.
For larger geometry sets, upload the geometry file to one of your organization's buckets and reference it with the load_uploaded_files process instead of inlining it.
Upload the file with the Arraylake client's obstore helper, which scopes writes to the bucket and its configured prefix:
import obstore
store = await client.aclient.get_obstore_for_bucket(
org="{org}", nickname="{bucket_nickname}", access="write"
)
with open("points.csv", "rb") as f:
await obstore.put_async(store, "openeo-uploads/points.csv", f.read())
Then reference it as an arraylake://{org}/bucket/{bucket_nickname}/{key} path, where org is the organization that owns the compute service you submit to and key is relative to the bucket's configured prefix:
points = connection.datacube_from_process(
"load_uploaded_files",
paths=["arraylake://{org}/bucket/{bucket_nickname}/openeo-uploads/points.csv"],
format="csv",
)
result = cube.aggregate_spatial(geometries=points, reducer="mean").save_result(format="parquet")
job = result.create_job(title="large point sampling")
Supported formats are geojson, csv, and parquet. A GeoParquet file's Point geometries are read from its primary geometry column, and reprojected to longitude/latitude if the file declares another CRS. A GeoParquet file can instead hold Polygon and MultiPolygon geometries, which aggregate_spatial averages over; polygons can come from several paths or a folder in one bucket, as long as every file has the same columns and geo metadata (such as its CRS), and none holds points. For CSV, and Parquet without GeoParquet metadata, coordinate columns named lon/longitude/x and lat/latitude/y are detected case-insensitively; other names can be given explicitly with options={"x_column": "...", "y_column": "..."} (which also overrides a GeoParquet geometry column). Multiple paths are concatenated in order. A parquet path may name a folder, as the Buckets browser copies a partitioned dataset's URI: every Parquet file in it is read, in key order, skipping writer bookkeeping such as _SUCCESS.
Using load_uploaded_files requires the org-wide CAN_READ_REPOS permission, since the referenced files are read with the organization's bucket credentials.
Geometries from an Iceberg Table
Points or polygons kept in one of your organization's Iceberg tables can be loaded with load_collection and passed as the geometries to aggregate over, without exporting them to a file:
stations = connection.datacube_from_process("load_collection", id="{org}/table/{namespace}.{table}/tag.v1")
result = cube.aggregate_spatial(geometries=stations, reducer="mean").save_result(format="parquet")
The table is read at the snapshot its ref resolves to when the job starts, so every part of a large job reads the same rows. Its geometries come from its geometry column when the table has a geo property, whose value is GeoParquet's geo metadata as JSON, naming its primary column, encoding (WKB or point), and CRS. Otherwise they are points from coordinate columns named lon/longitude/x and lat/latitude/y. A geometry column holds all Points, or all Polygons and MultiPolygons, which aggregate_spatial averages over.
spatial_extent keeps the points inside a bounding box (in WGS84, or the crs it names), each keeping its geometry id; it isn't supported for a table of polygons, and bands and temporal_extent don't apply to a table's geometries:
stations = connection.datacube_from_process(
"load_collection",
id="{org}/table/{namespace}.{table}",
spatial_extent={"west": -105, "south": 37, "east": -102, "north": 41},
)
Reading a table requires read access to it (a grant on the table or its namespace). POST /validation and POST /preview check it and read the table's geometries, as a job does.
Result Artifacts
Every execution writes its result as an object, an artifact, in one of your organization's bucket configurations, under a reserved prefix:
{bucket_prefix}/__earthmover_artifacts__/openeo_artifacts/{execution_id}/
By default, artifacts are written to the bucket where the Repo referenced by the graph's load_collection lives. Batch jobs can target a different bucket in your organization by passing its bucket config nickname in the bucket_nickname field (a non-standard extension) when creating the job:
job = result.create_job(
title="t2m point sampling",
additional={"bucket_nickname": "scratch"},
)
Synchronous executions stage their result the same way before streaming it back, always using the default bucket.
The asset hrefs in a finished job's results are presigned object-storage URLs, so any HTTP client can download them directly without a token. Deleting a job removes it from the API but does not remove its artifacts from the bucket; they can be managed like any other objects there.