Skip to main content

Environmental Data Retrieval (EDR)

info

EDR is currently in private beta. Features and capabilities may change with time.

EDR is a standard of the Open Geospatial Consortium (OGC)

The OGC API - Environmental Data Retrieval Standard provides a family of lightweight query interfaces to access spatio-temporal data resources by requesting data at a Position, within an Area, along a Trajectory or through a Corridor.

References​

These are links to OGC standards and websites:

Arraylake implements a subset of the full EDR standard, specifically the Position and Area queries. Discovery of EDR collections and capabilities is not supported at this time.

Activating EDR for Arraylake Datasets​

EDR can be activated using the Arraylake command line interface (CLI)

al compute enable {org} edr
note

Arraylake currently supports enabling EDR on an organization-wide basis. This is subject to change during the beta period.

EDR works best when datasets are compliant with CF Conventions, particularly around coordinates. Specifically there must be 1D coordinate variables with attributes axis: X, axis: Y, axis: Z, axis: T respectively.

EDR URL Structure​

Base URL​

EDR endpoints can be accessed via the following URL schema.

https://compute.earthmover.io/v1/services/edr/{org}/{repo}/{branch|commit|tag}/{path/to/group}/edr

Where

  • {org} is the name of your Arraylake organization
  • {repo} is the name of the Repo
  • {branch|commit|tag} is the branch, commit, or tag within the Repo to use to fulfill the request
  • {path/to/group} is the path to group within the Repo that contains an xarray Dataset

This URL will be called {base_url} in the following examples. All examples use the HTTP GET protocol.

Specific examples in this guide will use the Arraylake GFS Repo. The underlying GFS dataset has the following structure:

<xarray.Dataset> Size: 3TB
Dimensions:    (longitude: 1440, latitude: 721, time: 736, step: 209)
Coordinates:
  * latitude   (latitude) float64 6kB 90.0 89.75 89.5 ... -89.5 -89.75 -90.0
  * longitude  (longitude) float64 12kB 0.0 0.25 0.5 0.75 ... 359.2 359.5 359.8
  * step       (step) timedelta64[ns] 2kB 00:00:00 01:00:00 ... 16 days 00:00:00
  * time       (time) datetime64[ns] 6kB 2024-05-12T18:00:00 ... 2024-11-12T1...
Data variables:
    gust       (longitude, latitude, time, step) float32 639GB ...
    prate      (longitude, latitude, time, step) float32 639GB ...
    r2         (longitude, latitude, time, step) float32 639GB ...
    t2m        (longitude, latitude, time, step) float32 639GB ...
    tcc        (longitude, latitude, time, step) float32 639GB ...
Attributes:
    description:  GFS data ingested for forecasting demo

General Options​

  • By default all variables in the dataset are returned. Specific variables can be specified in the query string via parameter-name=t2m,tcc
  • Non-spatial parameters (e.g. time) can be passed as additional fields in the query string (e.g. time=2024-01-01T00:00:00)
  • The default output format is CoverageJSON. Other formats can be specified via the f parameter. Options include
    • GeoJSON: f=geojson
    • CSV: f=csv
    • NetCDF: f=nc

Metadata Queries​

Metadata queries return information about the dataset and its variables to help users understand the data available and make the service self-describing.

note

Typically this functionality is handled in EDR using the /collections endpoint. This is not yet supported in Arraylake, but is on the roadmap.

Examples

Get the metadata for the GFS dataset:

GET https://compute.earthmover.io/v1/services/edr/earthmover-demos/gfs/main/timeseries/edr/
response
{
  "links": [],
  "id": "unknown",
  "title": "unknown",
  "description": "GFS data ingested for forecasting demo",
  "keywords": [],
  "extent": {
    "spatial": {
      "bbox": [
        [0, -90, 359.75, 90]
      ],
      "crs": "EPSG:4326"
    },
    "temporal": {
      "interval": [
        "2024-05-13T00:00:00",
        "2025-01-15T06:00:00"
      ],
      "values": [
        "2024-05-13T00:00:00/2025-01-15T06:00:00"
      ],
      "trs": "TIMECRS[\"DateTime\",TDATUM[\"Gregorian Calendar\"],CS[TemporalDateTime,1],AXIS[\"Time (T)\",unspecified]]"
    }
  },
  "data_queries": {
    "position": {
      "link": {
        "href": "/edr/position?coords={coords}",
        "rel": "data",
        "hreflang": "en",
        "templated": true,
        "variables": {
          "title": "Position query",
          "description": "Returns position data based on WKT `POINT(lon lat)` or `MULTIPOINT(lon lat, ...)` coordinates",
          "query_type": "position",
          "coords": {
            "type": "string",
            "description": "WKT `POINT(lon lat)` or `MULTIPOINT(lon lat, ...)` coordinates",
            "required": true
          },
          "output_formats": [
            "cf_covjson",
            "csv",
            "geojson",
            "nc",
            "nc4",
            "netcdf",
            "netcdf4"
          ],
          "default_output_format": "cf_covjson",
          "crs_details": [
            {
              "crs": "EPSG:4326",
              "wkt": "GEOGCRS[\"WGS 84\",ENSEMBLE[\"World Geodetic System 1984 ensemble\",MEMBER[\"World Geodetic System 1984 (Transit)\"],MEMBER[\"World Geodetic System 1984 (G730)\"],MEMBER[\"World Geodetic System 1984 (G873)\"],MEMBER[\"World Geodetic System 1984 (G1150)\"],MEMBER[\"World Geodetic System 1984 (G1674)\"],MEMBER[\"World Geodetic System 1984 (G1762)\"],MEMBER[\"World Geodetic System 1984 (G2139)\"],MEMBER[\"World Geodetic System 1984 (G2296)\"],ELLIPSOID[\"WGS 84\",6378137,298.257223563,LENGTHUNIT[\"metre\",1]],ENSEMBLEACCURACY[2.0]],PRIMEM[\"Greenwich\",0,ANGLEUNIT[\"degree\",0.0174532925199433]],CS[ellipsoidal,2],AXIS[\"geodetic latitude (Lat)\",north,ORDER[1],ANGLEUNIT[\"degree\",0.0174532925199433]],AXIS[\"geodetic longitude (Lon)\",east,ORDER[2],ANGLEUNIT[\"degree\",0.0174532925199433]],USAGE[SCOPE[\"Horizontal component of 3D system.\"],AREA[\"World.\"],BBOX[-90,-180,90,180]],ID[\"EPSG\",4326]]"
            }
          ]
        }
      }
    },
    "area": {
      "link": {
        "href": "/edr/area?coords={coords}",
        "rel": "data",
        "hreflang": "en",
        "templated": true,
        "variables": {
          "title": "Area query",
          "description": "Returns data in a polygon based on WKT `POLYGON(lon lat, ...)` coordinates",
          "query_type": "position",
          "coords": {
            "type": "string",
            "description": "WKT `POLYGON(lon lat, ...)` coordinates",
            "required": true
          },
          "output_formats": [
            "cf_covjson",
            "csv",
            "geojson",
            "nc",
            "nc4",
            "netcdf",
            "netcdf4"
          ],
          "default_output_format": "cf_covjson",
          "crs_details": [
            {
              "crs": "EPSG:4326",
              "wkt": "GEOGCRS[\"WGS 84\",ENSEMBLE[\"World Geodetic System 1984 ensemble\",MEMBER[\"World Geodetic System 1984 (Transit)\"],MEMBER[\"World Geodetic System 1984 (G730)\"],MEMBER[\"World Geodetic System 1984 (G873)\"],MEMBER[\"World Geodetic System 1984 (G1150)\"],MEMBER[\"World Geodetic System 1984 (G1674)\"],MEMBER[\"World Geodetic System 1984 (G1762)\"],MEMBER[\"World Geodetic System 1984 (G2139)\"],MEMBER[\"World Geodetic System 1984 (G2296)\"],ELLIPSOID[\"WGS 84\",6378137,298.257223563,LENGTHUNIT[\"metre\",1]],ENSEMBLEACCURACY[2.0]],PRIMEM[\"Greenwich\",0,ANGLEUNIT[\"degree\",0.0174532925199433]],CS[ellipsoidal,2],AXIS[\"geodetic latitude (Lat)\",north,ORDER[1],ANGLEUNIT[\"degree\",0.0174532925199433]],AXIS[\"geodetic longitude (Lon)\",east,ORDER[2],ANGLEUNIT[\"degree\",0.0174532925199433]],USAGE[SCOPE[\"Horizontal component of 3D system.\"],AREA[\"World.\"],BBOX[-90,-180,90,180]],ID[\"EPSG\",4326]]"
            }
          ]
        }
      }
    }
  },
  "crs": [
    "EPSG:4326"
  ],
  "output_formats": [
    "cf_covjson",
    "csv",
    "geojson",
    "nc",
    "nc4",
    "netcdf",
    "netcdf4"
  ],
  "parameter_names": {
    "SUNSD": {
      "type_": "Parameter",
      "label": "unknown",
      "description": "Sunshine Duration",
      "data_type": "float32",
      "unit": {
        "label": "s",
        "symbol": {
          "value": "s"
        }
      },
      "observed_property": {
        "label": "unknown",
        "description": "Sunshine Duration"
      }
    },
    "gust": {
      "type_": "Parameter",
      "label": "unknown",
      "description": "Wind speed (gust)",
      "data_type": "float32",
      "unit": {
        "label": "m s**-1",
        "symbol": {
          "value": "m s**-1"
        }
      },
      "observed_property": {
        "label": "unknown",
        "description": "Wind speed (gust)"
      }
    },
    "prate": {
      "type_": "Parameter",
      "label": "unknown",
      "description": "Precipitation rate",
      "data_type": "float32",
      "unit": {
        "label": "kg m**-2 s**-1",
        "symbol": {
          "value": "kg m**-2 s**-1"
        }
      },
      "observed_property": {
        "label": "unknown",
        "description": "Precipitation rate"
      }
    },
    "r2": {
      "type_": "Parameter",
      "label": "relative_humidity",
      "description": "2 metre relative humidity",
      "data_type": "float32",
      "unit": {
        "label": "%",
        "symbol": {
          "value": "%"
        }
      },
      "observed_property": {
        "label": "relative_humidity",
        "description": "2 metre relative humidity"
      }
    },
    "step": {
      "type_": "Parameter",
      "label": "forecast_period",
      "description": "",
      "data_type": "timedelta64[ns]",
      "unit": {
        "label": "",
        "symbol": {
          "value": ""
        }
      },
      "observed_property": {
        "label": "forecast_period",
        "description": ""
      }
    },
    "t2m": {
      "type_": "Parameter",
      "label": "air_temperature",
      "description": "2 metre temperature",
      "data_type": "float32",
      "unit": {
        "label": "K",
        "symbol": {
          "value": "K"
        }
      },
      "observed_property": {
        "label": "air_temperature",
        "description": "2 metre temperature"
      }
    },
    "tcc": {
      "type_": "Parameter",
      "label": "unknown",
      "description": "Total Cloud Cover",
      "data_type": "float32",
      "unit": {
        "label": "%",
        "symbol": {
          "value": "%"
        }
      },
      "observed_property": {
        "label": "unknown",
        "description": "Total Cloud Cover"
      }
    }
  }
}

Position Queries​

Position queries return data for the requested coordinates using the following URL structure

{base_url}/position?coords={coords}

Coordinates are encoded as well-known text (WKT). The two supported options are

  • Single points: POINT(lng lat)
  • Multiple points: MULTIPOINT((lng1 lat1),(lng2 lat2))
note

Coordinates should be properly url-encoded, i.e. POINT(254.7295 40.0150) becomes POINT(254.7295%2040.0150)

By default, the nearest-neighbor location is returned. Multilinear interpolation can be activated by including the option method=interp.

warning

Multilinear interpolation is currently not as performant as nearest neighbor. The cause for this has been identified, and a fix is in development.

Examples

Get the precipitation (prate) forecast for New York City with the initialization time June 4, 2024, 12:00:00

GET https://compute.earthmover.io/v1/services/edr/earthmover-demos/gfs/main/timeseries/edr/position?coords=POINT(286%2040.0150)&time=2024-09-04T12:00:00&parameter-name=prate
response
{
  "type": "Coverage",
  "domain": {
    "type": "Domain",
    "domainType": "Grid",
    "axes": {
      "y": {
        "values": [
          40.0
        ]
      },
      "x": {
        "values": [
          286.0
        ]
      },
      "step": {
        "values": [
          0,
          3600000000000,
          7200000000000,
          10800000000000,
          14400000000000,
          18000000000000,
          21600000000000,
          25200000000000,
          28800000000000,
          32400000000000,
          36000000000000,
          39600000000000,
          43200000000000,
          46800000000000,
          50400000000000,
          54000000000000,
          57600000000000,
          61200000000000,
          64800000000000,
          68400000000000,
          72000000000000,
          75600000000000,
          79200000000000,
          82800000000000,
          86400000000000,
          90000000000000,
          93600000000000,
          97200000000000,
          100800000000000,
          104400000000000,
          108000000000000,
          111600000000000,
          115200000000000,
          118800000000000,
          122400000000000,
          126000000000000,
          129600000000000,
          133200000000000,
          136800000000000,
          140400000000000,
          144000000000000,
          147600000000000,
          151200000000000,
          154800000000000,
          158400000000000,
          162000000000000,
          165600000000000,
          169200000000000,
          172800000000000,
          176400000000000,
          180000000000000,
          183600000000000,
          187200000000000,
          190800000000000,
          194400000000000,
          198000000000000,
          201600000000000,
          205200000000000,
          208800000000000,
          212400000000000,
          216000000000000,
          219600000000000,
          223200000000000,
          226800000000000,
          230400000000000,
          234000000000000,
          237600000000000,
          241200000000000,
          244800000000000,
          248400000000000,
          252000000000000,
          255600000000000,
          259200000000000,
          262800000000000,
          266400000000000,
          270000000000000,
          273600000000000,
          277200000000000,
          280800000000000,
          284400000000000,
          288000000000000,
          291600000000000,
          295200000000000,
          298800000000000,
          302400000000000,
          306000000000000,
          309600000000000,
          313200000000000,
          316800000000000,
          320400000000000,
          324000000000000,
          327600000000000,
          331200000000000,
          334800000000000,
          338400000000000,
          342000000000000,
          345600000000000,
          349200000000000,
          352800000000000,
          356400000000000,
          360000000000000,
          363600000000000,
          367200000000000,
          370800000000000,
          374400000000000,
          378000000000000,
          381600000000000,
          385200000000000,
          388800000000000,
          392400000000000,
          396000000000000,
          399600000000000,
          403200000000000,
          406800000000000,
          410400000000000,
          414000000000000,
          417600000000000,
          421200000000000,
          424800000000000,
          428400000000000,
          432000000000000,
          442800000000000,
          453600000000000,
          464400000000000,
          475200000000000,
          486000000000000,
          496800000000000,
          507600000000000,
          518400000000000,
          529200000000000,
          540000000000000,
          550800000000000,
          561600000000000,
          572400000000000,
          583200000000000,
          594000000000000,
          604800000000000,
          615600000000000,
          626400000000000,
          637200000000000,
          648000000000000,
          658800000000000,
          669600000000000,
          680400000000000,
          691200000000000,
          702000000000000,
          712800000000000,
          723600000000000,
          734400000000000,
          745200000000000,
          756000000000000,
          766800000000000,
          777600000000000,
          788400000000000,
          799200000000000,
          810000000000000,
          820800000000000,
          831600000000000,
          842400000000000,
          853200000000000,
          864000000000000,
          874800000000000,
          885600000000000,
          896400000000000,
          907200000000000,
          918000000000000,
          928800000000000,
          939600000000000,
          950400000000000,
          961200000000000,
          972000000000000,
          982800000000000,
          993600000000000,
          1004400000000000,
          1015200000000000,
          1026000000000000,
          1036800000000000,
          1047600000000000,
          1058400000000000,
          1069200000000000,
          1080000000000000,
          1090800000000000,
          1101600000000000,
          1112400000000000,
          1123200000000000,
          1134000000000000,
          1144800000000000,
          1155600000000000,
          1166400000000000,
          1177200000000000,
          1188000000000000,
          1198800000000000,
          1209600000000000,
          1220400000000000,
          1231200000000000,
          1242000000000000,
          1252800000000000,
          1263600000000000,
          1274400000000000,
          1285200000000000,
          1296000000000000,
          1306800000000000,
          1317600000000000,
          1328400000000000,
          1339200000000000,
          1350000000000000,
          1360800000000000,
          1371600000000000,
          1382400000000000
        ]
      },
      "t": {
        "values": [
          "2024-09-04T12:00:00"
        ]
      }
    },
    "referencing": []
  },
  "parameters": {
    "prate": {
      "type": "Parameter",
      "observedProperty": {
        "id": "unknown",
        "label": {
          "en": "Precipitation rate"
        }
      },
      "description": {
        "en": "Precipitation rate"
      },
      "unit": {
        "label": {
          "en": "kg m**-2 s**-1"
        }
      }
    }
  },
  "ranges": {
    "prate": {
      "type": "NdArray",
      "dataType": "float",
      "axisNames": [
        "step"
      ],
      "shape": [
        209
      ],
      "values": [
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0000032000000373955118,
        0.00000960000033956021,
        0.000015999999959603883,
        0.000014399999599845614,
        0.000016399999367422424,
        0.000029200000426499173,
        0.000033200001780642197,
        0.00002879999919969123,
        0.0000152000002344721,
        0.0000016000000186977559,
        0.000003999999989900971,
        0.0000024000000848900527,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.00002320000021427404,
        0.0,
        0.000007999999979801942,
        0.0000032000000373955118,
        0.0001055999964592047,
        0.000756800000090152,
        0.0007592000183649361,
        0.00012880000576842576,
        0.000039999998989515007,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        4.0000000467443897E-7,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0000064000000747910235,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.000007199999799922807,
        0.0,
        0.0
      ]
    }
  }
}

Get all forecast variables for both New York City and Los Angeles for a single timestep in GeoJSON format

GET https://compute.earthmover.io/v1/services/edr/earthmover-demos/gfs/main/timeseries/edr/position?coords=MULTIPOINT((286%2040.0150),(241.8%2034.1))&time=2024-09-04T12:00:00&step=0%20hours&f=geojson
response
{
  "type": "FeatureCollection",
  "features": [
    {
      "id": "0",
      "type": "Feature",
      "properties": {
        "pts": 0,
        "SUNSD": 5263,
        "gust": 3.6060333251953125,
        "latitude": 40,
        "longitude": 286,
        "prate": 0,
        "r2": 55.70000076293945,
        "step": 0,
        "t2m": 292.3636779785156,
        "tcc": 0,
        "time": "2024-09-04T12:00:00"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          286,
          40
        ]
      }
    },
    {
      "id": "1",
      "type": "Feature",
      "properties": {
        "pts": 1,
        "SUNSD": 0,
        "gust": 1.2060332298278809,
        "latitude": 34,
        "longitude": 241.75,
        "prate": 0,
        "r2": 63.20000076293945,
        "step": 0,
        "t2m": 296.3636779785156,
        "tcc": 0,
        "time": "2024-09-04T12:00:00"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          241.75,
          34
        ]
      }
    }
  ]
}

Area Queries​

Area queries can be used to retrieve all of the data within a polygon.

{base_url}/area?coords={coords}

The only valid type for {coords} is

  • Polygon: POLYGON((lng1 lat1),(lng2 lat2)…(lngN latN))

Example

Create a bounding box around the state of Maine and retrieve all data variables as a NetCDF file

GET https://compute.earthmover.io/v1/services/edr/earthmover-demos/gfs/main/timeseries/edr/area?coords=POLYGON((288.9%2042.9,288.9%2047.5,293.1%2047.5,293.1%2042.9,288.9%2042.9))&time=2024-09-04&step=0%20hours&f=nc

Authentication​

For now EDR endpoints support HTTP Basic Auth with username and password.

The Arraylake compute service uses your organization name as the username and an Arraylake API key as the password.

Authenticated queries can be made by embedding the credential username (org) and password (api token) into the base URl as follows

https://{username}:{password}.compute.earthmover.io/v1/services/edr/{org}/{repo}/{branch|commit|tag}/{path/to/group}/edr/

Alternatively, the Authorization header can be used to pass the credentials:

Using curl

curl -u {username}:{password} "https://compute.earthmover.io/v1/services/edr/{org}/{repo}/{branch|commit|tag}/{path/to/group}/edr/"

Using Python

import requests
response = requests.get(
    "https://compute.earthmover.io/v1/services/edr/{org}/{repo}/{branch|commit|tag}/{path/to/group}/edr/",
    auth=("{username}", "{password}")
)