Changelog
This page includes a running list of user-facing changes to the Arraylake Client and Service.
0.12.0 - September 25, 2024
- [Web, Service, Client] - Support for role-based access delegation ("customer-managed role") for AWS S3 object stores
- [Web] - Renamed "Anonymous" credentials to "Self-Managed" credentials in the Web UI
- [Client] - Removed deprecated
bucket_nickname
(nowbucket_config_nickname
) argument
0.11.0 - September 6, 2024
- [Client] - Performance improvements when fetching large or many small chunks from storage
- [Web, CLI] - Allow org admins to set repo status to maintenance or offline mode
- [Client] - Added an advanced configuration option (
chunkstore.unsafe_use_fill_value_for_missing_chunks
) to make chunkstore more fault tolerant
0.10.0 - July 12, 2024
- [Client] - Added a new repository export tool
0.9.10 - July 05, 2024
- [Client] - Added the ability to login to Arraylake via python library.
- [Client] - Added a
read_only
flag for getting repositories withget_repo
. See client api docs.
0.9.9 - June 26, 2024
- [Client] - Added the ability to export Arraylake repositories to standard Zarr stores via the
arraylake repo export
command line utility.
0.9.8 - May 27, 2024
- [Client] - Added the Repo method
add_kerchunk_references
to support importing existing Kerchunk reference data into Arraylake repos.
0.9.7 - May 1, 2024
- [Client, Service] - Improve validation of new bucket configurations
- [Client] - Fix duplicate FutureWarning in
get_or_create_repo
- [Client, Service] - Disallow deleting the main branch of a repo
0.9.6 - Apr 26, 2024
- [Client, Service, Web] - Arraylake now supports user managed bucket configurations
- [Client, Service] - Arraylake now supports version tags
- [Client] - Improve performance of
add_virtual_netcdf
- [Client] - Improve performance of
repo.checkout
- [Client] - Drop support for Python 3.9, Arraylake now supports Python 3.10, 3.11 and 3.12. See the installation page for a current list of supported dependency versions.
- [Documentation] - Add documentation for seven storage integrations.
0.9.5 - Mar 25, 2024
- [Client] Improve performance of
add_virtual_zarr
- [Client] Add support for Python 3.12
0.9.4 - Mar 12, 2024
- [Client] Fixed incompatibility with
aiobotocore>=2.12.1
. - [Client] Retry HTTP connections when spurious network errors are encountered.
- [Web] Improved documentation for how to authenticate to Arraylake using a service account API token.
- [Web] Added new documentation on how to use Arraylake with Google Colab.
0.9.3 - Mar 08, 2024
- [Client] Improve client-side caching
0.9.2 - Mar 06, 2024
- [Client] Support virtual files on Google Cloud Storage
- [Client] Performance improvements around fetching a large number of commits.
- [Web] Add documentation for configuring Google Cloud Storage buckets.
0.9.1 - Feb 15, 2024
- [Client] Improve stability of HTTP connections.
- [Client, Service] Allow listing active sessions
0.9.0 - Feb 2, 2024
- [Client] Support for Google Cloud Storage buckets.
- [Client] Significant read performance improvements when fetching multiple Zarr chunks at once.
- [Client] Server-managed sessions on by default.
0.8.1 - Jan 19, 2024
- [Client] Retry on encountering read/write network errors.
- [Client] Documentation for server managed sessions.
0.8.0 - Jan 15, 2024
- [Client] Upgrade to Pydantic 2
- [Client] Drop Python 3.8, Arraylake now supports Python 3.9, 3.10, and 3.11.
0.7.8 - Jan 11, 2024
- [Client, Service] Implement server-managed session support
0.7.7 - Dec 18, 2023
- [Client, Service] Major performance improvements in many aspects.
- [Client]
repo.to_xarray
prefetches necessary data early, and is now significantly faster than the equivalentxarray.open_zarr
call. - [Client] Pin ecmwflibs to 0.5 to work around upstream packaging issues.
v0.7.6 - Nov 30, 2023
- [Web] Search in docs.
- [Web] Major update to Concepts section.
- [Service, Client] Large performance improvement for
repo.checkout
v0.7.5 - Nov 22, 2023
- [Client] Improved error handling for service exceptions.
v0.7.4 - Nov 13, 2023
- [Client] Fixed issue where some inline references were rejected by the Arraylake service.
- [Client] Fixed issue where
Repo.checkout('main')
failed when there are only commits on other branches.
v0.7.3 - Nov 10, 2023
- [Client] Add support for ingesting virtual TIFF and GRIB datasets with
add_virtual_tiff
andadd_virtual_grib
. - [Client] Change of
add_virtual_netcdf
andadd_virtual_zarr
to match thecp src dest
syntax. - [Client] Improve type annotations and distributed arraylake with the
py.typed
module. - [Client] Move
filter_metadata
out of the experimental module. - [Client] Mark
xarray
as an optional dependency. - [Client] Arraylake now requires the CLI dependencies (
rich
andtyper
). - [Service] Alert clients if they are using an outdated version of the arraylake client.
- [Client, Service] Support inlining small chunks in the Arraylake metastore.
v0.7.2 - Oct 27, 2023
- [Client] The client package was changed from
arraylake_client
toarraylake
. A consequence of this change is that Arraylake's default config directory has moved from~/.config/arraylake_client
to~/.config/arraylake
. Importingarraylake
will attempt to migrate the exiting config files. - [Client]
Client.get_repo
andClient.create_repo
now automatically checks out the head of themain
branch. CallingClient.checkout()
is no longer required after these methods. - [Client] Fixed issue where
arraylake auth login
would fail if refresh tokens were expired. - [Service] Fixed issue where some user names were shown as
None, None
. - [Web] Fixed issue where web app used incorrect refresh tokens.
v0.7.1 - Oct 20, 2023
- [Service, Client]
zarr.Group.move
is now a metadata-only change - [Service, Client] Add
--diagnostics
flag to CLI, log diagnostics to service when logging in - [Client] Add
aioitertools
as a required dependency - [Client] Improve error messages when
chunkstore.uri
is not configured correctly - [Client] Arraylake CLI is now also distributed under the
al
command - [Client] Improved CLI login experience with new log and clickable authorization URL.
v0.7.0 - Oct 13, 2023
- [Client] Disallow dependency on numcodecs version 0.12.0
- [Client] Use uvloop library when available
- [Client] Add
s3.anon
boolean configuration value, to support anonymous access to chunkstore - [Client]
list_repos
now returnsSequence[Repo]
v0.6.0 - Sept 22, 2023
- [Client] Fix memory leak when using Arraylake with Dask Distributed
- [Service, Client] - Improve performance of
repo.store.getsize()
- [Client] Remove
anyio
as a dependency - [Client]
Repo.close()
is no longer required and will now emit a deprecation warning - [Client] Improve type annotations and documentation for
Repo.to_xarray()
v0.5.0 - Sept 6, 2023
- [Client] Remove
PyMongo
as a dependency - [Client] Remove
types-aiobotocore
as a dependency - [Client] Remove
uvloop
as a required dependency - [Client] Improve type annotations for
Repo.checkout()
method - [Client] Added support for Zarr Version 2.16
- [Service, Client] Integrated the experimental metadata search functionality into the
Repo.tree()
method - [Client] Added new
Repo.to_xarray()
method to simply reading Xarray datasets from Arraylake - [Client] Support initializing client M2M tokens from environment variable (
ARRAYLAKE_CLIENT_TOKEN
) orarraylake_client.config
v0.4.7 - Aug 25, 2023
- [Client] Improved retry logic when experiencing service rate limiting
- [Client] Publish the Arraylake-client package on Conda-Forge
v0.4.6 - Aug 18, 2023
- [Service, Client] Introduced experimental metadata search functionality accessible via the arraylake client
- [Service] Improved login flow web experience
v0.4.5 - July 31, 2023
- [Service, Client] Improved error messaging for cases when a requested commit is not found in the target ref's history
- [Service, Client] Addressed a bug that caused
tree
generation to fail when the array compressor configuration wasnull
v0.4.4 - July 18, 2023
- [Client] Addressed a bug that caused failures when multiple writers attempted to commit at the same time
- [Client] Better client side validation for M2M tokens
v0.4.3 - July 10, 2023
- [Client] Update user auth handling for SSO integrations. For customers using an SSO integration, please see this section of our documentation for additional configuration that should be set to support this update.
- [Client] Additional improvements for client memory management
v0.4.2 - July 7, 2023
- [Client] Add
Client.get_or_create_repo
method - [Client] Support implicit groups when navigating an Arraylake store
- [Client] Improvements for client memory management
v0.4.1 - June 15, 2023
- [Client] - Added support for virtual Zarr V2 datasets
- [Service, Client] - Performance and reliability improvements for queries that include many docs / commits
v0.4.0 - June 12, 2023
- [Client] Improve serialization performance for
Repo.get_docs
- [Client, Service] Fix for complex data types in Zarr arrays
v0.3.8 - May 31, 2023
- [Client] Richer debug logging support
- [Client] Improved version spec for broader installation support
v0.3.7 - May 22, 2023
- [Service, Client] Fix bug in store prefix listing
v0.3.6 - May 16, 2023
- [Service, Client] - Performance and stability improvements for repos with a large number of commits
v0.3.4 - May 8, 2023
- [Client] - Support Ipytree in
repo.tree()
- [Client] - Performance and stability improvements
v0.3.3 - May 5, 2023
- [Client] - Delete support
- [Client, Service] - Efficient
tree
, accessible viarepo.tree()
andarraylake repo tree
v0.3.2 - May 2, 2023
- [Client] - Fix token refresh bug
- [Client, Service] - Implement getsize, available as
repo.store.getsize("some-prefix")
v0.3.1 - Apr 27, 2023
- [Client] Support limiting response size of
.status(limit=N)
..status
changed from property to method.
v0.3.0 - Apr 25, 2023
- [Client] Fix for malformed OAuth tokens after login
- [Client] Fix for committing large number of changes
v0.2.3 - Apr 21, 2023
- [Client] Adapt Author information to support service accounts
v0.2.2 - Apr 12, 2023
- [Client, Service] - Added support for Machine-to-machine access tokens
v0.2.1 - Apr 12, 2023
- [Client] Fix for intermittent
httpx.RemoteProtocolError
v0.2.0 - Apr 11, 2023
- [Client] - Added chunk key cache to avoid repeated writes of the same chunk key.
- [Client] - Fix for missing retries on POST requests
- [Client] - Automatically refresh expired OAuth credentials
- [Client] - Added support for virtual NetCDF3 datasets
v0.1.0 - Apr 7, 2023
- [Client] Support Zarr arrays without compression
- [Client] Added JSON output to some CLI commands
- [Client] Improved formatting of
arraylake repo list
when no repos are found - [Client] Fixed duplicate path in repo status
v0.0.10 - Mar 24, 2023
- [Client] Set username/email from authenticated user
- [Client] Emit warning if
ZARR_V3_EXPERIMENTAL_API
is not set - [Client] Prevent writes when not on branch tip
- [Client] Increase HTTP timeout in client
- [Client] Raise error if
chunkstore_uri
is None
v0.0.9 - Mar 10, 2023
- [Client] Fixed Python string/repr formatting for
MetastoreDatabase
classes - [Client] Support bulk get_docs and add_docs
v0.0.8 - Feb 27, 2023
- [Service] Added support for org-wide access control
- [Client] Added backoff and retries for failed requests to service
- [Client] Fixed listing directories with trailing slashes
- [Client] Improved styling and documentation CLI
v0.0.5 - Feb 14, 2023
- [Client] Refactor Arraylake client Python API
v0.0.4 - Feb 10, 2023
- [Documentation] - Added user guide tutorials from Jupyter notebooks
v0.0.3 - Feb 2, 2023
- [Client] Updated S3 endpoint configuration
- [Service] Added health check to Arraylake service
- [Client] Added documentation to CLI commands
- [Documentation] Added Arraylake overview documentation