Managing Repos

Once your Arraylake organization has been fully configured and you have installed the client library, you're ready to start managing data! 🎉

Create a Repo

For the purposes of this example, our org name will be earthmover. If running these commands interactively, replace earthmover with your org name.

For this example, we are going to create a Repo called ocean to hold oceanography data 🌊.

Web App

Navigate to your org's repositories page (for the org called earthmover this is https://app.earthmover.io/earthmover/repositories), then click New Repository.

The new repository button.

Type in the desired Repo name and hit Create.

The create repository dialog.

CLI

arraylake repo create earthmover/ocean

Python

from arraylake import Client
client = Client()
client.create_repo("earthmover/ocean")

Python (asyncio)

from arraylake import AsyncClient
aclient = AsyncClient()
await aclient.create_repo("earthmover/ocean")

You can optionally add a repo description and/or metadata to your repo. Descriptions are limited to 255 characters, while repo metadata can be at most 4kB total. Metadata must be a mapping of key-value pairs where values are strings, numbers, lists, booleans, or None.

CLI

arraylake repo create earthmover/ocean --description "This is some oceanography data!" --metadata '{"type": ["climate", "coastal", "environmental"], "model": "FVCOM"}'

Python

from arraylake import Client
client = Client()
client.create_repo(
    "earthmover/ocean",
    description="This is some oceanography data!",
    metadata={"type": ["climate", "coastal", "environmental"], "model": "FVCOM"}
)

Python (asyncio)

from arraylake import AsyncClient
aclient = AsyncClient()
await aclient.create_repo(
    "earthmover/ocean",
    description="This is some oceanography data!",
    metadata={"type": ["climate", "coastal", "environmental"], "model": "FVCOM"}
)

You can also modify the description and/or metadata of an existing repo.

CLI

arraylake repo modify earthmover/ocean --description "This is an updated description for some oceanography data!" -a '{"source": "NOAA"}' -r "type" -u '{"model": "ROMS"}'

Python

from arraylake import Client
client = Client()
client.modify_repo(
    "earthmover/ocean",
    description="This is an updated description for some oceanography data!",
    add_metadata={"source": "NOAA"},
    remove_metadata=["type"],
    update_metadata={"model": "ROMS"}
)

Python (asyncio)

from arraylake import AsyncClient
aclient = AsyncClient()
await aclient.modify_repo(
    "earthmover/ocean",
    description="This is an updated description for some oceanography data!",
    add_metadata={"source": "NOAA"},
    remove_metadata=["type"],
    update_metadata={"model": "ROMS"}
)

Where is Repo data stored?

Arraylake lets you configure the storage location for a Repo's data using org-level bucket configurations. Choose a specific bucket by providing bucket_config_nickname to create_repo. If not specified, the organization's default bucket is used.

Within the bucket and prefix set by a org-level bucket configuration, data for new Icechunk Repos are stored within another prefix. By default, the extra prefix is set to Repo name prefixed with 8 random characters. Choose a specific extra prefix by passing the prefix kwarg to create_repo. For example, for a bucket configured with bucket='my-bucket-name' and prefix='my-bucket-prefix',

1. create_repo("repo-A") stores data in my-bucket-name/my-bucket-prefix/[8-RANDOM_CHARACTERS]_repo_A
2. create_repo("repo-B", prefix='zoo') stores data in my-bucket-name/my-bucket-prefix/zoo/

Import an existing Icechunk Repo

You can also import an existing Icechunk Repository into Arraylake.

You'll need a bucket configuration for the bucket in which your Icechunk Repository is stored. Let's imagine you created a bucket config with nickname my-bucket, and your bucket contains an Icechunk Repository under the compound prefix my-bucket_prefix/my-icechunk-prefix.

Web App

Hit the New Repository button as before, and then open the Additional Settings box. Check the Import Existing Repository button, and type in the prefix under which your Icechunk repo is stored.

Importing an existing Icechunk repository in the web app.

CLI

arraylake repo import earthmover/ocean --bucket-config-nickname my-bucket --prefix my_icechunk_prefix

Python

from arraylake import Client
client = Client()
client.import_repo(
    "earthmover/ocean",
    bucket_config_nickname="my_bucket",
    prefix="my-icechunk-prefix",
)

Python (asyncio)

from arraylake import AsyncClient
aclient = AsyncClient()
await aclient.import_repo(
    "earthmover/ocean",
    bucket_config_nickname="my_bucket",
    prefix="my-icechunk-prefix",
)

Repo metadata and descriptions can be set in the same way as when creating a repo from scratch.

If your existing Icechunk Repository contains virtual chunks, this requires an additional configuation step - see the docs on virtual chunks.

Open a Repo

If you're working in Python, you can open a Repo and start interacting with your data.

Python

repo = client.get_repo("earthmover/ocean")

Python (asyncio)

arepo = await aclient.get_repo("earthmover/ocean")  # returns an AsyncRepo object

List Repos

You can list repos associated with an organization.

Web App

To browse repositories, navigate to your org's repositories page.

Viewing a list of your org's repositories in the web app.

CLI

arraylake repo list earthmover

Python

client.list_repos("earthmover")

Python (asyncio)

await aclient.list_repos("earthmover")

You can also filter repos on repo metadata.

Web App

In the web app, you can use the search bar in your org's Repositories page.

Searching the list of your org's repositories in the web app.

CLI

Metadata filters must be a mapping of key-value pairs where values are strings, numbers, lists, booleans, or None. Filtering is inclusive and will return repos that match all of the provided metadata.

arraylake repo list earthmover --filter-metadata '{"source": "NOAA"}'

Python

Metadata filters must be a mapping of key-value pairs where values are strings, numbers, lists, booleans, or None. Filtering is inclusive and will return repos that match all of the provided metadata.

client.list_repos("earthmover", filter_metadata={"source": "NOAA"})

Python (asyncio)

Metadata filters must be a mapping of key-value pairs where values are strings, numbers, lists, booleans, or None. Filtering is inclusive and will return repos that match all of the provided metadata.

await aclient.list_repos("earthmover", filter_metadata={"source": "NOAA"})

Delete a Repo

Finally, we can delete a repo.

warning

Deleting a repo cannot be undone! Use this operation carefully.

Web App

Navigate to the Settings tab of your Repo:

The settings tab of a repo in the web app.

Scroll to the bottom, and hit Delete Repository.

Deleting a Repostory via the web app.

CLI

arraylake repo delete earthmover/ocean

Python

client.delete_repo("earthmover/ocean", imsure=True, imreallysure=True)

Python (asyncio)

await aclient.delete_repo("earthmover/ocean", imsure=True, imreallysure=True)