Webhooks and Notifications

Arraylake supports webhooks and Slack notifications to alert you when events occur in your repositories. This allows you to integrate Arraylake with your CI/CD pipelines, monitoring systems, team communication channels, and other external tools.

info

Webhook and notification management requires admin privileges for the repository.

Overview

Arraylake can send notifications to external systems when specific events occur in your repositories. Currently supported events include:

Commit events: Triggered when a new commit is created in a repository

You can configure two types of notifications:

Generic Webhooks: HTTP POST requests to any HTTPS endpoint with customizable payloads and optional authentication
Slack Notifications: Simplified notifications sent directly to Slack channels via incoming webhooks

Generic Webhooks

Generic webhooks allow you to integrate Arraylake with any system that can receive HTTP POST requests.

Creating and Managing Webhooks

Navigate to your repository's settings page and scroll to the "Webhooks" section. Here you can:

Create: Click "Add Webhook" and configure the webhook URL, alias, optional secret token, and events
Test: Click "Test Webhook" in the creation dialog to verify your endpoint is configured correctly
View/Edit/Delete: Use the buttons next to each webhook entry to manage existing webhooks

Configuration options:

Webhook URL: The HTTPS endpoint where notifications will be sent
Alias: A friendly name to identify this webhook
Secret Token (optional): A token sent in the X-Secret-Token header for authentication
Events: Select which events should trigger this webhook (e.g., "On Commit")

Webhook Payload Format

When an event occurs, Arraylake sends a JSON payload to your webhook URL:

{
  "event": "NewCommit",
  "details": {
    "org_name": "earthmover",
    "repo_name": "ocean",
    "repo_id": "507f1f77bcf86cd799439011",
    "branch": "main",
    "snapshot_id": "NR8XYDR4CMS6GQV9MFN0",
    "parent_snapshot_id": "K2QF7B1XZTM4WVHC3P60",
    "previous_snapshot_id": "K2QF7B1XZTM4WVHC3P60",
    "commit_message": "add january",
    "written_at": "2026-09-15T12:00:00Z",
    "author_id": "alice@example.com",
    "commits": [
      {
        "snapshot_id": "NR8XYDR4CMS6GQV9MFN0",
        "commit_message": "add january",
        "written_at": "2026-09-15T12:00:00Z",
        "author_id": "alice@example.com"
      }
    ],
    "commits_unavailable_reason": null
  }
}

Arraylake sends one notification each time a branch head moves.

Payload schema

Field	Type	Description
`event`	string	`NewCommit` for a branch head move.
`details.org_name`	string	The organization that owns the repository.
`details.repo_name`	string	The repository name.
`details.repo_id`	string	The repository's Arraylake id.
`details.branch`	string	The branch whose head moved. Currently always `main`.
`details.snapshot_id`	string	The new head of `branch`.
`details.parent_snapshot_id`	string or null	The head's parent commit. Null when the head is the repository's first commit.
`details.previous_snapshot_id`	string or null	The head Arraylake last saw. Null on the first notification for this branch.
`details.commit_message`	string	The head's commit message.
`details.written_at`	string	When the head was committed, as an ISO 8601 timestamp in UTC. A reset can point a branch at an older commit, so this can be earlier than the previous notification's. Do not sort on it.
`details.author_id`	string	Who wrote the head, or `unknown` when the commit carries no author metadata.
`details.commits`	array or null	Every commit from just after `previous_snapshot_id` through the head, oldest first. Null when Arraylake cannot list the range.
`details.commits_unavailable_reason`	string or null	Why `commits` is null. Null when `commits` is present.

Each entry in details.commits has these fields:

Field	Type	Description
`snapshot_id`	string	The commit id.
`commit_message`	string	The commit message.
`written_at`	string	When the commit was written, as an ISO 8601 timestamp in UTC.
`author_id`	string	Who wrote the commit, or `unknown`.

When the commit list is absent

Reason	Meaning	What to do
`first_observation`	The first notification for this branch, so there is no earlier head to measure from.	Read the state at `snapshot_id`.
`history_rewritten`	The head no longer descends from the head Arraylake last saw, usually after a branch reset. Commits from earlier notifications may no longer be on the branch.	Discard any cached view of the branch and re-read from `snapshot_id`.
`too_many_commits`	More commits landed than one notification carries.	Walk the history yourself from `snapshot_id`.
`repo_format_v1`	The repository uses Icechunk spec version 1, where listing the range costs one request per commit.	Migrate the repository to spec version 2 to receive the list.

If you configured a secret token, it will be included in the request headers:

X-Secret-Token: your-secret-token

Authenticating Webhook Requests

The secret token is used to verify that webhook requests are actually coming from Arraylake and not a malicious third party. Arraylake simply includes the token in the request header - it's your responsibility to validate it in your webhook endpoint.

Your webhook endpoint should:

Extract the X-Secret-Token header from incoming requests
Compare it against the expected secret token you configured
Reject requests with missing or incorrect tokens

Example validation:

@app.route('/webhook', methods=['POST'])
def handle_webhook():
    expected_token = "your-secret-token"  # Store this securely
    provided_token = request.headers.get('X-Secret-Token')

    if provided_token != expected_token:
        return 'Unauthorized', 401

    # Process the webhook
    data = request.json
    # ... your logic here
    return 'OK', 200

Without proper validation, anyone could send fake webhook requests to your endpoint.

Webhook Security

Arraylake implements several security measures for webhooks:

HTTPS Only: All webhook URLs must use HTTPS
No Redirects: Webhook requests do not follow redirects
Timeouts: Requests timeout after 5 seconds
Response Limits: Response bodies are limited to 250KB

Slack Notifications

Slack notifications provide a simplified way to send repository event notifications directly to your Slack channels.

Setting Up and Managing Slack Notifications

1. Create a Slack Incoming Webhook

First, create an incoming webhook in your Slack workspace:

Go to Slack API: Incoming Webhooks
Click "Create your Slack app" or use an existing app
Enable "Incoming Webhooks"
Click "Add New Webhook to Workspace"
Select the channel where notifications should be posted
Copy the webhook URL (format: https://hooks.slack.com/services/...)

2. Configure in Arraylake

Navigate to your repository's settings page and scroll to the "Slack Notifications" section. Here you can:

Create: Click "Add Slack Notification" and configure the webhook URL, alias, and events
Test: Click "Test Notification" in the creation dialog to verify the Slack integration
View/Edit/Delete: Use the buttons next to each notification entry to manage existing configurations

Configuration options:

Webhook URL: The Slack incoming webhook URL
Alias: A friendly name to identify this notification
Events: Select which events should trigger notifications (e.g., "On Commit")

Slack Message Format

When a branch head moves, Arraylake sends a plain text message to your Slack channel:

earthmover/ocean branch main is now at NR8XYDR4CMS6GQV9MFN0: add january

The message gives the commit the branch points at now. After a branch reset that can be an older commit. Slack messages carry no commit list, so use a webhook if you need one.

Use Cases

Here are some common use cases for webhooks and notifications:

CI/CD Integration

Trigger automated builds, tests, or deployments when new commits are detected:

# Example webhook receiver that triggers a build
curl -X POST https://ci.example.com/trigger-build \
  -H "X-Secret-Token: your-secret" \
  -d '{"repo": "earthmover/ocean", "event": "NewCommit"}'

Data Pipeline Automation

Start ETL jobs or data processing workflows when repository data changes:

# Example webhook handler in Python
from flask import Flask, request

app = Flask(__name__)

@app.route('/webhook', methods=['POST'])
def handle_webhook():
    data = request.json
    if data['event'] == 'NewCommit':
        # Trigger your data pipeline
        start_etl_job(data['details']['repo_name'])
    return 'OK', 200

Team Notifications

Keep your team informed about repository changes via Slack:

Development teams can track when datasets are updated
Data scientists can be notified when new training data is available
Operations teams can monitor data pipeline activities

Audit Logging

Send commit events to security or compliance systems for audit trails:

# Example: Forward to a logging service
curl -X POST https://logging.example.com/audit \
  -H "X-Secret-Token: audit-token" \
  -d '{"timestamp": "2025-01-15T10:00:00Z", "event": "NewCommit", "repo": "earthmover/ocean"}'

Data Quality Checks

Trigger automated validation when new data is committed:

# Example: Validate data quality on new commits
@app.route('/validate', methods=['POST'])
def validate_data():
    data = request.json
    repo_name = data['details']['repo_name']

    # Run validation checks
    results = run_data_quality_checks(repo_name)

    if not results.passed:
        notify_team(f"Data quality issues in {repo_name}")

    return 'OK', 200

Limitations and Considerations

Webhooks must use HTTPS endpoints (HTTP is not supported)
Webhook requests timeout after 5 seconds
Response bodies are limited to 250KB
Currently, only commit events are supported
Each repository can have multiple webhook and Slack notification configurations
Notification aliases must be unique within a repository

Overview​

Generic Webhooks​

Creating and Managing Webhooks​

Webhook Payload Format​

Payload schema​

When the commit list is absent​

Authenticating Webhook Requests​

Webhook Security​

Slack Notifications​

Setting Up and Managing Slack Notifications​

Slack Message Format​

Use Cases​

CI/CD Integration​

Data Pipeline Automation​

Team Notifications​

Audit Logging​

Data Quality Checks​

Limitations and Considerations​