Skip to main content

Webhooks and Notifications

Arraylake supports webhooks and Slack notifications to alert you when events occur in your repositories. This allows you to integrate Arraylake with your CI/CD pipelines, monitoring systems, team communication channels, and other external tools.

info

Webhook and notification management requires admin privileges for the repository.

Overview​

Arraylake can send notifications to external systems when specific events occur in your repositories. Currently supported events include:

  • Commit events: Triggered when a new commit is created in a repository

You can configure two types of notifications:

  1. Generic Webhooks: HTTP POST requests to any HTTPS endpoint with customizable payloads and optional authentication
  2. Slack Notifications: Simplified notifications sent directly to Slack channels via incoming webhooks

Generic Webhooks​

Generic webhooks allow you to integrate Arraylake with any system that can receive HTTP POST requests.

Creating and Managing Webhooks​

Navigate to your repository's settings page and scroll to the "Webhooks" section. Here you can:

  • Create: Click "Add Webhook" and configure the webhook URL, alias, optional secret token, and events
  • Test: Click "Test Webhook" in the creation dialog to verify your endpoint is configured correctly
  • View/Edit/Delete: Use the buttons next to each webhook entry to manage existing webhooks

Configuration options:

  • Webhook URL: The HTTPS endpoint where notifications will be sent
  • Alias: A friendly name to identify this webhook
  • Secret Token (optional): A token sent in the X-Secret-Token header for authentication
  • Events: Select which events should trigger this webhook (e.g., "On Commit")

Webhook Payload Format​

When an event occurs, Arraylake sends a JSON payload to your webhook URL:

{
  "event": "NewCommit",
  "details": {
    "repo_name": "ocean",
    "repo_id": "507f1f77bcf86cd799439011"
  }
}

If you configured a secret token, it will be included in the request headers:

X-Secret-Token: your-secret-token

Authenticating Webhook Requests​

The secret token is used to verify that webhook requests are actually coming from Arraylake and not a malicious third party. Arraylake simply includes the token in the request header - it's your responsibility to validate it in your webhook endpoint.

Your webhook endpoint should:

  1. Extract the X-Secret-Token header from incoming requests
  2. Compare it against the expected secret token you configured
  3. Reject requests with missing or incorrect tokens

Example validation:

@app.route('/webhook', methods=['POST'])
def handle_webhook():
    expected_token = "your-secret-token"  # Store this securely
    provided_token = request.headers.get('X-Secret-Token')

    if provided_token != expected_token:
        return 'Unauthorized', 401

    # Process the webhook
    data = request.json
    # ... your logic here
    return 'OK', 200

Without proper validation, anyone could send fake webhook requests to your endpoint.

Webhook Security​

Arraylake implements several security measures for webhooks:

  • HTTPS Only: All webhook URLs must use HTTPS
  • No Redirects: Webhook requests do not follow redirects
  • Timeouts: Requests timeout after 5 seconds
  • Response Limits: Response bodies are limited to 250KB

Slack Notifications​

Slack notifications provide a simplified way to send repository event notifications directly to your Slack channels.

Setting Up and Managing Slack Notifications​

1. Create a Slack Incoming Webhook

First, create an incoming webhook in your Slack workspace:

  1. Go to Slack API: Incoming Webhooks
  2. Click "Create your Slack app" or use an existing app
  3. Enable "Incoming Webhooks"
  4. Click "Add New Webhook to Workspace"
  5. Select the channel where notifications should be posted
  6. Copy the webhook URL (format: https://hooks.slack.com/services/...)

2. Configure in Arraylake

Navigate to your repository's settings page and scroll to the "Slack Notifications" section. Here you can:

  • Create: Click "Add Slack Notification" and configure the webhook URL, alias, and events
  • Test: Click "Test Notification" in the creation dialog to verify the Slack integration
  • View/Edit/Delete: Use the buttons next to each notification entry to manage existing configurations

Configuration options:

  • Webhook URL: The Slack incoming webhook URL
  • Alias: A friendly name to identify this notification
  • Events: Select which events should trigger notifications (e.g., "On Commit")

Slack Message Format​

When a commit event occurs, Arraylake sends a plain text message to your Slack channel:

New commit abc123 in repository earthmover/ocean on branch main.

Use Cases​

Here are some common use cases for webhooks and notifications:

CI/CD Integration​

Trigger automated builds, tests, or deployments when new commits are detected:

# Example webhook receiver that triggers a build
curl -X POST https://ci.example.com/trigger-build \
  -H "X-Secret-Token: your-secret" \
  -d '{"repo": "earthmover/ocean", "event": "NewCommit"}'

Data Pipeline Automation​

Start ETL jobs or data processing workflows when repository data changes:

# Example webhook handler in Python
from flask import Flask, request

app = Flask(__name__)

@app.route('/webhook', methods=['POST'])
def handle_webhook():
    data = request.json
    if data['event'] == 'NewCommit':
        # Trigger your data pipeline
        start_etl_job(data['details']['repo_name'])
    return 'OK', 200

Team Notifications​

Keep your team informed about repository changes via Slack:

  • Development teams can track when datasets are updated
  • Data scientists can be notified when new training data is available
  • Operations teams can monitor data pipeline activities

Audit Logging​

Send commit events to security or compliance systems for audit trails:

# Example: Forward to a logging service
curl -X POST https://logging.example.com/audit \
  -H "X-Secret-Token: audit-token" \
  -d '{"timestamp": "2025-01-15T10:00:00Z", "event": "NewCommit", "repo": "earthmover/ocean"}'

Data Quality Checks​

Trigger automated validation when new data is committed:

# Example: Validate data quality on new commits
@app.route('/validate', methods=['POST'])
def validate_data():
    data = request.json
    repo_name = data['details']['repo_name']

    # Run validation checks
    results = run_data_quality_checks(repo_name)

    if not results.passed:
        notify_team(f"Data quality issues in {repo_name}")

    return 'OK', 200

Limitations and Considerations​

  • Webhooks must use HTTPS endpoints (HTTP is not supported)
  • Webhook requests timeout after 5 seconds
  • Response bodies are limited to 250KB
  • Currently, only commit events are supported
  • Each repository can have multiple webhook and Slack notification configurations
  • Notification aliases must be unique within a repository