Observability Showcase

Category: Engineering & DevOps · 🔒 Self-hosted only

This page is generated from the Air Pipe marketplace. Browse it live to install into your organization.

Self-hosted pack — deploy this config and three things appear immediately with zero extra work:

/metrics — Prometheus scrape endpoint, auto-populated with HTTP request metrics for every route
/documentation — Swagger UI, auto-generated from the assertions and metadata in the config
Distributed tracing — set AIRPIPE_OTEL_ENDPOINT and AirPipe emits OTLP spans to any compatible collector (Jaeger, Grafana Tempo, Honeycomb, Datadog, etc.). Every request becomes a trace you can inspect end-to-end: method, route, status code, latency, the actions that ran and the outbound calls they made (engine 3.11.2 or later).

No annotations beyond the config. No code. No separate instrumentation setup.

What's included

File	Purpose
`config.yml`	Sample API with `docs: true` and full OpenAPI metadata
`schema.sql`	Users table with indexes
`grafana-dashboard.json`	Pre-built Grafana dashboard — import and go

Auto-docs: how it works

Adding docs: true to the config root enables documentation for every interface in that config. The doc system reads:

Config field	OpenAPI output
`summary`	Operation title
`description`	Operation description
`tags`	Grouping in the Swagger UI
`notes`	Appended to description
`request_example`	Example request body in the UI
`response_example`	Example response in the UI
Assertions on `input: a\|body`	Request body schema (field types, required, constraints)
Assertions on `input: a\|headers`	Header parameters
Assertions on `input: a\|params`	Query parameters

The Swagger UI is served at /documentation. The raw OpenAPI spec is at /docs.json.

Prometheus metrics: how it works

Every self-hosted AirPipe instance exposes a Prometheus /metrics endpoint. As of engine 1.7.2 it is served on a dedicated bind, default 127.0.0.1:9090 (loopback) — not the public API port — so metrics aren't exposed alongside your API by default. Configure it with --metrics-address (env AIRPIPE__METRICS_BIND): 0.0.0.0:9090 to scrape from another host (firewall the port), or off to disable the endpoint (and drop the per-request instrumentation). (On engines before 1.7.2, /metrics was served on the API port.) There are two layers:

Built-in HTTP metrics

Metric	Type	Description
`airpipe_interface_requests_total`	Counter	Requests per route, labelled `interface` (e.g. `api/users/create`)
`airpipe_interface_duration_ms`	Summary	Latency per route: quantiles over about the last minute, plus `_sum` and `_count`
`airpipe_http_action_duration_ms`	Summary	Latency per outbound `http` action, with `action` and `status`
`axum_http_requests_total`	Counter	All requests, by `method`, `status` and `endpoint`
`axum_http_requests_duration_seconds`	Histogram	Latency of all requests
`axum_http_requests_pending`	Gauge	In-flight requests

The airpipe_* series need expose_metrics: true at the config root, which this pack sets. They are the per-route view: one series per interface, labelled with the interface name from the config, so IDs in a path never multiply series.

The axum_* series see every config route as a single endpoint, /{*catchall}. Use them for totals, status codes and overall latency.

Custom business metrics via `emit_metric`

Any action can be replaced with an emit_metric action to emit a custom Prometheus metric at a specific point in the pipeline:

- name: TrackSignup
  run_when_succeeded: [CreateUser]
  hide_data_on_success: true
  emit_metric:
    name: app_users_created_total  # Prometheus metric name
    type: counter                  # counter | gauge | histogram
    labels:
      source: api                  # custom labels (values support interpolation)

Three metric types are supported:

Type	Use for	Value field
`counter`	counts of events (signups, errors, purchases)	optional — defaults to increment by 1
`gauge`	current state (active users, queue depth)	required — set to this value
`histogram`	distributions (response sizes, scores)	required — record this observation

Every emit_metric action automatically attaches config, interface, and action labels alongside any custom ones you define. Label values support the full interpolation syntax — a|ActionName::field| to label a metric with a value from the pipeline.

This config emits four custom metrics:

Metric	Type	Where
`app_users_total`	Gauge	`api/health` — live user count sampled on each health check
`app_users_listed_total`	Counter	`api/users` — increments on every list request
`app_users_created_total`	Counter	`api/users/create` — increments on successful signup
`app_users_looked_up_total`	Counter	`api/users/get` — increments on successful lookup
`app_errors_simulated_total`	Counter	`api/simulate/error` — for testing alerting pipelines

Setup

1. Set the environment variable

export DATABASE_URL=postgresql://user:pass@host:5432/dbname

2. Create the table

Either run the schema directly:

psql $DATABASE_URL -f schema.sql

Or use the seed endpoint after starting AirPipe — it creates the table and loads sample users in one step:

curl -X POST http://localhost:44111/api/seed

3. Run AirPipe

airpipe server --config config.yml --address 0.0.0.0 --port 44111
# /metrics is served separately on 127.0.0.1:9090 by default (engine >= 1.7.2).
# To let Prometheus scrape it from another host: --metrics-address 0.0.0.0:9090 (then firewall it).

4. Open the Swagger UI

http://localhost:44111/documentation

You should see a fully documented API with request/response examples and field schemas derived from the config's assertions.

5. Configure Prometheus

Add a scrape job to prometheus.yml:

scrape_configs:
  - job_name: airpipe
    static_configs:
      # engine >= 1.7.2: the dedicated metrics bind (default 127.0.0.1:9090), NOT the API port.
      # If Prometheus runs on another host, start AirPipe with --metrics-address 0.0.0.0:9090.
      - targets: ['localhost:9090']
    metrics_path: /metrics
    scrape_interval: 10s

6. Import the Grafana dashboard

Open Grafana → Dashboards → Import
Upload grafana-dashboard.json
Select your Prometheus datasource when prompted

Generating test traffic

BASE=http://localhost:44111

# Successful reads
for i in $(seq 1 20); do curl -s $BASE/api/users > /dev/null; done

# Create some users
for i in $(seq 1 5); do
  curl -s -X POST $BASE/api/users/create \
    -H "Content-Type: application/json" \
    -d "{\"email\": \"user$i@example.com\", \"username\": \"user$i\", \"password\": \"Secret$i!\"}" > /dev/null
done

# Trigger 500s to populate the error-rate panel
for i in $(seq 1 5); do
  curl -s -X POST $BASE/api/simulate/error > /dev/null
done

Grafana dashboard panels

Panel	What it shows
Request Rate	Requests/sec by route
Error Rate	4xx + 5xx rate by status code
Latency Percentiles	p50 / p95 / p99 in ms, all routes
In-Flight Requests	Current concurrent requests
Total Requests	Cumulative counter
Request Volume by Route	Relative route popularity
Status Code Distribution	2xx vs 4xx vs 5xx
p95 Latency Table	Slowest routes ranked (p95 over the last minute)

OpenTelemetry tracing

Air Pipe emits OTLP spans automatically over HTTP Binary (not gRPC), from engine 3.11.2. Each request's root span is named <METHOD> <route> and carries http.method, http.route, http.response.status_code and server.address. Under it are the interface span, one span per action, and an outbound HTTP span per http call, named by its method. Spans never include a request path or an outbound URL, since either can hold secrets. See OpenTelemetry tracing for the full list.

Enable tracing by setting AIRPIPE_OTEL_ENDPOINT before starting AirPipe:

# the collector's traces URL, including /v1/traces: Air Pipe posts to it exactly as given
export AIRPIPE_OTEL_ENDPOINT=http://localhost:4318/v1/traces
airpipe server --config config.yml --address 0.0.0.0 --port 44111

Additional env vars:

Variable	Purpose	Default
`AIRPIPE_OTEL_ENDPOINT`	OTLP HTTP traces URL, e.g. `http://localhost:4318/v1/traces`	— (tracing disabled if unset)
`AIRPIPE_OTEL_HDR_<name>`	Extra request headers (e.g. `AIRPIPE_OTEL_HDR_AUTHORIZATION`)	—
`AIRPIPE__OTEL_SAMPLE_RATIO`	Sampling ratio 0.0–1.0	1.0
`AIRPIPE__OTEL_EXPORT_TIMEOUT_SECS`	Export timeout in seconds	30

To spin up a local collector (Jaeger UI at http://localhost:16686):

# Port 4318 = OTLP HTTP (required). Port 4317 = OTLP gRPC (not used by AirPipe).
docker run -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one

Notes

/metrics and tracing are available in self-hosted mode only.
/documentation is available in both modes. In managed mode it is served at /{org}/{env}/docs.
The description field on test assertions feeds directly into the Swagger UI field descriptions — write it as if it were API documentation.

Configuration

config.yml

name: ObservabilityShowcase
description: >
  A sample REST API that demonstrates Air Pipe's built-in observability.
  Deploy this config and you get Prometheus metrics at /metrics,
  a Swagger UI at /documentation, and custom business metrics emitted
  via emit_metric actions — no extra instrumentation required.

docs: true

# Per-route request counts and latency (airpipe_interface_* in /metrics). The
# built-in axum_* series count every config route as one endpoint, so the
# dashboard's per-route panels read these.
expose_metrics: true

# Self-hosted pack — uses a|env::NAME| for configuration.
# Run: airpipe server --config config.yml --address 0.0.0.0 --port 44111
#
# Optional OpenTelemetry tracing (OTLP HTTP, engine 3.11.2 or later):
#   AIRPIPE_OTEL_ENDPOINT        — the collector's traces URL, e.g. http://localhost:4318/v1/traces
#   AIRPIPE_OTEL_HDR_<name>      — extra headers (e.g. AIRPIPE_OTEL_HDR_AUTHORIZATION)
#   AIRPIPE__OTEL_SAMPLE_RATIO   — sampling ratio 0.0–1.0 (default: 1.0)
#   AIRPIPE__OTEL_EXPORT_TIMEOUT_SECS — export timeout in seconds (default: 30)

global:
  databases:
    main:
      driver: postgres
      conn_string: "a|env::DATABASE_URL|"

interfaces:

  # ── Seed ──────────────────────────────────────────────────────────────────

  # POST /api/seed
  # Creates the users table and indexes (idempotent) then loads sample users.
  # Safe to re-run — truncates before inserting.
  api/seed:
    output: http
    method: POST

    actions:
      - name: CreateUsersTable
        database: main
        hide_data_on_success: true
        query: |
          CREATE TABLE IF NOT EXISTS users (
            id            BIGSERIAL    PRIMARY KEY,
            email         TEXT         NOT NULL UNIQUE,
            username      TEXT         NOT NULL UNIQUE,
            password_hash TEXT         NOT NULL,
            created_at    TIMESTAMPTZ  NOT NULL DEFAULT NOW()
          );

      - name: CreateIndexes
        run_when_succeeded: [CreateUsersTable]
        database: main
        hide_data_on_success: true
        # two statements: the native driver prepares each query, and a prepared
        # statement holds one command unless multi: true
        multi: true
        query: |
          CREATE INDEX IF NOT EXISTS idx_users_email      ON users (email);
          CREATE INDEX IF NOT EXISTS idx_users_created_at ON users (created_at DESC);

      - name: TruncateUsers
        run_when_succeeded: [CreateIndexes]
        database: main
        hide_data_on_success: true
        query: |
          TRUNCATE users RESTART IDENTITY;

      - name: InsertUsers
        run_when_succeeded: [TruncateUsers]
        database: main
        hide_data_on_success: true
        query: |
          INSERT INTO users (email, username, password_hash) VALUES
            ('alice@example.com',   'alice',   'hashed_pw_1'),
            ('bob@example.com',     'bob',     'hashed_pw_2'),
            ('carol@example.com',   'carol',   'hashed_pw_3'),
            ('dave@example.com',    'dave',    'hashed_pw_4'),
            ('eve@example.com',     'eve',     'hashed_pw_5');

      - name: SeedSummary
        run_when_succeeded: [InsertUsers]
        database: main
        query: |
          SELECT COUNT(*) AS users_inserted FROM users;
        post_transforms:
          - extract_value: "[0]"

  # ── Health ────────────────────────────────────────────────────────────────

  api/health:
    output: http
    summary: Health check
    description: Liveness probe. Returns server time and connected database name.
    tags: [system]
    produces:
      example:
        server_time: "2024-01-01T12:00:00Z"
        database: mydb

    actions:
      - name: Health
        database: main
        query: |
          SELECT NOW() AS server_time, current_database() AS database;
        post_transforms:
          - extract_value: "[0]"

      # Gauge: total live users sampled on every health check.
      # Visible in Grafana as app_users_total{config=..., interface=api/health}
      - name: SampleUserCount
        run_when_succeeded: [Health]
        database: main
        hide_data_on_success: true
        query: |
          SELECT COUNT(*) AS total FROM users;
        post_transforms:
          - extract_value: "[0]"

      - name: EmitUserGauge
        run_when_succeeded: [SampleUserCount]
        hide_data_on_success: true
        emit_metric:
          name: app_users_total
          type: gauge
          value: a|SampleUserCount::total|

  # ── Users ─────────────────────────────────────────────────────────────────

  api/users:
    output: http
    summary: List users
    description: Returns the 50 most recently created users.
    tags: [users]
    produces:
      example:
        - id: 1
          email: alice@example.com
          username: alice
          created_at: "2024-01-01T12:00:00Z"

    actions:
      - name: ListUsers
        database: main
        query: |
          SELECT id, email, username, created_at
          FROM users
          ORDER BY created_at DESC
          LIMIT 50;

      # Counter: track how many times the list endpoint is called.
      - name: TrackListRequest
        hide_data_on_success: true
        emit_metric:
          name: app_users_listed_total
          type: counter

  api/users/create:
    output: http
    method: POST
    summary: Create user
    description: Creates a new user. Password is bcrypt-hashed before storage.
    tags: [users]
    accepts:
      example:
        email: jane@example.com
        username: jane
        password: "Secret1!"
    produces:
      example:
        id: 6
        email: jane@example.com
        username: jane
        created_at: "2024-01-01T12:00:00Z"

    actions:
      - name: ValidateBody
        input: a|body|
        hide_data_on_success: true
        assert:
          http_code_on_error: 400
          tests:
            - value: email
              is_not_null: true
              is_not_empty: true
              description: "User email address"
            - value: username
              is_not_null: true
              is_not_empty: true
              description: "Unique username"
            - value: password
              is_not_null: true
              is_not_empty: true
              # Declare the type so the comparison is about how many characters this has,
              # not what it would be worth as a number. Without it the engine infers, and
              # an all-digit password would be judged on its value instead.
              data_type: String
              is_greater_than: 7
              description: "Password (minimum 8 characters)"

      - name: HashPassword
        run_when_succeeded:
          actions: [ValidateBody]
          http_code_on_error: 400
        input: a|ValidateBody|
        hide_data_on_success: true
        hide_data_on_error: true
        post_transforms:
          - bcrypt:
              value: password
              cost: 12

      - name: CreateUser
        run_when_succeeded:
          actions: [HashPassword]
          http_code_on_error: 400
        database: main
        query: |
          INSERT INTO users (email, username, password_hash)
          VALUES ($1, $2, $3)
          RETURNING id, email, username, created_at;
        params:
          - a|ValidateBody::email|
          - a|ValidateBody::username|
          - a|HashPassword::password|
        post_transforms:
          - extract_value: "[0]"

      # Counter: increment on every successful signup.
      # Labels let you slice by any dimension you care about.
      - name: TrackSignup
        run_when_succeeded: [CreateUser]
        hide_data_on_success: true
        emit_metric:
          name: app_users_created_total
          type: counter
          labels:
            source: api

  api/users/get:
    output: http
    method: POST
    summary: Get user
    description: Fetch a single user by ID.
    tags: [users]
    accepts:
      example:
        id: 1
    produces:
      example:
        id: 1
        email: alice@example.com
        username: alice
        created_at: "2024-01-01T12:00:00Z"

    actions:
      - name: ValidateBody
        input: a|body|
        hide_data_on_success: true
        assert:
          http_code_on_error: 400
          tests:
            - value: id
              is_not_null: true
              description: "User ID"

      - name: GetUser
        run_when_succeeded:
          actions: [ValidateBody]
          http_code_on_error: 400
        database: main
        query: |
          SELECT id, email, username, created_at FROM users WHERE id = $1;
        params:
          - a|ValidateBody::id|
        assert:
          http_code_on_error: 404
          error_message: "User not found"
          tests:
            - value: count()
              is_equal_to: 1
        post_transforms:
          - extract_value: "[0]"

      # Counter: track lookups separately from list calls.
      - name: TrackLookup
        run_when_succeeded: [GetUser]
        hide_data_on_success: true
        emit_metric:
          name: app_users_looked_up_total
          type: counter

  # ── System ────────────────────────────────────────────────────────────────

  # This endpoint always returns 500. Use it to populate the error-rate
  # panels in Grafana and verify alerting works.
  api/simulate/error:
    output: http
    method: POST
    summary: Simulate error
    description: >
      Always returns HTTP 500. Use this to verify error-rate panels and alerts
      in Grafana without touching real data.
    tags: [system]
    notes: |
      This endpoint is intentionally broken. It is safe to call repeatedly.

    actions:
      # Counter: track simulated errors so you can verify your alerting pipeline
      # fires correctly without waiting for real errors to occur. It runs first:
      # the response status comes from the last action, so the failing check must
      # be last for the 500 to reach the caller.
      - name: TrackSimulatedError
        hide_data_on_success: true
        emit_metric:
          name: app_errors_simulated_total
          type: counter
          labels:
            endpoint: simulate_error

      - name: ForceError
        database: main
        query: SELECT 1 AS result;
        assert:
          http_code_on_error: 500
          tests:
            - value: count()
              is_equal_to: 9999

schema.sql

CREATE TABLE IF NOT EXISTS users (
  id            BIGSERIAL    PRIMARY KEY,
  email         TEXT         NOT NULL UNIQUE,
  username      TEXT         NOT NULL UNIQUE,
  password_hash TEXT         NOT NULL,
  created_at    TIMESTAMPTZ  NOT NULL DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS idx_users_email      ON users (email);
CREATE INDEX IF NOT EXISTS idx_users_created_at ON users (created_at DESC);

grafana-dashboard.json

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "type": "datasource",
      "pluginId": "prometheus"
    }
  ],
  "title": "AirPipe Overview",
  "uid": "airpipe-overview",
  "schemaVersion": 38,
  "version": 2,
  "refresh": "10s",
  "time": { "from": "now-1h", "to": "now" },
  "panels": [
    {
      "id": 1,
      "title": "Request Rate by Route (req/s)",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "sum by (interface) (rate(airpipe_interface_requests_total[1m]))",
          "legendFormat": "{{interface}}"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "reqps", "custom": { "lineWidth": 2 } }
      }
    },
    {
      "id": 2,
      "title": "Error Rate (4xx + 5xx)",
      "type": "timeseries",
      "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "sum by (status) (rate(axum_http_requests_total{status=~\"4..|5..\"}[1m]))",
          "legendFormat": "HTTP {{status}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "reqps",
          "color": { "mode": "fixed", "fixedColor": "red" },
          "custom": { "lineWidth": 2 }
        }
      }
    },
    {
      "id": 3,
      "title": "Latency Percentiles, all routes (ms)",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 8, "w": 16, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "histogram_quantile(0.50, sum by (le) (rate(axum_http_requests_duration_seconds_bucket[1m]))) * 1000",
          "legendFormat": "p50"
        },
        {
          "expr": "histogram_quantile(0.95, sum by (le) (rate(axum_http_requests_duration_seconds_bucket[1m]))) * 1000",
          "legendFormat": "p95"
        },
        {
          "expr": "histogram_quantile(0.99, sum by (le) (rate(axum_http_requests_duration_seconds_bucket[1m]))) * 1000",
          "legendFormat": "p99"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "ms", "custom": { "lineWidth": 2 } }
      }
    },
    {
      "id": 4,
      "title": "In-Flight Requests",
      "type": "stat",
      "gridPos": { "x": 16, "y": 8, "w": 4, "h": 4 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "sum(axum_http_requests_pending)",
          "legendFormat": "in-flight"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "short", "thresholds": {
          "steps": [
            { "color": "green", "value": 0 },
            { "color": "yellow", "value": 50 },
            { "color": "red", "value": 100 }
          ]
        }}
      }
    },
    {
      "id": 5,
      "title": "Total Requests",
      "type": "stat",
      "gridPos": { "x": 20, "y": 8, "w": 4, "h": 4 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "sum(axum_http_requests_total)",
          "legendFormat": "total"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "short" }
      }
    },
    {
      "id": 6,
      "title": "Request Volume by Route",
      "type": "bargauge",
      "gridPos": { "x": 16, "y": 12, "w": 8, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "sum by (interface) (increase(airpipe_interface_requests_total[1h]))",
          "legendFormat": "{{interface}}"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "short" }
      },
      "options": { "orientation": "horizontal", "reduceOptions": { "calcs": ["lastNotNull"] } }
    },
    {
      "id": 7,
      "title": "Status Code Distribution",
      "type": "piechart",
      "gridPos": { "x": 0, "y": 16, "w": 8, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "sum(increase(axum_http_requests_total[1h])) by (status)",
          "legendFormat": "HTTP {{status}}"
        }
      ]
    },
    {
      "id": 8,
      "title": "p95 Latency by Route (last minute)",
      "type": "table",
      "gridPos": { "x": 8, "y": 16, "w": 16, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "expr": "max by (interface) (airpipe_interface_duration_ms{quantile=\"0.95\"})",
          "legendFormat": "{{interface}}",
          "instant": true
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "ms" }
      },
      "options": { "sortBy": [{ "displayName": "Value", "desc": true }] }
    }
  ]
}

Observability Showcase

What's included​

Auto-docs: how it works​

Prometheus metrics: how it works​

Built-in HTTP metrics​

Custom business metrics via emit_metric​

Setup​

1. Set the environment variable​

2. Create the table​

3. Run AirPipe​

4. Open the Swagger UI​

5. Configure Prometheus​

6. Import the Grafana dashboard​

Generating test traffic​

Grafana dashboard panels​

OpenTelemetry tracing​

Notes​

Configuration​

config.yml​

schema.sql​

grafana-dashboard.json​