Test Observability

Overview

Connect your Playwright tests to your observability stack (Grafana, Prometheus, Loki, Tempo) for:

• Trace Correlation: See full trace from browser click → backend → database
• Test Metrics: Dashboard with pass rates, durations, flakiness
• Log Aggregation: Test logs alongside application logs
• Failure Analysis: Quickly identify root cause across distributed systems

---

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Playwright     │────▶│  OTEL Collector  │────▶│  Tempo (Traces) │
│  Tests          │     │                  │────▶│  Loki (Logs)    │
│                 │     │                  │────▶│  Prometheus     │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                          │
                                                          ▼
                                                  ┌─────────────────┐
                                                  │    Grafana      │
                                                  │  Test Dashboard │
                                                  └─────────────────┘

---

Quick Start: OTEL Reporter

1. Install Dependencies

npm install playwright-opentelemetry-reporter @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http

2. Configure Reporter

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-opentelemetry-reporter', {
      serviceName: 'playwright-tests',
      endpoint: process.env.OTEL_ENDPOINT || 'http://localhost:4318/v1/traces',
    }],
  ],
});

3. Run Tests

# With local OTEL collector
OTEL_ENDPOINT=http://localhost:4318/v1/traces npx playwright test

# With Railway OTEL collector
OTEL_ENDPOINT=http://otel-collector.railway.internal:4318/v1/traces npx playwright test

4. View in Grafana

• Open Grafana → Explore → Tempo
• Search for service.name = "playwright-tests"
• See test spans with duration, status, steps

---

Tracetest Integration (Advanced)

Tracetest enables trace-based testing - assert on any span in the distributed trace.

Installation

npm install @tracetest/playwright

Configuration

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html'],
    ['@tracetest/playwright', {
      serverUrl: process.env.TRACETEST_URL || 'http://localhost:11633',
      apiKey: process.env.TRACETEST_API_KEY,
    }],
  ],
});

Trace-Based Assertions

// tests/checkout.spec.ts
import { test, expect } from '@playwright/test';
import { Tracetest } from '@tracetest/playwright';

test('checkout creates order', async ({ page }) => {
  const tracetest = new Tracetest();

  // Start trace capture
  await tracetest.capture();

  // Perform user action
  await page.goto('/checkout');
  await page.click('#place-order');
  await expect(page.locator('.order-confirmation')).toBeVisible();

  // Assert on backend trace!
  await tracetest.assertOnTrace({
    assertions: [
      // API response time
      {
        selector: 'span[name="POST /api/orders"]',
        assertion: 'attr:http.status_code = 201',
      },
      // Database query time
      {
        selector: 'span[name="INSERT orders"]',
        assertion: 'attr:db.duration < 100ms',
      },
      // Payment service
      {
        selector: 'span[name="payment.process"]',
        assertion: 'attr:payment.status = "success"',
      },
    ],
  });
});

Benefits

• 80% faster debugging: See exactly where failures occur
• Full visibility: Browser → API → Database → Services
• Backend assertions: Test database queries, service calls
• Performance validation: Assert on span durations

---

Grafana Dashboard

Import Dashboard

1. Open Grafana → Dashboards → Import
2. Upload dashboards/test-results-dashboard.json
3. Select Prometheus and Tempo data sources

Key Panels

Custom Metrics

// Export custom metrics from tests
import { test } from '@playwright/test';
import { metrics } from '@opentelemetry/api';

const testMeter = metrics.getMeter('playwright-tests');
const testDuration = testMeter.createHistogram('test.duration');
const testCounter = testMeter.createCounter('test.count');

test('custom metrics', async ({ page }) => {
  co

...