Skip to Content
DocumentationReporterOverview

@lytics/playwright-reporter

Adapter-based Playwright reporter with pluggable storage backends.

Installation

npm install @lytics/playwright-reporter @lytics/playwright-annotations

Overview

CoreReporter is a flexible Playwright reporter that uses the adapter pattern to write test results to multiple storage backends simultaneously. It:

  • ✅ Validates required annotations (testCaseId, journeyId)
  • ✅ Tracks unique tests and handles retries
  • ✅ Detects flaky tests (passed after retry)
  • ✅ Parses and structures error information
  • ✅ Enriches results with environment data
  • ✅ Runs all adapters in parallel

Quick Start

Playwright reporters must be specified as file paths, not class instances. Create a reporter file:

// reporter.ts
import { CoreReporter } from '@lytics/playwright-reporter';
import { FilesystemAdapter } from '@lytics/playwright-adapters/filesystem';
 
class CustomReporter extends CoreReporter {
  constructor() {
    super({
      adapters: [
        new FilesystemAdapter({ outputDir: './test-results' })
      ]
    });
  }
}
 
export default CustomReporter;

Then reference it in your config:

// playwright.config.ts
import { defineConfig } from '@playwright/test';
 
export default defineConfig({
  reporter: [
    ['list'], // Built-in reporter for console output
    ['./reporter.ts'],
  ],
});

Features

Adapter Pattern

Write results to multiple backends simultaneously:

// reporter.ts
import { CoreReporter } from '@lytics/playwright-reporter';
import { FilesystemAdapter } from '@lytics/playwright-adapters/filesystem';
import { SlackAdapter } from '@lytics/playwright-adapters/slack';
import { FirestoreAdapter } from '@lytics/playwright-adapters/firestore';
 
class CustomReporter extends CoreReporter {
  constructor() {
    super({
      adapters: [
        new FilesystemAdapter({ outputDir: './test-results' }),
        new SlackAdapter({ webhookUrl: process.env.SLACK_WEBHOOK_URL! }),
        new FirestoreAdapter({ projectId: 'my-project', collections: { /* ... */ } }),
      ]
    });
  }
}
 
export default CustomReporter;

Annotation Validation

Tests must have journeyId and testCaseId annotations. Tests missing these annotations are skipped with a warning.

// ✅ Valid - has required annotations
test('my test', async ({}, testInfo) => {
  pushTestAnnotations(testInfo, {
    journeyId: 'MY_FEATURE_ACTION',
    testCaseId: 'MY_FEATURE_ACTION_VALID',
  });
});
 
// ⚠️ Skipped - missing annotations
test('unannotated test', async ({ page }) => {
  // This test will be skipped by the reporter
});

Retry Handling

The reporter intelligently handles test retries:

  • Tracks unique tests by testCaseId
  • Only the latest execution is counted in final stats
  • Detects flaky tests (passed after retry)
  • Reports both totalTests (unique) and totalExecutions (including retries)
Test A: fail → retry → pass  ✅ (flaky)
Test B: fail → retry → fail  ❌
Test C: pass                  ✅

totalTests: 3
totalExecutions: 5
passed: 2
failed: 1
flakyTests: 1

Structured Errors

Playwright errors are parsed into structured format:

interface TestError {
  matcher: string;      // e.g., "toHaveText", "toBeVisible"
  expected: string;
  actual: string;
  locator: string;
  location: {
    file: string;
    line: number;
    column: number;
  };
  message: string;
  snippet: string[];    // Code snippet around error
}

Environment Enrichment

Add custom metadata to test runs:

// reporter.ts
class CustomReporter extends CoreReporter {
  constructor() {
    super({
      adapters: [/* ... */],
      environmentEnricher: () => ({
        branch: process.env.GITHUB_REF,
        commit: process.env.GITHUB_SHA,
        author: process.env.GITHUB_ACTOR,
        buildNumber: process.env.GITHUB_RUN_NUMBER,
        nodeVersion: process.version,
      })
    });
  }
}

How It Works

Execution Flow

1. onBegin()
   └─> Initialize all adapters in parallel

2. For each test:
   onTestBegin(test)
   └─> Track test start
   
   onTestEnd(test, result)
   └─> Validate annotations
   └─> Track unique test state (handle retries)
   └─> Map to CoreTestResult
   └─> Write to all adapters in parallel

3. onEnd(result)
   └─> Calculate aggregate statistics
   └─> Map to CoreTestRun
   └─> Write to all adapters in parallel
   └─> Close all adapters

Core Types

CoreTestResult

Individual test result with full metadata:

interface CoreTestResult {
  testCaseId: string;
  journeyId: string;
  title: string;
  annotations: TestAnnotations;
  status: "passed" | "failed" | "timedOut" | "skipped" | "cancelled" | "interrupted" | "unknown";
  projectName: string;
  durationMs: number;
  error?: TestError;
  timestamp: Date;
  buildId: string;
  reportLink?: string;
}

CoreTestRun

Test run summary with aggregate statistics:

interface CoreTestRun {
  runId: string;
  timestamp: Date;
  overallStatus: "passed" | "failed" | "timedOut" | "skipped" | "cancelled" | "interrupted" | "unknown";
  totalTests: number;           // Unique tests (deduped)
  totalExecutions: number;      // Total executions (including retries)
  passed: number;
  failed: number;
  skipped: number;
  durationMs: number;
  passRate: number;             // 0-1
  averageTestDuration: number;
  slowestTestDuration: number;
  flakyTests: number;           // Tests that passed after retries
  environment: Record<string, unknown>;
}

Next Steps

Last updated on
ValidationUsage & Configuration