Skip to Content
DocumentationCore Concepts

Core Concepts

Understanding the annotation-driven testing approach.

The Problem

Traditional Playwright tests lack metadata that connects them to business requirements:

// ❌ No traceability
test('user can create token', async ({ page }) => {
  // Which feature area is this?
  // What user journey does it validate?
  // How do we track trends over time?
});

When tests fail, you need to manually trace back to understand:

  • What feature is affected?
  • Is this a critical user journey?
  • Has this test been flaky before?

The Solution: Annotation-Driven Testing

Every test is annotated with metadata linking it to user journeys and business requirements:

// ✅ Full traceability
test('user can create token', async ({ page }, testInfo) => {
  pushTestAnnotations(testInfo, {
    journeyId: 'ACCOUNT_SECURITY_CREATE-TOKEN',
    testCaseId: 'ACCOUNT_SECURITY_CREATE-TOKEN_VALID',
  });
  // Now we know: feature area, journey, and specific scenario
});

The Three-Level Hierarchy

Annotations follow a Suite → Journey → Test Case hierarchy:

┌─────────────────────────────────────────────────────────────┐
│ Test Suite: ACCOUNT_SECURITY                                │
│ (Product/feature area - set in beforeEach)                  │
│                                                             │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ Journey: ACCOUNT_SECURITY_VIEW-TOKENS                  │ │
│  │ (User flow - one per test file)                        │ │
│  │                                                        │ │
│  │  ┌──────────────────────────────────────────────────┐  │ │
│  │  │ Test Case: ACCOUNT_SECURITY_VIEW-TOKENS_VALID    │  │ │
│  │  │ (Specific scenario - unique per test)            │  │ │
│  │  └──────────────────────────────────────────────────┘  │ │
│  │  ┌──────────────────────────────────────────────────┐  │ │
│  │  │ Test Case: ACCOUNT_SECURITY_VIEW-TOKENS_EMPTY    │  │ │
│  │  └──────────────────────────────────────────────────┘  │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                             │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ Journey: ACCOUNT_SECURITY_CREATE-TOKEN                 │ │
│  │                                                        │ │
│  │  ┌──────────────────────────────────────────────────┐  │ │
│  │  │ Test Case: ACCOUNT_SECURITY_CREATE-TOKEN_VALID   │  │ │
│  │  └──────────────────────────────────────────────────┘  │ │
│  └────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Test Suite (testSuiteName)

Scope: Product or feature area (1 suite : N journeys)

Set in: beforeEach at the describe level

Examples:

  • ACCOUNT_SECURITY
  • USER_MANAGEMENT
  • DATA_PIPELINES

Journey (journeyId)

Scope: A specific user flow (1 journey : N test cases)

Format: {testSuiteName}_{JOURNEY-NAME}

Set in: Each test() function

Examples:

  • ACCOUNT_SECURITY_VIEW-TOKENS
  • USER_MANAGEMENT_CREATE-USER
  • DATA_PIPELINES_CREATE-JOB

Test Case (testCaseId)

Scope: A specific scenario (unique per test)

Format: {journeyId}_{SCENARIO}

Set in: Each test() function (unique per test)

Examples:

  • ACCOUNT_SECURITY_VIEW-TOKENS_VALID
  • ACCOUNT_SECURITY_VIEW-TOKENS_EMPTY
  • USER_MANAGEMENT_CREATE-USER_INVALID

Test Case IDs must be globally unique and stable. They’re used for trend analysis and tracking test history over time.

Naming Conventions

Flexible Delimiters

Hyphens and underscores are normalized for validation:

// All of these are valid and equivalent for hierarchy validation:
testSuiteName: "HOME_DASHBOARD"
journeyId: "HOME-DASHBOARD_RECIPE"     // Hyphen variant OK
testCaseId: "HOME-DASHBOARD_RECIPE_VALID"
LevelPatternExample
SuitePRODUCT_FEATUREACCOUNT_SECURITY
Journey{SUITE}_{ACTION}ACCOUNT_SECURITY_VIEW-TOKENS
Test Case{JOURNEY}_{SCENARIO}ACCOUNT_SECURITY_VIEW-TOKENS_VALID

Common scenario suffixes:

  • _VALID — Happy path
  • _INVALID — Validation errors
  • _EMPTY — Empty state
  • _ERROR — Error handling
  • _EDGE — Edge cases

Validation Rules

The annotation framework enforces these rules:

RuleLevelDescription
RequiredErrorAll three annotations must be present
HierarchyErrortestCaseId must start with journeyId
UniquenessErrortestCaseId cannot equal journeyId (must add suffix)
ConsistencyWarningjourneyId should start with testSuiteName

Benefits

1. Traceability

Link tests directly to requirements:

  • Test → Journey → Feature → Requirement
  • Quickly identify what’s affected when tests fail

2. Observability

Build dashboards showing:

  • Pass rates by suite, journey, or test case
  • Trends over time
  • Flaky test patterns

3. Coverage Tracking

Identify gaps:

  • Which journeys have no tests?
  • Which scenarios are missing?
  • What’s the coverage per feature area?

4. Trend Analysis

Track stability:

  • Is this test consistently failing?
  • When did it start failing?
  • Is it flaky (passes after retry)?

Custom Schemas

The Contentstack conventions are a reference implementation. You can define your own schema:

import { pushAnnotations, createValidator } from '@lytics/playwright-annotations';
import type { TestAnnotations } from '@lytics/playwright-annotations';
 
// Define your own schema
interface MyTeamAnnotations extends TestAnnotations {
  epic: string;
  story: string;
  severity: 'blocker' | 'critical' | 'major' | 'minor';
}
 
// Create a validator
const validateMyAnnotations = createValidator<MyTeamAnnotations>({
  rules: [
    (a) => ({
      valid: Boolean(a.epic && a.story),
      errors: !a.epic || !a.story ? ['epic and story are required'] : [],
      warnings: [],
    }),
  ]
});
 
// Use in tests
test('my test', async ({}, testInfo) => {
  pushAnnotations(testInfo, {
    epic: 'AUTH-2023',
    story: 'AUTH-2023-01',
    severity: 'critical'
  });
});

See Annotations → Custom Schema for more details.

Last updated on
QuickstartOverview