Validation

Validation framework for ensuring annotation correctness.

Contentstack Validation

validateContentstackAnnotations

Validate Contentstack annotation format.


function validateContentstackAnnotations(
  annotations: ContentstackAnnotations
): ValidationResult

Returns:


interface ValidationResult {
  valid: boolean;    // true if no errors
  errors: string[];  // Validation failures
  warnings: string[]; // Non-blocking issues
}

Example:


import { validateContentstackAnnotations } from '@lytics/playwright-annotations';
 
const result = validateContentstackAnnotations({
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
  testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS_VALID',
});
 
if (!result.valid) {
  console.error('Validation errors:', result.errors);
}
 
if (result.warnings.length > 0) {
  console.warn('Validation warnings:', result.warnings);
}

Validation Rules

The Contentstack validator enforces these rules:

Rule	Level	Description
Required fields	Error	All three annotations must be present
Hierarchy	Error	`testCaseId` must start with `journeyId`
Uniqueness	Error	`testCaseId` cannot equal `journeyId` (must add suffix)
Consistency	Warning	`journeyId` should start with `testSuiteName`

Rule 1: Required Fields

All three annotations must be present and non-empty.


// ❌ Invalid - missing journeyId
{
  testSuiteName: 'ACCOUNT_SECURITY',
  testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS_VALID',
}
// Error: "journeyId is required"
 
// ✅ Valid
{
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
  testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS_VALID',
}

Rule 2: Hierarchy (testCaseId starts with journeyId)

The testCaseId must start with the journeyId to maintain hierarchy.


// ❌ Invalid - testCaseId doesn't start with journeyId
{
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
  testCaseId: 'SOMETHING_ELSE_VALID',
}
// Error: "testCaseId must start with journeyId"
 
// ✅ Valid
{
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
  testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS_VALID',
}

Rule 3: Uniqueness (testCaseId ≠ journeyId)

The testCaseId must add a suffix to differentiate from journeyId.


// ❌ Invalid - testCaseId equals journeyId
{
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
  testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
}
// Error: "testCaseId must be different from journeyId"
 
// ✅ Valid - added _VALID suffix
{
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
  testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS_VALID',
}

Rule 4: Consistency (Warning)

The journeyId should start with testSuiteName for consistency.


// ⚠️ Warning - journeyId doesn't start with testSuiteName
{
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'USER_MANAGEMENT_CREATE-USER',
  testCaseId: 'USER_MANAGEMENT_CREATE-USER_VALID',
}
// Warning: "journeyId should start with testSuiteName for consistency"
 
// ✅ No warning
{
  testSuiteName: 'ACCOUNT_SECURITY',
  journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
  testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS_VALID',
}

Flexible Delimiters

Hyphens and underscores are normalized for comparison:


// All valid - delimiters are normalized
{
  testSuiteName: 'HOME_DASHBOARD',
  journeyId: 'HOME-DASHBOARD_RECIPE',     // Hyphen variant OK
  testCaseId: 'HOME-DASHBOARD_RECIPE_VALID',
}

Custom Validators

createValidator

Factory function for creating custom validators.


function createValidator<T extends TestAnnotations>(
  config: ValidatorConfig<T>
): (annotations: T) => ValidationResult

Parameters:


interface ValidatorConfig<T extends TestAnnotations> {
  rules: ValidationRule<T>[];
}
 
type ValidationRule<T extends TestAnnotations> = (
  annotations: T
) => ValidationResult;

Example:


import { createValidator } from '@lytics/playwright-annotations';
import type { TestAnnotations } from '@lytics/playwright-annotations';
 
// 1. Define your schema
interface MyAnnotations extends TestAnnotations {
  epic: string;
  story: string;
  severity: 'blocker' | 'critical' | 'major' | 'minor';
}
 
// 2. Create validation rules
const rules = [
  // Rule: epic and story are required
  (annotations: MyAnnotations) => ({
    valid: Boolean(annotations.epic && annotations.story),
    errors: !annotations.epic || !annotations.story 
      ? ['epic and story are required'] 
      : [],
    warnings: [],
  }),
 
  // Rule: story should start with epic
  (annotations: MyAnnotations) => {
    const storyStartsWithEpic = annotations.story?.startsWith(annotations.epic);
    return {
      valid: true, // This is a warning, not an error
      errors: [],
      warnings: storyStartsWithEpic 
        ? [] 
        : ['story should start with epic for consistency'],
    };
  },
 
  // Rule: blocker severity requires epic prefix "CRITICAL-"
  (annotations: MyAnnotations) => {
    if (annotations.severity === 'blocker' && !annotations.epic.startsWith('CRITICAL-')) {
      return {
        valid: false,
        errors: ['blocker severity requires epic to start with "CRITICAL-"'],
        warnings: [],
      };
    }
    return { valid: true, errors: [], warnings: [] };
  },
];
 
// 3. Create the validator
const validateMyAnnotations = createValidator<MyAnnotations>({ rules });
 
// 4. Use it
const result = validateMyAnnotations({
  epic: 'AUTH-2023',
  story: 'AUTH-2023-01',
  severity: 'critical',
});
 
if (!result.valid) {
  throw new Error(`Invalid annotations: ${result.errors.join(', ')}`);
}

Writing Validation Rules

Each rule is a function that receives annotations and returns a ValidationResult:


type ValidationRule<T> = (annotations: T) => {
  valid: boolean;    // false = validation failure
  errors: string[];  // Error messages (blocking)
  warnings: string[]; // Warning messages (non-blocking)
};

Best Practice: Keep rules focused on a single concern. Multiple simple rules are easier to maintain than one complex rule.

Combining Multiple Validators

You can compose validators for complex scenarios:


function validateAll(annotations: MyAnnotations): ValidationResult {
  const results = [
    validateMyAnnotations(annotations),
    validateAdditionalRules(annotations),
  ];
 
  return {
    valid: results.every(r => r.valid),
    errors: results.flatMap(r => r.errors),
    warnings: results.flatMap(r => r.warnings),
  };
}

Integration with Tests

Validate on Push

Validate annotations when pushing them:


import { pushAnnotations, validateContentstackAnnotations } from '@lytics/playwright-annotations';
 
test('my test', async ({}, testInfo) => {
  const annotations = {
    testSuiteName: 'ACCOUNT_SECURITY',
    journeyId: 'ACCOUNT_SECURITY_VIEW-TOKENS',
    testCaseId: 'ACCOUNT_SECURITY_VIEW-TOKENS_VALID',
  };
 
  const validation = validateContentstackAnnotations(annotations);
  if (!validation.valid) {
    throw new Error(`Invalid annotations: ${validation.errors.join(', ')}`);
  }
 
  pushAnnotations(testInfo, annotations);
});

Validate in Reporter

Validate annotations when processing results:


import { getContentstackAnnotations, validateContentstackAnnotations } from '@lytics/playwright-annotations';
 
// In a custom reporter
onTestEnd(test, result) {
  const annotations = getContentstackAnnotations(test);
  
  if (!annotations) {
    console.warn(`Test "${test.title}" is missing annotations`);
    return;
  }
 
  const validation = validateContentstackAnnotations(annotations);
  if (!validation.valid) {
    console.error(`Test "${test.title}" has invalid annotations:`, validation.errors);
  }
}

Custom Schemas

Example: Jira-Based Schema


interface JiraAnnotations extends TestAnnotations {
  project: string;      // JIRA project key
  issueKey: string;     // Full issue key (e.g., "PROJECT-123")
  testType: 'unit' | 'integration' | 'e2e';
}
 
const validateJiraAnnotations = createValidator<JiraAnnotations>({
  rules: [
    // issueKey must match pattern
    (a) => ({
      valid: /^[A-Z]+-\d+$/.test(a.issueKey),
      errors: /^[A-Z]+-\d+$/.test(a.issueKey) 
        ? [] 
        : ['issueKey must match pattern: PROJECT-123'],
      warnings: [],
    }),
 
    // issueKey should start with project
    (a) => ({
      valid: true,
      errors: [],
      warnings: a.issueKey.startsWith(a.project) 
        ? [] 
        : ['issueKey should start with project key'],
    }),
  ],
});

Example: BDD-Style Schema


interface BDDAnnotations extends TestAnnotations {
  feature: string;
  scenario: string;
  given: string;
  when: string;
  then: string;
}
 
const validateBDDAnnotations = createValidator<BDDAnnotations>({
  rules: [
    (a) => ({
      valid: Boolean(a.feature && a.scenario && a.given && a.when && a.then),
      errors: ['All BDD fields (feature, scenario, given, when, then) are required'],
      warnings: [],
    }),
  ],
});