Test Runs

Test Runs are instances of executing a test or set of tests. They capture the results, timeline, and any issues discovered during execution, creating an audit trail of verification activities.

Overview

Test Runs provide a complete record of test execution, documenting who performed the test, when it was executed, what was observed, and whether the criterion was met.

Test Runs:

• Record who performed the test
• Capture when the test was executed
• Document the outcome (pass/fail)
• Include notes and observations
• Store evidence and artifacts
• Create an audit trail
• Enable test history tracking

The Test Run Model

Creating Test Runs

Via the UI

1. Navigate to a test
2. Click "Execute Test"
3. Follow each test function
4. Record results for each function:
   • Actual result observed
   • Pass/Fail status
   • Any notes
5. Add overall test run information:
   • General observations
   • Evidence files
   • Environment details
6. Click "Complete Test Run"

Via the API

// Create a new test run
const testRun = await apiClient.testRuns.create({
  testId: "test_id",
  executedBy: "user_id",
  executedAt: new Date().toISOString(),
  environment: "Production",
  functionResults: [
    {
      functionId: "function_1",
      status: "passed",
      actualResult: "Login page loaded with all expected elements",
      notes: "Page load time was 1.2 seconds"
    },
    {
      functionId: "function_2",
      status: "passed",
      actualResult: "Email address accepted and displayed correctly",
      notes: null
    },
    {
      functionId: "function_3",
      status: "passed",
      actualResult: "Password was properly masked",
      notes: null
    },
    {
      functionId: "function_4",
      status: "failed",
      actualResult: "Login failed with error: 'Invalid credentials'",
      notes: "Test data may be incorrect"
    }
  ],
  status: "failed",
  notes: "Test failed at login step. Need to verify test credentials.",
  duration: 5
});

Test Run Statuses

Overall Test Run Status

• Passed: All test functions passed
• Failed: One or more test functions failed
• Blocked: Test could not be completed due to a blocking issue
• Skipped: Test was intentionally not executed

Function-Level Status

Each function within a test run can have its own status:

• Passed: Function executed successfully with expected results
• Failed: Function did not produce expected results
• Blocked: Function could not be executed
• Skipped: Function was intentionally skipped

Recording Evidence

Test runs can include various types of evidence:

1. Screenshots: Visual proof of test results
2. Log Files: System or application logs
3. Data Exports: CSV, JSON, or other data files
4. Videos: Screen recordings of test execution
5. Documents: Supporting documentation

To attach evidence:

// Attach evidence to a test run
await apiClient.testRuns.attachEvidence(
  "test_run_id",
  {
    filename: "login_error_screenshot.png",
    type: "screenshot",
    description: "Screenshot showing login error message",
    data: base64EncodedImageData
  }
);

Test Run History

Test runs create a historical record that enables:

• Trend Analysis: Track pass/fail rates over time
• Issue Identification: Spot recurring problems
• Audit Trail: Demonstrate testing compliance
• Performance Tracking: Monitor test execution times
• Root Cause Analysis: Investigate failures

Viewing Test Run History

Via the UI

1. Navigate to a test
2. Click "Run History"
3. View list of all executions
4. Click on any run to see details

Via the API

// Get test run history
const history = await apiClient.tests.getRunHistory("test_id", {
  limit: 10,
  orderBy: "executedAt",
  order: "desc"
});

// Get specific test run details
const runDetails = await apiClient.testRuns.get("test_run_id");

Best Practices

• Complete Documentation: Record all observations, not just pass/fail
• Timely Execution: Execute test runs promptly when needed
• Detailed Notes: Include context that might help future debugging
• Evidence Collection: Capture screenshots or logs for failed functions
• Environment Details: Always note which environment was tested
• Accurate Timing: Record actual time spent on test execution
• Objective Results: Report what was observed, not interpretations

Test Run Reports

Test runs can be aggregated into reports showing:

• Overall test coverage
• Pass/fail trends
• Most frequently failing tests
• Average execution times
• Tester productivity
• Environment-specific results

API Reference

For complete API details, refer to the API Reference.