Automated Acceptance Testing
Link repository tests to your SOW, run selected checks, and review the evidence from each run
Use automated acceptance tests to check a SOW against your application and keep the results with the work they verify. A developer links an existing repository test to an acceptance test in Tanda. Editors can then run the linked checks, and reviewers can open the results in the same SOW.
Each run records the selected tests, code commit, environment, start time, and results. Starting another run creates a separate record; earlier results keep the definitions used when they ran.
Open your SOW's Acceptance Test Procedure → Run code tests for automation. Run Full Test Procedure opens the manual testing workflow. These are currently separate actions; starting a manual procedure does not launch repository tests.
Before your first run
You need:
- An acceptance test under a criterion in your SOW, with a clear expected result.
- An existing repository test whose assertions check that result.
- A runner connection enabled for the project or SOW by your Tanda administrator. A runner is the CI job or local process that executes the tests and sends results to Tanda.
- Editor access to configure or start runs. Viewers can inspect results.
If you see No runner is connected to this SOW, ask your administrator to connect the repository and enable the environments you need. They can follow the runner setup guide. Saving a test definition alone does not connect a runner.
Choose project defaults or SOW settings
On the project page, open Code test settings to select an approved connection and set the repository, workflow, Git ref, enabled environments, default environment, and run timeout. Click Save settings.
In a SOW, open Run code tests → Execution settings to see the inherited values. Override individual fields when this SOW needs a different configuration, or select Use project defaults for all fields to remove its overrides. A SOW without a project can use its own settings.
Project changes apply to future runs for SOWs that inherit those fields. They do not change completed runs. Available choices depend on the connection your administrator approved.
Link a repository test
- Open Run code tests → Configure code tests → Add repository test.
- Select the Acceptance test that the code verifies and its Framework.
- Enter the repository details below, then click Save bindings.
- Reopen Configure code tests and expand Import or copy repository JSON. Have the developer commit that same configuration as
.tanda/tests.jsonalongside the tests before running the selected commit.
The link between a Tanda test and a repository test is called a binding. New configurations start empty. You can also paste an existing repository configuration into Import or copy repository JSON and review it before saving.
| Field | What to enter |
|---|---|
| Stable test key | A unique, lasting label such as MY-TASKS-LOADS. Keep it stable when the test's display title changes. |
| Working directory | The directory where the test command runs, relative to the repository root. Use . for the root. |
| Test file | The existing file path relative to the working directory. |
| Test name or Karate tags | Optional filter for cases within the file. Playwright and Vitest use literal name matching; Karate uses tags. Leaving this empty selects the file's tests. |
| Report path | Required for Karate or JUnit; relative to the working directory. Other adapters generate their reports. |
| Timeout (seconds) | Maximum execution time for this binding; the default is 300 seconds. |
For example, if a developer has written a Playwright check for My Tasks loads without an API error, they can link its actual file and test name to that acceptance test. Review that the code checks the tasks request and the visible result. A test that only checks the page title does not establish that the API works.
Each binding must use a different stable key and a different Tanda acceptance test. A binding can select a group of cases; every selected case must pass for the binding to pass. Use separate acceptance tests and bindings when you need separate outcomes. A run supports up to 100 bindings.
Supported frameworks
| Framework | How results reach Tanda |
|---|---|
| Playwright | The runner executes browser tests and collects a JUnit report. |
| Vitest | The runner executes selected tests and collects a JUnit report. |
| Karate | The runner executes Maven tests. The repository must be configured to produce JUnit XML at the report path. |
| ESLint | The runner checks the selected file and collects a JSON report. |
| JUnit | Your existing build executes the tests, then the runner imports its report. |
The repository's CI setup must install the framework and its dependencies, including browsers when needed. Selecting a framework in Tanda does not install it. There is currently no project or SOW plugin marketplace.
Run the selected checks
- Under Selected tests, check the bindings to run.
- Enter the full 40-character Git commit SHA containing the tests and matching
.tanda/tests.json. - Choose the Environment you intend to test, such as staging.
- Choose an execution method:
- Start GitHub Actions, then Run in GitHub Actions: Tanda creates a run and asks the connected workflow to execute it.
- Local or existing CI, then Create run for external reporting: Tanda creates the record. Give its run ID and displayed commands to the developer or CI operator to execute and submit the results.
GitHub runs must use a commit allowed by the connection's configured Git ref. The runner also checks that the selected bindings match the configuration committed with that code.
Creating an external run does not execute tests by itself. The operator must submit the report before the run's deadline. See the runner setup guide for local execution and importing reports from an existing build.
Review results and evidence
Open Recent runs in the same panel and select a run. Each entry shows its status, environment, commit, and creation time. The report includes the full commit, run ID, per-binding outcomes, notes, durations when available, and any artifact links supplied by the runner. Active runs refresh automatically while the panel is open.
| Result | Meaning and next step |
|---|---|
| Passed | All selected checks passed. Review their scope and evidence before accepting the requirement. |
| Failed | A check ran and an assertion failed. Inspect the report, fix the behavior, and run again. |
| Skipped or blocked | A selected check was not completed. Resolve the reason and run it again; this is not a passing result. |
| Error | Execution or reporting failed, for example because of a timeout, missing test, or invalid report. Inspect the diagnostic notes and CI logs. |
| Dispatch failed | Tanda could not confirm a successful workflow dispatch. Check GitHub before starting a replacement run. |
An automated result is recorded against the corresponding acceptance test and cannot be rewritten through the manual result editor. A passing subset does not mean every criterion in the SOW has been tested.
Artifact links open the runner's storage, such as GitHub Actions. You may need access to that system. Tanda retains the results, links, and file hashes; it does not keep a permanent copy of externally stored logs, screenshots, or reports. Their availability depends on the storage provider's retention settings. Screenshots and traces require the repository workflow to upload them.
Run again and keep the history
After a fix, select the tests, enter the new commit if needed, and start another run. Keep the earlier run as evidence of what happened before the fix. To repeat only failed checks, select those bindings manually; there is currently no one-click failed-only rerun.
Runs retain the test definitions, SOW scope, execution settings, and results captured for that run. Later edits do not rewrite earlier evidence. Tanda also records who requested the run in its audit history. Recent runs shows the latest 30 code runs; a known older run ID can be retrieved through the API or MCP.
To stop collecting a queued or running result, open the run and click Cancel result collection. Cancel the job in your CI provider as well if you need to stop the executing process.
Troubleshooting
| What you see | What to check |
|---|---|
| No connection or environment is available | Ask your administrator to enable a connection for this project or SOW, then select it in execution settings. |
| Save bindings is unavailable | Check editor access, the selected connection, required fields, duplicate keys or acceptance tests, and paths. An archived or locked SOW cannot be edited. |
| An acceptance test is unavailable | The imported configuration references a test outside this SOW or one that was removed. Choose an existing test from this SOW. |
| Start is unavailable | Select at least one binding, a valid full commit SHA, and an environment. Check any execution settings error. |
| Repository manifest differs | The saved bindings and .tanda/tests.json at the selected commit disagree. Save and commit the same configuration, then use that commit. |
| An external run stays queued | Confirm that the operator started the runner for that run ID and environment. |
| No artifact links | Ask the CI operator whether evidence was uploaded and its links submitted. Results can be recorded without uploaded artifacts. |
Work with an IDE assistant
An assistant connected through MCP can inspect criteria, help identify tests in the repository it can access, configure bindings, start runs, and read reports. The team sees those same code runs in Tanda. Follow the IDE testing workflow for example requests.
Automatic repository discovery, generated executable tests, and a combined manual-and-automated Start Tests workflow are planned work. For now, a developer or assistant reviews the existing tests and explicitly links them using the steps above.