User Guide
Connected to FPD

User Guide

ES EN

User guide — Full Testing Web (FTW)

This guide explains how to use Full Testing Web, the web control panel that orchestrates the Full Testing AI automation framework against the demo application Banco Americano. It is written for the QA team: what each screen does, how tests are authored with AI, the exact decision rules of the natural-language automation and the self-healing, and where every artifact lives.

The same guide exists in Spanish (user-guide.es.md) and can be read inside the app itself on the User Guide page in the sidebar.

1. What FTW is and how it fits into the ecosystem

FTW is a FastAPI + Jinja + htmx application running locally (http://localhost:9000). It does not execute tests itself: it coordinates three pieces:

                        ┌──────────────────────────────────────────────┐
                        │  FPD — Full Project Director (port 8000)     │
                        │  · Serves the app under test: Banco Americano│
                        │    (/playground/banco-americano)             │
  reads test cases      │  · Test-case API (GET /api/tcs)              │
  (GET /api/tcs)        │  · LLM gateway                               │
 ┌──────────────────┐   │    (POST /api/ai/complete, task=generation)  │
 │                  │ ─▶ └──────────────────────────────────────────────┘
 │  Full Testing    │                ▲
 │  Web (FTW)       │ ───────────────┘  every AI call goes through FPD
 │  port 9000       │   LLM calls       with the FTW_FPD_TOKEN
 │                  │
 │  FastAPI + htmx  │   ┌──────────────────────────────────────────────┐
 │                  │   │  Full Testing AI (framework, separate repo)  │
 │  Spawns Maven    │ ─▶│  Java 21 · Maven · JUnit 5 · Playwright      │
 │  processes and   │   │  POM + Flow Objects pattern                  │
 │  reads results   │   │  Portable toolchain: ../.tools/jdk,          │
 └───────┬──────────┘   │  ../.tools/maven                             │
         │              └───────────────┬──────────────────────────────┘
         │ push to main only when       │ push (always a human decision)
         │ a human hits Publish         ▼
         │                     ┌──────────────────┐
         └────────────────────▶│  GitHub (repo     │
                               │  full-testing-ai, │
                               │  Actions CI)      │
                               └──────────────────┘
  • FPD (Full Project Director) — the host application. It serves Banco Americano (the app under test), the test-case (TC) read API, and the only gateway to the LLM: POST /api/ai/complete with task: "generation". FTW never talks to the AI provider directly.
  • Full Testing AI (FTA) — the test framework, an independent repo in Java 21 / Maven / JUnit 5 / Playwright with a hybrid POM + Flow Objects pattern (three layers: page objects with locators and atomic actions; flow objects with reusable business flows like login; tests with orchestration and assertions). Executed tests are deterministic code: AI only takes part in authoring and healing, never inside mvn test.
  • GitHub — the JulioCOropeza/full-testing-ai repo with GitHub Actions (tests.yml, manual workflow_dispatch). Nothing runs on private servers.

The publish philosophy (golden rule)

The framework repo is treated as read-only. FTW reads files (coverage registry, sources, surefire reports) and runs the framework's own Maven commands, but it never writes to the repo until a human hits a publish button:

  • Push to main on a scaffold or a verified automation.
  • Apply & push to main on a finished refactor.
  • In self-healing, main is never touched: the fix is pushed to a heal/... branch and a PR is suggested for human review.

On top of that, publish buttons only version exactly the files that job generated (never git add -A): unrelated changes you may have in the framework working tree are never touched. The one deliberate exception: a dirty coverage/coverage.json (regenerated by the verification run) rides along in the publish commit, so the @Covers registration reaches main together with the code.

Verification gates: nothing reaches main unverified. The NL automation path only enables Push to main at status verified (green headless run); the refactor path has the same gate since 2026-08-04 — Apply & push to main stays locked until Verify before publish goes green, and the server refuses the publish otherwise (HTTP 409).

Closing the loop: after a successful push, FTW immediately posts the coverage registry to FPD (POST /api/tcs/automations/coverage), so FPD's "Run automated" button knows about the new automation right away — no GitHub Actions run needed. The sync is soft-fail: it never blocks a publish.

Rollback: every publish is recorded (see Publishes on the Run page) and can be undone with one click. Rollback is a git revert --no-edit + push — history keeps both the publish and its revert, never a reset or force-push — followed by the same coverage sync, so FPD ends consistent with the reverted state. If the revert conflicts (later commits touched those files), it is aborted and reported for manual resolution; nothing is forced.

2. Getting started

Prerequisites: Python 3.12+, the full-testing-ai repo checked out with its portable toolchain (TestProjects/.tools/jdk and .tools/maven), and FPD running on port 8000.

cd /c/Users/JulioAI/TestProjects/full-testing-web
.venv/Scripts/python -m uvicorn app.main:app --port 9000

To enable the TC Explorer and every AI feature you need an FPD API token (in FPD: Profile → API tokens, starts with fpd_). Configure it with the FTW_FPD_TOKEN environment variable or in the .env file (see .env.example). Without a token the app still starts, but the TC Explorer is disabled and the AI features cannot reach the LLM.

3. Screen tour

3.1 Dashboard (/)

Environment health panel:

  • Framework: whether the full-testing-ai directory exists and contains coverage/coverage.json (the coverage registry).
  • Banco Americano: whether the app under test answers (GET on the bank login page).
  • FPD API: whether GET /api/tcs answers (an anonymous 401 already counts as "up").
  • Coverage: how many TC keys are in the registry and, with a token configured, how many FPD TCs are automated vs. not automated.
  • Discovered test classes: table of every *Test.java class found in the framework with its @Covers("TC-KEY") annotations (abstract base classes are skipped).

3.2 TC Explorer (/tcs)

Lists the FPD test cases (GET /api/tcs, with server-side q search). Each row shows:

  • Key, title and folder of the TC.
  • Coverage state: green badge Automated → Class#method (the TC key is in the framework's coverage/coverage.json registry as committed on origin/main), amber badge Automated (unpublished) when the coverage entry exists only in the local registry — generated by a local verify run and not pushed to main yet (tooltip: "generated locally, not pushed to main yet") — or amber badge Not automated. The comparison reads git show origin/main:coverage/coverage.json (falling back to HEAD, and to plain local behavior when git fails).
  • Expand: loads the full TC spec (precondition + action → expected steps table) and two AI authoring buttons: Scaffold test and Automate with AI (with a headless checkbox). If the TC is already covered, an amber already covered by ... warning appears to prevent duplicate coverage. When the TC is Automated (unpublished), a Push to main button also appears: it commits and pushes exactly the files generated for that TC (recovered from the persisted automation records — the automation job pages are in-memory and gone after a restart) plus the dirty local coverage registry, and re-syncs coverage to FPD — the recovery path when the push was not done at automation time.

Both buttons are explained in detail in section 4.

3.3 Run (/run)

Launches the framework suite as a background job (mvn -B test).

The form offers:

  • Scope: All tests, Bank UI only (TransferFlowTest + DepositVariationsTest), FPD API only (FpdTcsApiTest), or a single discovered class.
  • Headless browser: windowless browser (passed as FTA_HEADLESS).
  • Auto-heal with AI on failure: automatic self-healing for this run — checked by default; it is the per-run opt-out (see section 5).
  • Device: emulation profile (Desktop, iPhone 13, Pixel 7, iPad; passed as FTA_DEVICE with viewport, user-agent and touch).
  • Base URL: which server to run against (passed as FTA_BASE_URL; defaults to the local FPD).

While running, the job page (/run/{id}) refreshes itself every 2 seconds (htmx) showing the live Maven log. When finished it shows:

  • Summary cards: tests, passed, failures, errors, skipped and time.
  • Per failure: class#method, message, stack trace, inline screenshot and a link to the page HTML captured at failure time (artifacts served from the framework's target/artifacts/).
  • Allure report button: generates (or regenerates) the static Allure site for that run from its stored results and opens it at a permanent per-run URL (/reports/allure/<run-id>/index.html) — every run keeps its own report, not the latest generated one.
  • The self-healing sections (see section 5): the automatic-heal list with its classifications, and the manual Heal ... with AI buttons, one per failed class.

Run pages are persistent: every run is recorded in recordings/runs.json (scope, options, status, summary and self-healing decisions) and its page keeps rendering after an FTW restart (the live Maven log itself is memory-only). A run that was in flight when FTW restarted comes back as interrupted (server restarted mid-run).

Below the form, the Recent runs section lists the latest persisted runs (newest first: start time, scope, status badge, passed/total and any auto-heal classifications), each linking to its run page.

Below the form, the Publishes section lists the pushes to origin main made from this panel (newest first: short sha, source — scaffold / refactor / automate — TC key, message and timestamp). Each entry that is still live offers a Rollback button: git revert --no-edit + push (history keeps both commits — never a reset or force-push) followed by a coverage re-sync to FPD, so the reverted state is consistent everywhere. A revert that conflicts with later commits is aborted and reported as "conflict — resolve manually". Already-rolled-back entries show a gray badge with their revert commit.

Known limitations of this screen:

  • The live Maven log is not retained across restarts: past run pages show the summary, failures and Allure, but not the log.
  • Lists rendered at page load (like the Record & Play recordings list) do not refresh themselves: press F5 to see new entries.

3.4 Record & Play (/record)

Records a manual flow and turns it into raw Playwright Java code:

  1. Pick the TC: an FPD-backed select (with coverage badge) when a token is configured, or a free-text key field otherwise.
  2. System under test: a dropdown fed by the Systems registry (config/apps.json, see §3.5) shown as name — start_url; the recording starts at the app's configured URL. A Custom URL… option reveals the old free-text field as an escape hatch — only http(s) URLs without & or spaces (shell-safety whitelist), and every URL, configured or custom, passes the same whitelist before reaching the command line.
  3. Device: recording viewport (full emulation applies later at run time via FTA_DEVICE).
  4. Start recording: a real browser window opens on this machine (Playwright codegen via mvn exec:java). Perform the flow normally; the code is flushed to the output file as you record.
  5. To finish, close the browser window: it is the clean way and keeps everything recorded. The Stop button kills the session (taskkill /T /F) and may lose the last actions not yet flushed — the UI itself warns about this.

The job page shows the codegen log and the captured Java. When done, the file is saved to recordings/<tc-key>-<yyyymmdd-HHMMSS>.java (a git-ignored folder) and appears in the Saved recordings panel on the right. Recordings are grouped by TC key (parsed from the file name; files whose names don't match the pattern fall under Other), each group a collapsible section showing its recording count and latest saved date, with the newest recordings first inside. Actions per recording: View / Download / Refactor / Delete — Delete is permanent (no trash) and asks for confirmation first. The list is rendered when the page loads: if you finish a recording with the page open, press F5 to see it.

3.5 Systems (/systems)

The systems under test registry — the panel for config/apps.json:

  • List: every configured app with its key, name, start_url and user role names (credentials are never displayed).
  • Add / Edit / Delete: name and start_url are required; the URL must be http(s) and pass the same shell-safety whitelist as Record & Play. The key of a new system is derived from its name (lowercase slug, suffixed if it collides). Edit keeps the app's users block untouched — manage credentials by editing config/apps.json directly.
  • No restart: the file is hot-reloaded, so changes apply immediately. The page writes the file back as canonical 2-space-indented JSON, preserving its structure ({key: {name, start_url, users{role: creds}}}).

This registry is shared: the Record & Play dropdown lists it (§3.4) and Automate with AI reads the same entry for its start URL and login credentials (§4.3).

4. AI authoring

The three authoring features share the same model: the AI generates code following the framework's live conventions, the result is saved outside the repo for review, and only a human publishes it with a button.

4.1 Scaffold test (from a TC)

What it does: runs the framework's TestScaffolder tool (mvn -q test-compile exec:java) to create the skeleton of a test class from the TC key.

Steps: TC Explorer → TC row → Expand → Scaffold test button (it may take a minute: it is Maven).

What gets written where: a new *.java file under the app's tests directory in the framework repo, with @Covers("<key>"), the spec title as javadoc, and a single @Disabled method with one // TODO(step n): <action> -> <expected> comment per TC step. It compiles but does not run until implemented. It never overwrites an existing file.

Decision rules:

  • The dedup check runs first: if the key is already in coverage/coverage.json, nothing is written and an amber ALREADY COVERED by ... box is shown (extend the existing test instead of creating a new one).
  • Conflict (the file already exists): aborts without writing.

Publishing: the result offers a Push to main button that commits and pushes to origin main only the generated file (plus a dirty coverage/coverage.json, if one exists). After the push, FTW syncs the coverage registry to FPD immediately and the result page offers a Rollback button (revert + push, never a reset). Once published, the button renders disabled with an Already pushed (\<short sha>) tooltip (a rollback unlocks it). You can also review it in the repo and delete it if unwanted.

4.2 Refactor with AI (on a recording)

What it does: rewrites raw Playwright codegen Java into the framework's conventions (POM + Flow Objects, Allure, DataFactory) using FPD's LLM.

Steps: Record & Play → finish a recording → Refactor button (on the finished job page or in the saved-recordings list). The refactor page refreshes itself every 2 seconds while the AI works.

How it works: the raw Java is sent to POST {FPD}/api/ai/complete with a prompt enriched with the framework's real conventions: an inventory of existing pages/flows (scanned from bank/pages and bank/flows), the scaffolder's own templates, a real test as style anchor, and the TC's coverage state. The AI answers strict JSON {files: [{kind, class_name, package, code}], notes} which is shape-validated (max 8 files, 60,000 chars per file, valid class and package names).

What gets written where: the validated files are saved to recordings/<base>.refactored/ for review — the framework repo stays untouched.

Verify before publish (the gate): a finished refactor cannot be published directly. The Verify before publish button applies the files into the framework checkout (all-or-nothing, same semantics as publish) and runs the produced test class headless (mvn test -Dtest=<Class>); the page polls itself while the run executes. On green, the job shows a Verified badge with the run summary and the publish button unlocks. On failure, the page shows the failure summary (compilation errors or failed cases with screenshot / page HTML links), nothing is committed or pushed, and the publish button stays locked: fix the code in the framework checkout and hit Re-verify — the re-run tests the checkout as-is, so your manual fixes are what gets verified. The server enforces the same rule: publishing an unverified refactor is refused with HTTP 409.

Publishing: Apply & push to main on a verified job commits the checkout state of the generated files (what was verified green, manual fixes included) — pages and flows live under src/main/java, tests under src/test/java — plus the regenerated coverage/coverage.json, and pushes to origin main. If any target already exists with different content, the whole publish aborts (nothing is overwritten): reconcile by hand in the repo. After the push, FTW immediately syncs the coverage registry to FPD (soft-fail) and the result page offers a Rollback button.

4.3 Automate with AI (natural-language automation)

What it does: turns a manual FPD test case (natural-language steps + expected results) into a deterministic Java test, green and verified, in one click.

Steps: TC Explorer → TC row → Expand → Automate with AI button (headless checkbox on by default). The job page opens, refreshing every 2 seconds, and automatically chains every stage: discovery → generating → applied → verifying ⇄ healing → verified.

Stage 1 — Discovery

An agent drives a real Chromium browser through native Python Playwright (no Node.js: the original plan called for Playwright MCP, but the host has no Node; the technique is the same — aria snapshot + refs — and the owner accepted the deviation after checking that every acceptance failure was in Java codegen, never in browser driving).

On each iteration the loop:

  1. Tags visible interactive elements with data-fta-ref attributes (max 80 refs) and takes an aria snapshot of the page (max 12,000 chars).
  2. Sends to FPD (/api/ai/complete, task generation) the TC spec (steps and expected results), the app config (start URL and users/roles from config/apps.json), the snapshot, the ref table, and the action history.
  3. Receives one strict-JSON action (click | fill | press | navigate | assert_text | assert_visible | done | fail), executes it in the browser, and records the outcome. Errors are fed back into the next prompt, so the loop self-corrects online.

The result is an action log: ordered steps with concrete locator strategies (priority data-testid > role+name > css > text), typed values, and per-step success. It is persisted to recordings/automations/<TC>-<timestamp>.json.

Discovery guardrails: max 40 steps by default (cap 200), 15-minute wall clock, 8 s page timeout, and abort after 3 consecutive LLM call failures (endpoint down or bad token). The model declares done only when it executed every step and verified every expected result; fail when the case cannot be completed.

Stage 2 — Generating (Java conversion)

The action log is converted to Java reusing the Refactor with AI profile plus hardening rules learned from acceptance runs: exact DataFactory/Money signatures, the admin-precondition pattern, substring assertions (never exact-string equality for assert_text), a preference for composing Flow classes, the correct TestConfig package, annotations in the right place (@Covers/@Feature/@Story at class level; @Description/@Test on the method), and no invented assertions beyond what the discovery observed. Complete, compilable files are generated under the POM + Flows conventions (pages in bank/pages, flows in bank/flows, test in bank/tests) and staged in recordings/automations/<stem>.generated/.

Stage 3 — Applied (writing into the framework)

The staged files are written into the framework checkout with an all-or-nothing policy: if any target exists with different content, the stage fails without writing anything and must be reconciled by hand (the UI explains this and offers a pipeline re-run).

Stage 4 — Verifying (and auto-repair)

mvn test -Dtest=<GeneratedClass> runs headless. "Verified" means exactly: the run finished cleanly, the surefire report for that class exists, and it has 0 failures and 0 errors.

On failure, the auto-repair loop kicks in (orange healing badge), up to 2 rounds:

  1. Failure evidence is distilled (javac/surefire [ERROR] lines or failed case details, capped at ~6–8k chars) together with the current file contents.
  2. The LLM proposes corrected files (same class names).
  3. They are re-applied with a strict guard: only files this very job generated may be overwritten.
  4. Headless re-verify. If still red, a second and final round.

Publishing — Push to main

The Push to main button is only enabled at status verified. It is a human decision: it commits only the files the job applied (message NL automation of <TC> into POM/Flows (job <id>), plus the regenerated coverage/coverage.json when the verify run left it dirty) and pushes them to origin main. Right after the push, FTW posts the coverage registry to FPD (soft-fail), so the TC flips to automated in FPD without waiting for a GitHub Actions run, and the publish outcome offers a Rollback button (revert + push, never a reset). From that point the button renders disabled with an Already pushed (\<short sha>) tooltip — FTW records every publish and locks the button against double-pushing; a rollback unlocks it again.

The owner's economic goal: the first pass pays for discovery and generation; every rerun afterwards is plain mvn test — zero AI cost, fully deterministic.

Exact decision rules, summarized:

Question Rule
When does discovery retry an action? When the action fails or the reply is not valid JSON, the error is fed back into the next prompt (each attempt consumes 1 step of the budget).
When does discovery abort? 3 consecutive LLM failures, 40 steps exhausted (cap 200), 15-minute wall clock, or the model declares fail.
How many repair rounds? At most 2 (MAX_REPAIR_ROUNDS), each re-verified.
What is "verified"? A finished mvn run + a surefire report for the generated class with 0 failures and 0 errors.
When does it push to main? Never automatically: only when a human hits Push to main at status verified.
What if apply hits a conflict? All-or-nothing: nothing is written; the human reconciles in the repo and re-runs the pipeline.

5. Self-healing — the full rule set

Self-healing fixes tests that fail on rerun because the application changed (typically a locator), and proposes the fix as a PR for human review. It is the piece with the strictest rules in the module.

5.1 Auto-trigger on ANY failed run

Healing fires automatically when any run finishes with failures (a runner hook), as long as two conditions hold:

  • the master switch FTW_AUTO_HEAL (on by default), and
  • the per-run Auto-heal with AI on failure checkbox in the Run form (checked by default — uncheck it to exclude that run, e.g. in demos).

The Heal ... with AI button on the results page is only the manual path; it is never the only mechanism. The automation pipeline's internal verification runs never trigger healing (they have their own repair loop).

5.2 Failure classifier

Each failed case is classified with the framework's FailureAnalyzer heuristics (same precedence order), using the failure message and the visible text of the captured page HTML:

Category Signature Auto-healed?
LOCATOR_DRIFT TimeoutError / "timeout" / "waiting for ..." in the output (the element no longer appears) Yes — the only eligible category
BEHAVIOR_MISMATCH Assertion failure (AssertionFailedError, expected...) — the app responds but not as the spec says No, never: it is a candidate real bug and gets reported
APP_ERROR The captured page is a 5xx (internal server error, bad gateway, etc.) No, never: an application bug, gets reported
UNKNOWN None of the above Only if it still carries a timeout/"waiting for" signature; otherwise no

Fundamental rule: BEHAVIOR_MISMATCH and APP_ERROR are never auto-healed — that would mask real application bugs. On the results page each failure shows its classification and, when not healable, the note not online-healable — review as a likely app bug.

5.3 Guards and limits

  • Max 2 heal rounds per test class (HEAL_ROUNDS = 2), each verified before the next is attempted.
  • Max 3 auto-heals per run (MAX_AUTO_HEALS_PER_RUN = 3); from the fourth eligible failure on, the entry is noted auto-heal cap reached — use the manual button.
  • One heal job per class+method at a time (duplicates in flight are refused).
  • Per-run opt-out via the form checkbox (default ON) plus the FTW_AUTO_HEAL master switch.

5.4 What a heal job does

  1. Evidence (max 8,000 chars): from the class's newest surefire XML, the stack trace and the locator line that timed out ("waiting for ..."); plus the newest captured HTML snapshot of the failing step (title and visible text of the current page).
  2. Context: current content of the test class and the pages/flows it imports.
  3. LLM (same /api/ai/complete channel, 600 s timeout): HEAL-mode prompt — minimal diff, fix only what the evidence points at (typically a stale locator in a page object), never weaken assertions, rename, or create classes; it may only return corrected versions of the given files. A page-object-only fix is valid.
  4. Apply with the strictest guard: overwrite only existing files directly under bank/tests|pages|flows, all-or-nothing, keeping the previous content of every file.
  5. Headless re-verify (mvn test -Dtest=<Class>). If still red, a second and final round with the new evidence. There is one extra retry within the same round on transport errors (e.g. a read timeout does not burn a round).

5.5 Possible outcomes

  • Healed and verified (green): a heal/<Class>-<timestamp> branch is created, only the healed files are committed and pushed, and the UI shows the compare link to open the PR (/compare/main...<branch>). The merge is always human — nothing is ever committed to main. The checkout returns to main when done.
  • Green but empty diff vs main (the "failure" came from uncommitted local edits, not real drift): no branch is created and the job carries the note healed — no diff vs main, nothing to PR.
  • Failure (out of rounds or error): full revert — every overwritten file is restored to its pre-heal content; the framework tree is never left dirty.
  • If the PR suggestion fails (e.g. git without credentials) but the fix verified, the job stays healed with the error recorded (pr_error) — the fix remains in the working tree.

You can follow any heal live on its page (/heal/{id}, refreshes every 2 s) with attempt details, evidence, changed files, branch and PR link.

5.6 Known operational caveat

Heal LLM calls can hit FPD's ~240 s AI upstream ceiling when the provider is degraded (observed: 28 s even for one-word replies, and 502 on full-file generations), even though the heal channel waits up to 600 s. Typical symptom: heal jobs failing with ReadTimeout/502 on degraded days. It is not an FTW bug. On the backlog: shrinking the heal prompt (mapping the failing locator to the file containing it, so only the test + matching files are sent) and, on the FPD side, a wider AI upstream timeout or a dedicated cheaper/faster model for heals.

6. Configuration reference

All settings are environment variables with defaults matching the local demo layout (see app/config.py). A .env file at the project root is also supported (copy from .env.example; real environment variables take precedence).

Variable Default Meaning
FTW_FRAMEWORK_DIR C:\Users\JulioAI\TestProjects\full-testing-ai Framework repo root (read-only except publish)
FTW_JAVA_HOME C:\Users\JulioAI\TestProjects\.tools\jdk Portable JDK (Temurin 21) for Maven subprocesses
FTW_MVN C:\Users\JulioAI\TestProjects\.tools\maven\bin\mvn.cmd Portable Maven launcher (3.9.16)
FTW_FPD_BASE_URL http://localhost:8000 FPD server (bank app + TC API + LLM gateway)
FTW_FPD_TOKEN (empty) FPD Bearer token; empty = TC Explorer and AI disabled
FTW_PORT 9000 Port for this app
FTW_AUTO_HEAL true Master switch for automatic self-healing

Additional configuration:

  • config/apps.json — the systems-under-test registry: start_url and users/roles with credentials per app. Editable by hand or from the Systems page (§3.5); hot-reloaded, no restart needed. It feeds the Record & Play dropdown and the NL automation alike. Today it only defines banco-americano.
  • Code constants (not environment variables): pipeline repair rounds MAX_REPAIR_ROUNDS = 2 and discovery budgets (40 steps, 15 min) in app/services/automate_service.py; heal rounds HEAL_ROUNDS = 2 and heal LLM timeout HEAL_LLM_TIMEOUT = 600 s in app/services/heal_service.py; auto-heals per run MAX_AUTO_HEALS_PER_RUN = 3 in app/config.py.
  • Portable toolchain: no system Java/Maven needed; FTW passes JAVA_HOME and the portable mvn.cmd to every subprocess, wrapping calls in cmd.exe /c (Windows cannot exec batch files directly) with strict validation of everything that reaches the command line.

7. Troubleshooting & FAQ

The TC Explorer says "FPD token required". Create a token in FPD (Profile → API tokens, starts with fpd_) and start FTW with FTW_FPD_TOKEN=fpd_... or put it in .env.

A run queues but nothing happens. While executing, run jobs are in-memory threads: if FTW restarted mid-run, the job died and its page shows the interrupted status (finished runs do survive the restart with their summary and Allure). Check the FTW log (the uvicorn output; in the current nohup setup, /tmp/ftw-9000.log), that the framework exists at FTW_FRAMEWORK_DIR, and that the portable toolchain is under .tools/. Large Maven runs take a while: the pipeline's internal verification waits up to 15 minutes before declaring a timeout.

Symptoms of AI provider degradation. Slow replies (tens of seconds for any call), 502 from FPD's upstream (~240 s ceiling), discovery aborting with "LLM endpoint failed 3 times in a row", refactors/automations failing at the LLM call, heals dying with ReadTimeout. It is not an FTW bug: wait for the provider to recover and retry (failed pipelines can be re-run from their page).

Where are the logs and artifacts?

What Where
Surefire reports (XML) full-testing-ai/target/surefire-reports/
Failure screenshots and HTML full-testing-ai/target/artifacts/<Class>/
Allure results full-testing-ai/target/allure-results/ (site: target/site/allure-maven-plugin/)
Run registry (Recent runs) full-testing-web/recordings/runs.json
Per-run Allure results + sites full-testing-web/recordings/runs/<run-id>/{results,report}/
Framework triage report full-testing-ai/target/triage/triage-report.md
Recordings full-testing-web/recordings/
Discovery results (action logs) full-testing-web/recordings/automations/*.json
Generated Java (staging) full-testing-web/recordings/automations/<stem>.generated/
Refactor outputs full-testing-web/recordings/<base>.refactored/
Heal staging full-testing-web/recordings/heals/
FTW log uvicorn output (currently /tmp/ftw-9000.log)

How do I keep self-healing out of a demo? Uncheck Auto-heal with AI on failure in the Run form for that run (the per-run opt-out), or start FTW with FTW_AUTO_HEAL=false to turn the master switch off.

An automation failed at the applied stage with a conflict. Someone already created those files in the repo with different content. Reconcile by hand in full-testing-ai (keep the repo version or replace it with the staged one the UI points to) and hit Re-run pipeline on the job page.

The automation generated an odd class name (e.g. ...DiscoveryTest). It is harmless and known (on the polish backlog). The test is valid as is.

Does AI get involved when rerunning already-automated tests? No. Reruns are deterministic mvn test, with zero AI cost. The only exception is self-healing: it fires only if the run fails, only for LOCATOR_DRIFT failures, within the limits of section 5, and its fix reaches main only after human PR review.

Record & Play lost the last actions when stopping. Known: the Stop button kills the process and may lose what was not flushed. Close the browser window to finish cleanly (the file is written as you record).

8. Further reading

  • full-testing-web/README.md — technical architecture, Windows notes, and the read-only/publish philosophy.
  • full-testing-ai/docs/decisions.md — living register of approved decisions and guard rails (the authoritative specification of the rules).
  • full-testing-ai/docs/ai-authoring.md — the authoring loop inside the framework repo (scaffolder, triage, coverage).

Source: docs/user-guide.en.md — rendered at request time.