A purchasing manager in a German manufacturing company approves an order in a supplier portal. The updated interface looks good, the release pipeline is green, and the delivery team has introduced an AI agent that maintains browser tests. Yet the order went through without the second approval required above the customer's spending limit. The agent repaired a failing test by following the new screen. Nobody checked whether the new screen was right.
AI-assisted test automation is useful when it accelerates evidence collection while leaving business acceptance rules under explicit human ownership. For German B2B software teams, that distinction belongs in the development brief. Otherwise, an impressive test count can conceal the exact defects that matter to customers.
The new capability is real, but its job needs boundaries
Playwright's test-agent documentation describes a planner that explores an application, a generator that creates executable tests and a healer that investigates failing tests. These are useful building blocks for a test-maintenance service. They do not establish what a customer's approval policy, export entitlement or billing rule ought to be.
Make that boundary visible in the repository. Keep a short set of protected acceptance statements beside the tests: which role may approve, what must happen after a rejected payment, which tenant may export a report. A generated change that weakens one of those assertions should require a named reviewer. Updating a locator after a button moves is a different decision from accepting a missing approval step.
The first purchase should therefore be a bounded coverage project. Select one revenue-critical journey and define the failure conditions before giving the agent access. A supplier who begins by promising hundreds of generated tests may be measuring the easiest thing to produce.
Give the agent a realistic, isolated customer world
A B2B application rarely behaves like a single-user demo. Create two unrelated customer organisations, several roles, at least one restricted account and a realistic sequence of existing transactions. Use synthetic or appropriately prepared test data. Keep production customer records, real payment methods and live notification channels outside the test environment.
Define what the seed environment promises. It should be possible to rebuild the same starting state, including a predictable clock where time matters. An invoice that becomes overdue during a long test run should not make the suite randomly fail. Conversely, a fixed clock must not hide a genuine expiry rule that the test is supposed to exercise.
Include localisation in the chosen journey. A German decimal separator, a long legal company name or a translated validation message can change how the interface behaves. The purpose is to test meaningful outcomes across supported language settings, rather than asserting every piece of translated punctuation.
Make failures informative enough to review
When a test fails, reviewers need the action, expected business outcome and observed state. Playwright Trace Viewer can expose the recorded actions and page snapshots associated with a run. Integrate that evidence into a review process with controlled access and a retention period appropriate to the test data.
Ask the supplier to distinguish three outcomes. A selector may be outdated while the feature still works. Test data may no longer satisfy a prerequisite. Or the product may have regressed. Those cases require different changes. A healer that silently skips a difficult test has not repaired coverage; the skipped scenario must remain visible as an unresolved acceptance item.
Record the original failure and the proposed repair together. If a test previously expected two approvers and now expects one, the review should show that semantic change immediately. If only a label or locator changed, the evidence should support that narrower explanation. Maintain the previous test until the change is accepted.
Challenge the suite with deliberate defects
A useful acceptance exercise is to introduce a small, controlled defect into the isolated environment and see whether the relevant test rejects it. Remove a permission check, change a calculation or suppress an expected status transition. Restore the correct implementation after the exercise. This measures whether the suite protects the agreed rule instead of merely completing a happy path.
For the purchasing portal, test these concrete cases:
- ▸An order just below and just above the approval threshold.
- ▸A user attempting to approve their own restricted purchase.
- ▸A direct request for another customer's order.
- ▸A repeated click while the first approval is still processing.
- ▸An integration timeout after the order has already been accepted.
- ▸A healed test whose proposed patch removes an important assertion.
The quote should include the time to design these cases, not just time to install a framework. Buyers also need an explicit decision about which tests run for every change and which require a longer scheduled environment.
Buy maintainability as part of the delivery
Ask who owns the fixtures, how an engineer can run one failing case locally, and what happens when the AI provider is unavailable. Ordinary engineers should still be able to read and maintain the resulting tests. The model configuration, prompts and generated patches are supporting assets; the accepted business behaviour is the durable part of the investment.
Measure useful outcomes during the pilot: time to diagnose a failure, defects detected before release, unexplained skips and reviewer effort per repair. Compare those observations with the existing process. A faster green pipeline is beneficial only if the same important checks remain in place.
Our guide to secure CI/CD delivery covers the surrounding release controls, while software handover and support explains what the receiving team should be able to operate. Selected engineering work is available in Anis Khalef's portfolio.
TuniCyberLabs builds custom software and delivery tooling. Bring us one important user journey and its current tests. We can scope an AI-assisted testing pilot around the mistakes your business needs to catch.