The test that clicks through billing like a person went red, because the upgrade button’s label changed and the old name no longer matched. That kind of test is an end-to-end test: it starts at a page a user would open and it stops only when a result a user would see is on the screen. Playwright is the tool that opens a real browser and clicks the control you named. e2e is a test runner that, on the web, still opens that browser through Playwright. So the e2e vs playwright choice is not two browsers. It is whether you name every click yourself, or you let a model find one brittle click and then check the ending with an exact line of text. Keep the exact check. Add the model only on the click that keeps moving.
Does e2e replace Playwright?
No. On the web, e2e opens the page with Playwright, through a package called the web engine. A locator is the name you give a button, a heading, or a status line so the test can find that same control again. Those names still work, and they do not ask a model to read the page. A model, here, is the part that looks at the screen and chooses the next click. On a one-page billing screen, a test clicked the button whose full name was Upgrade to Pro and checked that the status line said Pro. That test passed in 355ms. The run header still printed a model name, because the config has one. No model was called. The test never asked.
A phone is different. The iOS and Android path uses a different engine, and no simulator was started for this test. If the broken flow is a website, you are still in Playwright’s browser. If the broken flow is a mobile app, e2e is a second install, not a rename of your old file.
Should I use e2e or Playwright?
Use Playwright alone when the button has a stable name and the check is exact. Use an e2e agent step when the path to that check is the part that rots: a wizard, a menu that gets renamed, a dialog that moves between releases. The last line should still be exact. A model is useful on a fuzzy path. It is a bad witness for “the status must say Pro and nothing else,” because a sentence can be almost true.
| Playwright alone | e2e on the web | |
|---|---|---|
| Who finds the control | You name it | You name it, or you describe one goal |
| Which browser | Playwright | Playwright, through the web engine |
| When you pay a model | Never | Only on agent steps |
| What you keep in the repo | Your current specs | Those specs, plus files ending in .e2e.ts |
| What the ending should be | An exact check | Still an exact check |
The last row is the one to remember. If you move the ending into a sentence, you agreed to a bill on every run.
How do I install e2e?
You need Node, then one command in the project:
npx e2e init --yes
npm installIn a fresh folder, init wrote e2e.config.ts, an example test, and a skill folder a coding agent can read. A coding agent is the app that edits the repo with you, such as Claude Code or Codex. A skill is the folder of instructions that app loads when the task calls for it. Init also wrote a small config so that app can start the runner. The first e2e run downloaded a browser. The click tests after that did not.
The wizard pointed the config at a hosted gateway even though the passing tests never called it. A gateway, in this setup, is a front door that forwards the request to a model provider. That default is quiet until you uncomment an agent line. It was left in place on purpose, so the failure below is the one a fresh --yes install hits, not a cleaned-up config.
If the skill folder never shows up inside the coding app, the app is often looking in a different directory. That is the same miss as a skill you copied by hand. Are Matt Pocock’s skills worth it is about what belongs in that folder. How Claude Code works is the loop that loads it. Init cannot fix a folder the app does not read.
How do I write an e2e test without a model?
Open the page, click by role and name, then assert the exact text. Role means the kind of control it is, button or status, not the color it is painted. Name means the words a person can already see. Here is the test that passed:
import { test } from '@e2e-dev/web';
import { expect } from 'e2e';
test('upgrade button changes the plan', async ({ app, screen }) => {
await app.open('/');
await screen.getByRole('button', { name: 'Upgrade to Pro' }).click();
await expect(screen.getByRole('status')).toHaveText('Pro');
});app is how you open a URL. screen is the page, the stand-in for page in a Playwright test. expect is the check. None of those three ask a model anything. The example file from init even imports test from the web package. You can live in this shape for a long time, and the suite will feel like the Playwright you already know.
Point the run at your app with an environment variable:
APP_URL=http://127.0.0.1:8765 npx e2e run tests/billing.e2e.tsThe test page was a static file with one button and one status line. Yours can be the dev server. In e2e.config.ts the address sits on app.url. If you would rather the runner start the server, that is app.command. Setting the variable in the shell and leaving the command commented out was enough for a page that was already up.
Why did getByText match nothing?
Because the status said Free plan, and a text query here is the whole string, with the same capital letters. The query asked for Free. The runner looked for about five seconds, then failed: match count 0. Playwright often treats a fragment as a hit. A line that was green there can go red here for that reason alone, with the app unchanged. Match the full words, or pass { exact: false } when you truly want a piece of the label.
That wait is the part that wastes a morning. A wrong name does not fail on the first look. It keeps looking, then tells you the count was zero. When a moved spec goes red, read the locator text before you blame the page. The button test used the full name Upgrade to Pro, and the status check asked for the whole word Pro. Both of those were exact, and both were fast. The fragment was the only one that sat there.
The Playwright migration notes list the other mismatches worth checking in the same pass. getByTitle is not there. A role query does not see hidden nodes. A CSS locator chained inside another locator is not the way through screen. None of those three came up on this tiny page. The whole-string rule was enough to break a query most people would write without thinking.
When does e2e skip the model?
Only for agent.act, and only after a later check has already passed once. agent.act is the line where you describe one goal in a sentence, such as “add one item to the cart,” and the model does the clicks. The replay cache is the saved list of those clicks. Next time, the runner repeats them and does not call the model, if the screen still matches. If a control is gone or the ending does not show up, the model takes over from the screen it is on.
There was no free replay on the first run, because no saved pass existed yet. A recording is written only when a later check passes. expect on a locator counts. expect on a plain value does not. Another agent.act does not count as the check for the one before it. A step that changes nothing the runner can see is not saved. Follow the act with the outcome, or you pay again and you store nothing.
The cache directory is gitignored unless you opt in. A fresh checkout misses, calls the model, and writes a recording only after a check passes. Say that before anyone puts this in CI. Changing the test name, the instruction, or which agent is configured also misses. Changing the model does not. A value that changes every run, a time or a fresh email, has to be marked or the replay looks for the old string. The rules live on the cache page. The project itself is tester-army/e2e.
Does agent.assert call the model every time?
Yes. agent.assert, agent.waitFor, and agent.extract always run live. Assert means “look at the screen and decide if this sentence is true.” Extract means “read a value off the screen.” Wait-for means “keep looking until the sentence is true,” and it can skip calls while the screen is sitting still, but it is still a model step. None of them are the replay. A verdict written as a sentence is a bill on every run. A verdict written as expect(screen.getByRole('status')).toHaveText('Pro') is not.
The other half shows up with no key. With the wizard’s gateway still in the config, agent.act('click the upgrade button') failed in a few hundred milliseconds. The message was an unauthenticated request, and it said to set AI_GATEWAY_API_KEY. The summary said the cache missed. There is no quiet fallback to a recorded click when the model cannot be reached. If your init picked a different provider, the missing variable will be that provider’s key, not this one. The shape of the failure is the same: the step stops, and nothing is replayed.
If you want a suite with no key at all, do not call agent. Init can be pointed at no model. The --yes run was not. It wrote the gateway and left the agent example commented out until a key is in the environment. Leave it commented until one flow has earned a sentence.
Is e2e worth it?
Worth it for one or two flows where the click path changes and the ending is still an exact line of text. Not worth it as a rewrite of a suite that is already fast and already stable. You would add a model, a cache you have to remember to keep, and a stricter text match, and you would still be on Playwright for the browser.
A hosted TesterArmy run, the paid product that executes tests for you, is a different choice from this command-line runner. The 355ms result is from the runner on a local page, with no account, so do not read a cloud price into it. Same family of names. Different bill. If a teammate forwards you a pricing table, ask which of the two they mean before you argue.
Maestro and other natural-language runners also let you describe a flow. They were not run side by side here, so there is no ranking. The reason to pick this one, if you pick it, is that the exact check stays in code and a paid click is allowed to become a recording. If you need a pixel diff or the HTML report you already have, leave those specs in Playwright. Both runners can sit in one repo. Old files stay. New files end in .e2e.ts. Move one painful wizard, keep the original until the new file is green, and do not migrate a test that never opened a browser.
If a coding agent is going to write and run these tests, give it the skill init created, and do not let it run the suite on the machine that holds your keys. coop is the version of that worry to start from: a disposable machine, the project copied in, the host left outside. The loop inside agent.act, read the screen and then ask for a click, is the same shape as a coding agent loop, pointed at a browser instead of your repo. Step Code versus Claude Code is the reminder that the permissions around that loop matter as much as the model inside it. A test runner that can click is still a program that can click.
Leave the suite on Playwright. Add one e2e file for the upgrade path, with the model only on the click and the word Pro in an expect. If that recording still misses every morning, the screen is not stable enough for a cache, and the honest move is to name the button.
Common questions about e2e vs Playwright
Does e2e work without an API key?
Yes, when the test never calls agent. The upgrade test did that, and it passed. An agent line, with the gateway init wrote, fails until AI_GATEWAY_API_KEY is set or you point the config at a provider you actually have. The header can print a model name on a run that never calls one. Do not read that line as a charge.
Can I use e2e and Playwright in the same repo?
Yes. Different config files, different file names. The migration notes tell you to keep playwright.config.ts if you have one, and to keep @playwright/test installed. The web engine brings its own browser driver, so the two runners do not share one config object. That separation is what makes a slow move possible: one file first, the rest when they hurt.
Does the replay cache work if I do not commit it?
No. Init gitignores .e2e/cache/. A new machine misses, calls the model, and writes a new recording only after a check passes. Commit the cache only once you know a recording is a list of controls and clicks, not a video, and that it still describes your app. Watch one miss and see why the handoff happened before you share a recording.
Why does getByText fail in e2e when it passed in Playwright?
A text query in e2e matches the whole string, with the same capital letters, where Playwright often accepts a fragment. Asking for Free on a status that says Free plan waits about five seconds and fails with a match count of 0. Use the full words, or pass { exact: false }.
Is the e2e CLI the same product as the hosted runner?
No. The command-line runner is the open source framework. The company also sells a hosted TesterArmy runner under the same family of names, with its own pricing. The 355ms locator test, the five-second miss on the word Free, and the unauthenticated agent step all come from the open source runner.