4 ms·

Obligatory alternative tool plug / comparison: I work for https://replay.io https://replay.io , which is a true time-traveling debugger for JS. We have forks

Obligatory alternative tool plug / comparison: I work for https://replay.io https://replay.io , which is a true time-traveling debugger for JS. We have forks of Firefox, Chrome, and Node, instrumented to capture syscalls at the OS level. Those recordings are uploaded to the cloud, and devs can then use our web client (effectively the Firefox DevTools as an app + a bunch of new features) to debug the recording at _any_ point in time, including adding print statements after the fact that show values every time a line was hit, step debugging, network requests, React + Redux DevTools integration, DOM inspection, and more. Currently, our main usage is manually recorded replays, but we're actually working on similar test integration features as well. We can record Playwright and Cypress tests, upload recordings of test runs, show results per test in our dashboard, and let you debug the full recordings of each successful and failed test (code, DOM, network, console messages, errors, etc). The test suite feature is early closed beta atm - we've been dogfooding it ourselves and it's _really_ helpful! Based on your description + a quick glance at your home page, sounds like we're addressing the same use case in similar ways, with some differences in the underlying recording technology and implementations

mikeshi42 4y ago
Great to see you guys here, I've heard a few mentions of your team's e2e integration in the works for a bit! We indeed do approach from a different direction, having been built around the e2e testing use case from the start, we've focused on integrating with existing CI setups without swapping browsers (some teams really love the bundled Electron runner!), as well as knowing exactly what actions are running in your test runner relative to the replay in a likely familiar UI (answering questions like what DOM element did Cypress match for this command? What was the DOM like exactly when Cypress started running that command? Which command executed before this network request?). Since you're from the team - one thing I've always wondered is how the pricing will work out for CI test runs? From the current public pricing I've found, it looks like it works great for developers recording test runs manually, but is extremely cost prohibitive at any scale if you're running it continually in CI.
- acemarke 4y ago
  Nice, yeah, I can see the focus on "what actions" being useful. I'm an engineer, not GTM, so I'm not sure how the test runs feature plays into the listed "X recordings per month" at https://www.replay.io/pricing https://www.replay.io/pricing . Agreed that there's a distinct difference between engineers/QA making manual recordings, and CI cranking them out - right now we've got 52 E2E tests that run on every PR, often multiple times due to re-pushes, and each of those tests generates a recording per run. So, obviously that burns through real fast :) If I had to guess we'd probably distinguish between those two use cases. I've tagged our main GTM person in case he wants to respond here.
  - mikeshi42 4y ago
    Hahaha yup! We get a ton of volume on our platform from teams churning out test runs per-commit in their CI, where debugability is most important. Glad to hear from others in the space! Hope to learn more if your team's GTM person jumps on :)
- jasonlaster11 4y ago
  Congrats on launching! Using Session Replay in the CI space makes a lot of sense. And agreed that there’s a lot that can be done by hooking into Cypress’s events. By the way, we’re doing the same and hope to show a Cypress reporter soon as well. Re pricing: we’re still refining the pricing, but we assume that most of the time you’ll only want to debug the failing tests and while the recordings are fairly small, the larger piece is actually replaying the browser as it ran before so you can add print statements in your application and play to when a network request was returned or an error was thrown.
  - mikeshi42 4y ago
    Thanks Jason, can't wait to see what your team's been working on the Cypress end soon then!

mikeshi42 4y ago

Great to see you guys here, I've heard a few mentions of your team's e2e integration in the works for a bit! We indeed do approach from a different direction, having been built around the e2e testing use case from the start, we've focused on integrating with existing CI setups without swapping browsers (some teams really love the bundled Electron runner!), as well as knowing exactly what actions are running in your test runner relative to the replay in a likely familiar UI (answering questions like what DOM element did Cypress match for this command? What was the DOM like exactly when Cypress started running that command? Which command executed before this network request?). Since you're from the team - one thing I've always wondered is how the pricing will work out for CI test runs? From the current public pricing I've found, it looks like it works great for developers recording test runs manually, but is extremely cost prohibitive at any scale if you're running it continually in CI.

acemarke 4y ago
Nice, yeah, I can see the focus on "what actions" being useful. I'm an engineer, not GTM, so I'm not sure how the test runs feature plays into the listed "X recordings per month" at https://www.replay.io/pricing https://www.replay.io/pricing . Agreed that there's a distinct difference between engineers/QA making manual recordings, and CI cranking them out - right now we've got 52 E2E tests that run on every PR, often multiple times due to re-pushes, and each of those tests generates a recording per run. So, obviously that burns through real fast :) If I had to guess we'd probably distinguish between those two use cases. I've tagged our main GTM person in case he wants to respond here.
- mikeshi42 4y ago
  Hahaha yup! We get a ton of volume on our platform from teams churning out test runs per-commit in their CI, where debugability is most important. Glad to hear from others in the space! Hope to learn more if your team's GTM person jumps on :)
jasonlaster11 4y ago
Congrats on launching! Using Session Replay in the CI space makes a lot of sense. And agreed that there’s a lot that can be done by hooking into Cypress’s events. By the way, we’re doing the same and hope to show a Cypress reporter soon as well. Re pricing: we’re still refining the pricing, but we assume that most of the time you’ll only want to debug the failing tests and while the recordings are fairly small, the larger piece is actually replaying the browser as it ran before so you can add print statements in your application and play to when a network request was returned or an error was thrown.
- mikeshi42 4y ago
  Thanks Jason, can't wait to see what your team's been working on the Cypress end soon then!