4 ms·

Speaking from experience, the promise of completing tests in 1 minute or less is very appealing. This is one thing I don't want to compromise. In the case of t

Speaking from experience, the promise of completing tests in 1 minute or less is very appealing. This is one thing I don't want to compromise. In the case of the two companies that are helping me to test this service, they both pay a fixed fee of USD 2000 for an agreed maximum parallelization of 50 browsers. This means that up to 50 concurrent tests, I can still deliver the promise of 1 minute tests. For context, BrowserStack, which they used before, is charging USD 999 for 5 parallel browsers. I believe BrowserStack has this pricing model because they assume long running tests. It is a lot harder to share VMs if tests take 10-15 minutes than if they take 1 minute. I could have pricing as USD 999 for 50 browser parallelization, and then charge extra if either of their tests take longer than 1 minute to execute. This model would ensure that we don't have one client hogging all of the resources.

Silhouette 4y ago
Speaking from experience, the promise of completing tests in 1 minute or less is very appealing. Indeed. There is a qualitative difference between tests completing fast enough that a dev is OK with just waiting a few seconds for the results and tests becoming a "go do something else for a while" background activity. There is also a qualitative difference between tests running fast enough that if something does go wrong the code can often be backed out before it interferes with anyone else and tests that run slowly enough that the CI system can't keep up with merges from the team and changes start piling up that then need sorting out somehow every time a set of tests does fail.
- punkpeye 4y ago
  Another thing to consider is that most engineers do not run all integration tests locally before committing them, simply because they take long time, i.e. A typical workflow is: commit changes -> raise a PR -> wait for all tests to complete -> [if tests fails, go back to step 1] This hugely impacts developer ability to switch context, because they need to stash code, switch branches, etc. If all integration tests could complete in pre-push commit hook (which is reasonable if they take less than a minute), then we would dramatically reduce the context switching involved in day to day engineering.
  - Silhouette 4y ago
    I agree. That's another reasonable point. If I were in your position I would absolutely be emphasizing these kinds of qualitative benefits in the marketing even if your pricing model is actually very quantitative in the end. Anyone who runs the kind of engineering group you're currently aiming at won't have to think very hard to get an idea of the quantitative scale of the benefits they might enjoy as a result. Ideally you want authorising a few (or more than a few) thousand bucks a month to be a quick and obvious decision for them. "We provide a simple tool that enables radically better development processes with likely ROI between 10x and 100x" is a decent elevator pitch to people like VP Engineering and CFO who are going to end up making a purchasing decision.

Silhouette 4y ago

Speaking from experience, the promise of completing tests in 1 minute or less is very appealing. Indeed. There is a qualitative difference between tests completing fast enough that a dev is OK with just waiting a few seconds for the results and tests becoming a "go do something else for a while" background activity. There is also a qualitative difference between tests running fast enough that if something does go wrong the code can often be backed out before it interferes with anyone else and tests that run slowly enough that the CI system can't keep up with merges from the team and changes start piling up that then need sorting out somehow every time a set of tests does fail.

punkpeye 4y ago
Another thing to consider is that most engineers do not run all integration tests locally before committing them, simply because they take long time, i.e. A typical workflow is: commit changes -> raise a PR -> wait for all tests to complete -> [if tests fails, go back to step 1] This hugely impacts developer ability to switch context, because they need to stash code, switch branches, etc. If all integration tests could complete in pre-push commit hook (which is reasonable if they take less than a minute), then we would dramatically reduce the context switching involved in day to day engineering.
- Silhouette 4y ago
  I agree. That's another reasonable point. If I were in your position I would absolutely be emphasizing these kinds of qualitative benefits in the marketing even if your pricing model is actually very quantitative in the end. Anyone who runs the kind of engineering group you're currently aiming at won't have to think very hard to get an idea of the quantitative scale of the benefits they might enjoy as a result. Ideally you want authorising a few (or more than a few) thousand bucks a month to be a quick and obvious decision for them. "We provide a simple tool that enables radically better development processes with likely ROI between 10x and 100x" is a decent elevator pitch to people like VP Engineering and CFO who are going to end up making a purchasing decision.