3 ms·

test minute = test * minute, i.e. 50 tests = USD 1 However, based on everything I've read in this thread, I am leaning towards max parallelization based model,

test minute = test * minute, i.e. 50 tests = USD 1 However, based on everything I've read in this thread, I am leaning towards max parallelization based model, e.g. maximum 50, maximum 100, etc, and charge extra for tests that take longer than 1 minute. This allows straightforward sales while providing most of the value to an avg. client.

OJFord 4y ago
I understand the meaning of a 'test-minute', but in OP you said however many tests, it ran under a minute. Not a minute per test. So if that's the case then fine. (But also now there are 'tests that take longer than 1 minute'? It's gone from wow to not really clear what you're offering beyond managed test-running (that's otherwise equivalent to self-hosted) to be honest.) Edit: do you mean they're parallelised, so they all complete in however long the longest one takes (which in OP is <1m but in comments might be more) and you'll charge for #concurrent jobs?
- punkpeye 4y ago
  Yeah, you've got it in the edit. Using Cypress you are going to have specs and specs will have tests. You can parallelize Cypress per spec. Meaning that if you had 50 specs and 50 VMs, then your entire test suite will take at most what your longest spec takes. Most companies are not going to have 50 VMs ready to run tests. However, even if you did, you still have the problem that your tests will run for as long as it takes to run the longest spec, which can be a lot. I've seen companies try to diminish the latter issue by splitting specs into multiple files. However, that will get you only so far. So what I do is... I read all your specs, extract individual tests, and spin up VM for each test. If all your individual tests complete in under a minute, then you will wait at most 1 minute. However, if there are individual tests that take longer, than yes – you will wait for the longest test to complete.

OJFord 4y ago

I understand the meaning of a 'test-minute', but in OP you said however many tests, it ran under a minute. Not a minute per test. So if that's the case then fine. (But also now there are 'tests that take longer than 1 minute'? It's gone from wow to not really clear what you're offering beyond managed test-running (that's otherwise equivalent to self-hosted) to be honest.) Edit: do you mean they're parallelised, so they all complete in however long the longest one takes (which in OP is <1m but in comments might be more) and you'll charge for #concurrent jobs?

punkpeye 4y ago
Yeah, you've got it in the edit. Using Cypress you are going to have specs and specs will have tests. You can parallelize Cypress per spec. Meaning that if you had 50 specs and 50 VMs, then your entire test suite will take at most what your longest spec takes. Most companies are not going to have 50 VMs ready to run tests. However, even if you did, you still have the problem that your tests will run for as long as it takes to run the longest spec, which can be a lot. I've seen companies try to diminish the latter issue by splitting specs into multiple files. However, that will get you only so far. So what I do is... I read all your specs, extract individual tests, and spin up VM for each test. If all your individual tests complete in under a minute, then you will wait at most 1 minute. However, if there are individual tests that take longer, than yes – you will wait for the longest test to complete.