3 ms·

Stripe also bears some responsibility here, as they don't support production testing, so it's impossible to have a test suite checking for charge related behavi

Stripe also bears some responsibility here, as they don't support production testing, so it's impossible to have a test suite checking for charge related behaviors in production. If you use stripe, please contact them and request this long overdue feature. (I do not think that is the primary issue, but it does not help)

mickotron 5y ago
Why not test with a stub? Issue would have been picked up in testing even without Stripe integration.
- caspar 5y ago
  Exactly, https://github.com/thoughtbot/fake_stripe https://github.com/thoughtbot/fake_stripe works well enough for this sort of thing.
- rajin444 5y ago
  Yeah, the gem library changes are pretty brutal but this 100% should have been caught by a test.
nightpool 5y ago
For more context, what exactly do you mean by "production testing" and how do you think it would have fixed this issue? I spent 9 months developing a moderately complex Stripe integration, with thorough automated tests, and I never ran into any Stripe issues that I really would have considered a showstopper in terms of testing. Stripe's test environment setup was super super helpful throughout. But this breaking change is on another level of subtle, and it impacts their entire application at a very basic level, but in a way that would be exceedingly hard to detect in almost every case. Frankly, I don't think any reasonable level of testing would have been thorough enough to catch such a subtle issue.
- brian_cloutier 5y ago
  I'm not sure it's that subtle. In the screenshot it looks like a random user was charged each time the method was called. A test for "run the query, notice the user was charged" would have failed. Maybe they got very unlucky and the test randomly picked the correct user, or maybe they ran the test against a database which contained just a single user, but that last part is within their control. > I don't think any reasonable level of testing would have been thorough enough to catch such a subtle issue. If there was a staging environment which tried to match production as closely as possible, and end to end tests of this feature were run, then this bug would have been caught. That doesn't seem like an unreasonable level of care for something as sensitive as billing.
emptysea 5y ago
Why doesn’t test mode work for testing?

mickotron 5y ago

Why not test with a stub? Issue would have been picked up in testing even without Stripe integration.

caspar 5y ago
Exactly, https://github.com/thoughtbot/fake_stripe https://github.com/thoughtbot/fake_stripe works well enough for this sort of thing.
rajin444 5y ago
Yeah, the gem library changes are pretty brutal but this 100% should have been caught by a test.

nightpool 5y ago

For more context, what exactly do you mean by "production testing" and how do you think it would have fixed this issue? I spent 9 months developing a moderately complex Stripe integration, with thorough automated tests, and I never ran into any Stripe issues that I really would have considered a showstopper in terms of testing. Stripe's test environment setup was super super helpful throughout. But this breaking change is on another level of subtle, and it impacts their entire application at a very basic level, but in a way that would be exceedingly hard to detect in almost every case. Frankly, I don't think any reasonable level of testing would have been thorough enough to catch such a subtle issue.

brian_cloutier 5y ago
I'm not sure it's that subtle. In the screenshot it looks like a random user was charged each time the method was called. A test for "run the query, notice the user was charged" would have failed. Maybe they got very unlucky and the test randomly picked the correct user, or maybe they ran the test against a database which contained just a single user, but that last part is within their control. > I don't think any reasonable level of testing would have been thorough enough to catch such a subtle issue. If there was a staging environment which tried to match production as closely as possible, and end to end tests of this feature were run, then this bug would have been caught. That doesn't seem like an unreasonable level of care for something as sensitive as billing.

emptysea 5y ago

Why doesn’t test mode work for testing?