4 ms·

Just accidentally spent $300 on a single BigQuery query

39 points by susam 4y ago

gnfargbl 4y ago
Yup. If you're a regular BQ user and you have VC funding, buy reserved slots. If you're a regular BQ user and you're bootstrapped, then get into the habit of checking the top right of the console screen before you press the query button. I have my own Japanese train conductor ritual, in which I verbalise to myself "The screen says 3TB will be scanned. This will cost me $15. That is as expected." After a couple of seconds pause, I press the button.
- bushbaba 4y ago
  This is one reason Eng leaders like flat rate cost infra. You tell your team I’ve allocated N$ of compute for analytics. And if there’s slowdowns from too much traffic the team will reschedule their work around it. Where-as BQ and its “infinite” scale encourages more and more work. Up until the bill far outweighs the benefit it brought the organization. Sure you can design around this. But it’s great things like BQ flat rate exist as a middle ground.
- coatue 4y ago
  Sounds a bit like dancing with a cobra but in a careful way. For transparency, I'm hydra.so cofounder
sitkack 4y ago
By design, won't fix.
Havoc 4y ago
All three major clouds have regular threads on Reddit of people shooting themselves in the foot. Clouds are complicated and learning involves mistakes. It really needs an ability to cap it while experimenting in a sandbox or something. not just budget alerts that helpfully notify you of said foot amputation post fact
- deleted 4y ago
  [deleted]
- ericpauley 4y ago
  This gets brought up a fair bit. A major issue is that hard bill caps cause not just an availability concern, but also a durability risk (data loss). From the cloud provider’s perspective, it’s probably easier to save face by comping accidental overages than risk deleting user data. One solution here would be to identify SKUs that are not durability-critical, such as just instance runtime (though even this has caveats), and enforce caps by killing these.
  - Havoc 4y ago
    It's definitely a tricky question once you get to proper production context & I think their stance there is overall reasonable given the thorny questions. However I don't see why they can't provide a sandbox mode with associated terms. They delete user data & impose restrictions all the time with trial accounts after all. If they want to get people to try new parts of the million cloud products providing a safer mode to experiment would likely do a lot to encourage that. Hell I mostly steer clear from learning BigQuery precisely because I've observed it seems to feature disproportionately frequently in posts like these. If I could designate a section of the account as testing/non-prod and limit it to $100 I'd definitely explore more freely. >easier to save face by comping accidental overages than risk deleting user data. With the added benefit that you get to keep the money from the likely substantial group that writes off minor mistakes as school fees / not worth the hassle. Probably immaterial but worth keeping in mind that comp cost flow through 100% to bottom line profit/loss. Granted all the clouds seem very generous with their comp'ing accidents, but still feels like a very intentional power dynamic - rather than giving the user control the clouds reserve unilateral power to decide the users fate. The fact that they're magnanimous doesn't change that.
mejutoco 4y ago
Unnest is the easier way to multiply the cost in Bigquery.
killingtime74 4y ago
if you want fixed cost there are many options...Spark, Clickhouse, Presto.

39 points by susam 4y ago

Yup. If you're a regular BQ user and you have VC funding, buy reserved slots. If you're a regular BQ user and you're bootstrapped, then get into the habit of checking the top right of the console screen before you press the query button. I have my own Japanese train conductor ritual, in which I verbalise to myself "The screen says 3TB will be scanned. This will cost me $15. That is as expected." After a couple of seconds pause, I press the button.

bushbaba 4y ago
This is one reason Eng leaders like flat rate cost infra. You tell your team I’ve allocated N$ of compute for analytics. And if there’s slowdowns from too much traffic the team will reschedule their work around it. Where-as BQ and its “infinite” scale encourages more and more work. Up until the bill far outweighs the benefit it brought the organization. Sure you can design around this. But it’s great things like BQ flat rate exist as a middle ground.
coatue 4y ago
Sounds a bit like dancing with a cobra but in a careful way. For transparency, I'm hydra.so cofounder

sitkack 4y ago

By design, won't fix.

Havoc 4y ago

All three major clouds have regular threads on Reddit of people shooting themselves in the foot. Clouds are complicated and learning involves mistakes. It really needs an ability to cap it while experimenting in a sandbox or something. not just budget alerts that helpfully notify you of said foot amputation post fact

deleted 4y ago
[deleted]
ericpauley 4y ago
This gets brought up a fair bit. A major issue is that hard bill caps cause not just an availability concern, but also a durability risk (data loss). From the cloud provider’s perspective, it’s probably easier to save face by comping accidental overages than risk deleting user data. One solution here would be to identify SKUs that are not durability-critical, such as just instance runtime (though even this has caveats), and enforce caps by killing these.
- Havoc 4y ago
  It's definitely a tricky question once you get to proper production context & I think their stance there is overall reasonable given the thorny questions. However I don't see why they can't provide a sandbox mode with associated terms. They delete user data & impose restrictions all the time with trial accounts after all. If they want to get people to try new parts of the million cloud products providing a safer mode to experiment would likely do a lot to encourage that. Hell I mostly steer clear from learning BigQuery precisely because I've observed it seems to feature disproportionately frequently in posts like these. If I could designate a section of the account as testing/non-prod and limit it to $100 I'd definitely explore more freely. >easier to save face by comping accidental overages than risk deleting user data. With the added benefit that you get to keep the money from the likely substantial group that writes off minor mistakes as school fees / not worth the hassle. Probably immaterial but worth keeping in mind that comp cost flow through 100% to bottom line profit/loss. Granted all the clouds seem very generous with their comp'ing accidents, but still feels like a very intentional power dynamic - rather than giving the user control the clouds reserve unilateral power to decide the users fate. The fact that they're magnanimous doesn't change that.

mejutoco 4y ago

Unnest is the easier way to multiply the cost in Bigquery.

killingtime74 4y ago

if you want fixed cost there are many options...Spark, Clickhouse, Presto.