4 ms·

This website makes it seem like this “public” dataset is for the community to use, but it is instead a for-profit money maker for Google Cloud and you can lose

This website makes it seem like this “public” dataset is for the community to use, but it is instead a for-profit money maker for Google Cloud and you can lose tens of thousands of dollars. Last week I ran a script on BigQuery for historical HTTP Archive data and was billed $14,000 by Google Cloud with zero warning whatsoever, and they won’t remove the fee. This official website should be updated to warn people Google is apparently now hosting this dataset to make money. I don’t think that was the original mission, but that’s what it is today, there’s basically zero customer support, and you can lose $14k in the blink of an eye. Academics, especially grad students, need to be aware of this before they give a credit card number to Google. In fact, I’d caution against using this dataset whatsoever with this new business model attached.

throwaway888abc 3y ago
[flagged]
- httparchive 3y ago
  I used this data when I was a grad student, back when there wasn't a fee for it, so I'm mostly concerned students will get hit with charges that will make it so they can't buy groceries. The website has the Internet Archive logo on it, and it looks like a public resource for researchers, and it used to be free to use. The point of this is for the HTTP Archive to make it clear this is a paid product from Google Cloud, not a "public service".
  - MattGaiser 3y ago
    That is pretty clearly documented in the setup instructions. https://github.com/HTTPArchive/httparchive.org/blob/main/docs/gettingstarted_bigquery.md https://github.com/HTTPArchive/httparchive.org/blob/main/doc... There are multiple notes about cost. In particular, this one stands out. > Note: The size of the tables you query are important because BigQuery is billed based on the number of processed data. There is 1TB of processed data included in the free tier, so running a full scan query on one of the larger tables can easily eat up your quota. This is where it becomes important to design queries that process only the data you wish to explore
    - httparchive 3y ago
      yeah, but don't "eat up your quota" seems rather tame, whereas "you can get billed $14k with no warning" is the truth.
  - catchnear4321 3y ago
    the data is a public service. the platform allowing you to query it is not. you can print at a public library. each page costs a small amount. the printer in that case is the service. and if you print out millions of pages, you may owe hundreds of thousands of dollars. slow down a bit, lest you blow off your other foot.
  - johnnyo 3y ago
    So, you gave someone your credit card number without understanding how or what they were going to charge you for?
    - threeseed 3y ago
      Welcome to the cloud. There are countless companies who specialise in managing cloud costs because of how difficult it is to know when and for what you are going to be charged. Especially for things like data transfer. And by default they don't have a daily spending limit so it's very easy to see a major cost over-run at the beginning.
    - yjftsjthsd-h 3y ago
      You have to give them a credit card in order to use the free tier, and they refuse to implement any features that would let you add safeguards (beyond setting an alert so you can find out after you've already spent the money). Edit: I apologize; they did in fact add something beyond alerts: https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa... ...which is less them implementing a feature and more telling you how to badly implement it yourself. I don't believe this changes the gist of my comment, but it is worth pointing out in the interest of precision. Edit 2: Per https://news.ycombinator.com/item?id=39447499 https://news.ycombinator.com/item?id=39447499 , GCP actually does have a way to cap some resources. It still strikes me as the most "how can we technically claim to be supporting that feature request while still making it as easy as possible to spend more money than you intended to" but there it is.
    - httparchive 3y ago
      [flagged]
  - rvnx 3y ago
    It can be confusing, since the httparchive itself is provided for free by AWS S3 (where AWS is the one footing the bill).
  - deleted 3y ago
    [deleted]
MrDarcy 3y ago
Did the cost estimate calculator provide an inaccurate estimate? https://cloud.google.com/bigquery/docs/best-practices-costs https://cloud.google.com/bigquery/docs/best-practices-costs Estimate query costs BigQuery provides various methods to estimate cost: Use the query dry run option to estimate costs before running a query using the on-demand pricing model. Calculate the number of bytes processed by various types of query. Get the monthly cost based on projected usage by using the Google Cloud Pricing Calculator.
- cornel_io 3y ago
  When I use the BQ interface, it estimates the bytes for each query in real time before I run it, does that turn off if the query is too big? I guess that isn't directly a cost estimate, but if I saw hundreds of TB I'd think twice before hitting "Run"...
judge2020 3y ago
Do you run httparchive, or did you make your username "httparchive" just because it's the subject of your post?
- abeyer 3y ago
  +1 If the latter... I'm not sure that it's explicitly against the rules, but coopting a name of something as your handle just to complain about it is in poor taste and probably should be.
  - SushiHippie 3y ago
    > The worst part is you posting this to hackernews under the username ‘httparchive’ to make it look like it was the httparchive posting this themselves. This was the last comment in TFA, so it seems like they just used it because it was the topic...
    - SushiHippie 3y ago
      Dang changed the username to "nothttparchive" https://news.ycombinator.com/item?id=39451976 https://news.ycombinator.com/item?id=39451976
mike_d 3y ago
> Google is apparently now hosting this dataset to make money Public datasets are hosted for free by Google (Amazon has a similar program) to take the burden off public projects. You didn't pay for the data, you paid for the query you ran against it.
- Jgrubb 3y ago
  Well sure, but how do you query the data they're hosting for free without using google services?
  - anon84873628 3y ago
    Well, sure. But it is convenient to have lots of sample data. Also you get the first TiB per month free in BQ. Also note that anyone can make a dataset available for public use, where they pay the storage and the consumer pays the compute. The official Google datasets are just curated and maintained by Google itself.
gnfargbl 3y ago
The real issue here is that you didn't quite understand what BigQuery was when you pressed the button. What it is, roughly, is a publicly-accessible data supercomputer. If you lost $14k in a blink of the eye, then I would think you consumed at least $4k of Google's actual resources -- maybe $7k. Maybe more. That thing can move some serious data, and you apparently moved around over 2PB. Google bears some significant responsibility for not making the cost transparent to you, it's true. But on the the other hand, don't they bear some significant credit for making such an awesome power available to a lowly peon with a credit card?
- buremba 3y ago
  This happens because Google hides the query cost behind its abstracted "TBs scanned" (for their data format, not even open-source so it's hard to estimate in advance) or even worse "slots" mechanism. Only a fraction of people try to understand how much these slots cost and most of them are the people who got an unexpected bill after using BigQuery and became more aware of how the product works. If GCP would return the query cost in the API and show it directly in the console when you run a query, it would be much easier for their users but unfortunately, it's not Google's interest for obvious reasons.
  - httparchive 3y ago
    Exactly, even after seeing the issue I can't make heads or tails of what the hell a "TBs scanned" is relative to row counts, etc. Likewise, it seems to place a lot of assumptions on knowing what tables include - and on a dataset you didn't build yourself how can you know the tables are optimized to lower your costs? Hell, how can you even know what the costs are?
    - gnfargbl 3y ago
      "TBs scanned" is the number of tebibytes of stored data that the system had to scan to serve your query. This is how BQ is billed, in the on-demand model. The console shows you this number (in very small letters) after you have entered the query but before you press go. In the on-demand billing model, which is what you were using, you can multiply this number by $6.25 to understand your query cost, exactly. It's a design that's hostile to new customers, I agree. But it is comprehensible.
neurostimulant 3y ago
If you're going to make a throwaway account to criticize a website, you shouldn't use that website name as your username. That makes you look like a troll even if you have legitimate complaints.

throwaway888abc 3y ago

[flagged]

httparchive 3y ago
I used this data when I was a grad student, back when there wasn't a fee for it, so I'm mostly concerned students will get hit with charges that will make it so they can't buy groceries. The website has the Internet Archive logo on it, and it looks like a public resource for researchers, and it used to be free to use. The point of this is for the HTTP Archive to make it clear this is a paid product from Google Cloud, not a "public service".
- MattGaiser 3y ago
  That is pretty clearly documented in the setup instructions. https://github.com/HTTPArchive/httparchive.org/blob/main/docs/gettingstarted_bigquery.md https://github.com/HTTPArchive/httparchive.org/blob/main/doc... There are multiple notes about cost. In particular, this one stands out. > Note: The size of the tables you query are important because BigQuery is billed based on the number of processed data. There is 1TB of processed data included in the free tier, so running a full scan query on one of the larger tables can easily eat up your quota. This is where it becomes important to design queries that process only the data you wish to explore
  - httparchive 3y ago
    yeah, but don't "eat up your quota" seems rather tame, whereas "you can get billed $14k with no warning" is the truth.
- catchnear4321 3y ago
  the data is a public service. the platform allowing you to query it is not. you can print at a public library. each page costs a small amount. the printer in that case is the service. and if you print out millions of pages, you may owe hundreds of thousands of dollars. slow down a bit, lest you blow off your other foot.
- johnnyo 3y ago
  So, you gave someone your credit card number without understanding how or what they were going to charge you for?
  - threeseed 3y ago
    Welcome to the cloud. There are countless companies who specialise in managing cloud costs because of how difficult it is to know when and for what you are going to be charged. Especially for things like data transfer. And by default they don't have a daily spending limit so it's very easy to see a major cost over-run at the beginning.
  - yjftsjthsd-h 3y ago
    You have to give them a credit card in order to use the free tier, and they refuse to implement any features that would let you add safeguards (beyond setting an alert so you can find out after you've already spent the money). Edit: I apologize; they did in fact add something beyond alerts: https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa... ...which is less them implementing a feature and more telling you how to badly implement it yourself. I don't believe this changes the gist of my comment, but it is worth pointing out in the interest of precision. Edit 2: Per https://news.ycombinator.com/item?id=39447499 https://news.ycombinator.com/item?id=39447499 , GCP actually does have a way to cap some resources. It still strikes me as the most "how can we technically claim to be supporting that feature request while still making it as easy as possible to spend more money than you intended to" but there it is.
  - httparchive 3y ago
    [flagged]
- rvnx 3y ago
  It can be confusing, since the httparchive itself is provided for free by AWS S3 (where AWS is the one footing the bill).
- deleted 3y ago
  [deleted]

MrDarcy 3y ago

Did the cost estimate calculator provide an inaccurate estimate? https://cloud.google.com/bigquery/docs/best-practices-costs https://cloud.google.com/bigquery/docs/best-practices-costs Estimate query costs BigQuery provides various methods to estimate cost: Use the query dry run option to estimate costs before running a query using the on-demand pricing model. Calculate the number of bytes processed by various types of query. Get the monthly cost based on projected usage by using the Google Cloud Pricing Calculator.

cornel_io 3y ago
When I use the BQ interface, it estimates the bytes for each query in real time before I run it, does that turn off if the query is too big? I guess that isn't directly a cost estimate, but if I saw hundreds of TB I'd think twice before hitting "Run"...

judge2020 3y ago

Do you run httparchive, or did you make your username "httparchive" just because it's the subject of your post?

abeyer 3y ago
+1 If the latter... I'm not sure that it's explicitly against the rules, but coopting a name of something as your handle just to complain about it is in poor taste and probably should be.
- SushiHippie 3y ago
  > The worst part is you posting this to hackernews under the username ‘httparchive’ to make it look like it was the httparchive posting this themselves. This was the last comment in TFA, so it seems like they just used it because it was the topic...
  - SushiHippie 3y ago
    Dang changed the username to "nothttparchive" https://news.ycombinator.com/item?id=39451976 https://news.ycombinator.com/item?id=39451976

mike_d 3y ago

> Google is apparently now hosting this dataset to make money Public datasets are hosted for free by Google (Amazon has a similar program) to take the burden off public projects. You didn't pay for the data, you paid for the query you ran against it.

Jgrubb 3y ago
Well sure, but how do you query the data they're hosting for free without using google services?
- anon84873628 3y ago
  Well, sure. But it is convenient to have lots of sample data. Also you get the first TiB per month free in BQ. Also note that anyone can make a dataset available for public use, where they pay the storage and the consumer pays the compute. The official Google datasets are just curated and maintained by Google itself.

gnfargbl 3y ago

The real issue here is that you didn't quite understand what BigQuery was when you pressed the button. What it is, roughly, is a publicly-accessible data supercomputer. If you lost $14k in a blink of the eye, then I would think you consumed at least $4k of Google's actual resources -- maybe $7k. Maybe more. That thing can move some serious data, and you apparently moved around over 2PB. Google bears some significant responsibility for not making the cost transparent to you, it's true. But on the the other hand, don't they bear some significant credit for making such an awesome power available to a lowly peon with a credit card?

buremba 3y ago
This happens because Google hides the query cost behind its abstracted "TBs scanned" (for their data format, not even open-source so it's hard to estimate in advance) or even worse "slots" mechanism. Only a fraction of people try to understand how much these slots cost and most of them are the people who got an unexpected bill after using BigQuery and became more aware of how the product works. If GCP would return the query cost in the API and show it directly in the console when you run a query, it would be much easier for their users but unfortunately, it's not Google's interest for obvious reasons.
- httparchive 3y ago
  Exactly, even after seeing the issue I can't make heads or tails of what the hell a "TBs scanned" is relative to row counts, etc. Likewise, it seems to place a lot of assumptions on knowing what tables include - and on a dataset you didn't build yourself how can you know the tables are optimized to lower your costs? Hell, how can you even know what the costs are?
  - gnfargbl 3y ago
    "TBs scanned" is the number of tebibytes of stored data that the system had to scan to serve your query. This is how BQ is billed, in the on-demand model. The console shows you this number (in very small letters) after you have entered the query but before you press go. In the on-demand billing model, which is what you were using, you can multiply this number by $6.25 to understand your query cost, exactly. It's a design that's hostile to new customers, I agree. But it is comprehensible.

neurostimulant 3y ago

If you're going to make a throwaway account to criticize a website, you shouldn't use that website name as your username. That makes you look like a troll even if you have legitimate complaints.