4 ms·
BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.
by deepsun 3mo ago
BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.
- dpoloncsak 3mo agoThe crawlers would have still just hammered their site though, right?
- pavel_lishin 3mo agoI think the idea is that they could store the data in BigQuery, and point users of the site there.
- PunchyHamster 3mo agoThe crawlers would have still just hammered their site though, right?
- 40four 3mo agoWhy is your comment exactly word for word of another comment just one level above in the comment chain?
- taneq 3mo agoBecause the crawlers would still have hammered their site, though. (The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)
- mijowi 3mo agoMaybe because they restated what they said instead of addressing to the previous commenter’s point.
- 40four 3mo agoThe comment I’m speaking of was from a different user, and it’s giving a strong smell that both users are bots unfortunately. Maybe I’m wrong, maybe it’s a coincidence those exact strings of characters were formed in isolation from each other in the same thread. I hope I’m wrong, but it’s a red flag
- dpoloncsak 3mo agoI'm not a bot, and the other commenter was restating the question as the solution provided did not seem to actually solve the problem, just redesign the entire project (See comment from u/taneq). We didn't form "those exact strings of characters in isolation from each other in the same thread" they are directly referencing my comment in a reply thread to said comment.
- PunchyHamster 3mo agoBecause I am pointing out that the same exact thing would still happen even when commenter's "fix" would be applied. You're bad at understanding context. Word for word was just me poking fun at that comment
- dpoloncsak 3mo agoSure, but now you're moving the site from "Free data presented in a pleasant way to view" to a "pay-as-you-go database". Your audience shifts dramatically, and you lose the ability to share the data you're trying to present.
- pavel_lishin 3mo agoTrue. But it sounds like they already lost that.
- deepsun 3mo agoNo, it's free for everyone for the data sizes they have: Free tier: 10 GB of active storage and 1 TiB of query data processed per month.
- tekne 3mo agoThere is also the deep magic... https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs
- primitivesuave 3mo agoI'm curious how this compares to just using DuckDB in the browser? https://duckdb.org/2021/10/29/duckdb-wasm https://duckdb.org/2021/10/29/duckdb-wasm
- tekne 2mo agoDifferent pros and cons -- SQLite is smaller is the big one
- primitivesuave 3mo agoYes, if I wanted to put a nice HTML interface over the query results then I would still end up with the same problem where some combinatoric explosion of query parameters to the `/search` endpoint, most of which are cache misses, leads to many many TBs of network egress.
- lokar 3mo agoI think snowflake has similar, you can rent it out or make it free
- primitivesuave 3mo agoThanks for that tip, this is exactly how I would do this if I had to do it from scratch. Just in case it's useful for anyone else: https://docs.cloud.google.com/bigquery/public-data https://docs.cloud.google.com/bigquery/public-data