vgt

15 ms·

1. 121.▲
   by vgt 9y ago

   And the first terabyte of analysis for free in BigQuery!! (Work at g)
2. 122.▲
   NCAA Basketball Public Dataset(pantheon.corp.google.com)

   1 points by vgt 9y ago|0 comments
3. 123.▲
   by vgt 9y ago

   BigQuery specifically pioneered many of these concepts since it's release in 2012 - "serverless manageability", pay-per-query consumption-based pricing, pure separation of compute and storage (no intermediate ssh mesh), and m
4. 124.▲
   Data Definition Language in Google BigQuery(cloud.google.com)

   1 points by vgt 9y ago|0 comments
5. 125.▲
   by vgt 9y ago

   I respectfully disagree. While what "serverless" truly means varies depending on who you ask, in this context I think it's: - Multi-tenant architecture. - Pay-per-job/query. - Complete abstraction away of underlying reso
6. 126.▲
   by vgt 9y ago

   Here's a good presentation by Jordan, the BigQuery Eng lead that describes what you're talking about: https://www.youtube.com/watch?v=UueWySREWvk
7. 127.▲
   by vgt 9y ago

   Resource at [0] comes to mind. There are differences, biggest one within the context of this discussion is that Google cloud carries a multi-regional and global philosophy, whereas aws tends to be zonal. This is expressed in things like glo
8. 128.▲
   by vgt 9y ago

   Thanks for reading and using Google cloud :)
9. 129.▲
   by vgt 9y ago

   Yeah definitely streaming API. This use case is MADE for BigQuery :) Check out, as well as the blog posts on the required reading list on the topic of inserts. There's also a pretty good Kafka-bq connector. It sounds like "ingest&
10. 130.▲
    by vgt 9y ago

    I think the messages are crossing paths here. Your initial concern of 1000 loads per project is actually 50000 and can be raised. The limit of 1000 loads per TABLE per day is there, can't be raised, and a couple posts above explain how
11. 131.▲
    by vgt 9y ago

    Jordan has since become the engineering lead on bigquery and Dremel! The book is still very relevant but would be nice to get a refresh - many things have changed. Perhaps the biggest one is the execution engine. Jordan gave a nice talk on
12. 132.▲
    by vgt 9y ago

    It's actually per table: https://cloud.google.com/bigquery/quotas#import
13. 133.▲
    by vgt 9y ago

    I work at Google and authored some of those posts. The limitation is per table[0] and for "batch loads" only. If you have per row inserts, you can always use bigquery's amazing streaming API, you can also do materialization a
14. 134.▲
    by vgt 9y ago

    I work at Google and was on the bigquery/dremel team. I've written on this topic in the past at [0]. Turns out analytic SQL workloads are incredibly volatile - we know from operating these big ol Dremel/bq clusters in a multi
15. 135.▲
    by vgt 9y ago

    my plot's been revealed!
16. 136.▲
    by vgt 9y ago

    I personally prefer to weave the fact that I work at Google right in the middle of the point that I am trying to make.
17. 137.▲
    by vgt 9y ago

    and Kaggle and api.ai and apigee and Qwiklabs and Orbitera and Anvato and you get the point :)
18. 138.▲
    by vgt 9y ago

    I think your point is a microcosm of the popular "cloud TCO" debate - benefits of higher levels of abstraction. Can you get Google's network elsewhere? No. Does it cost more? Sometimes. Is it useful? For a lot of folks runnin
19. 139.▲
    by vgt 9y ago

    While I agree that there are situations where "network egress" can become a large part of one's bill, I encourage you to think of the technical impact of having something like Spanner and something like the Google network. A
20. 140.▲
    Google Cloud Dataprep Hits Public Beta(cloud.google.com)

    3 points by vgt 9y ago|0 comments
21. 141.▲
    Google Analytics data is now streamed to BigQuery(analytics.googleblog.com)

    3 points by vgt 9y ago|0 comments
22. 142.▲
    New German GCP Region – Open Now(cloudplatform.googleblog.com)

    3 points by vgt 9y ago|0 comments
23. 143.▲
    by vgt 9y ago

    I sometimes ask Google Cloud clients who post this type of stuff, what's your motivation? The one consistent theme is that telling the world that you're doing cool stuff with cool tech raises your organization's profile and h
24. 144.▲
    by vgt 9y ago

    Also nice to have sub-millisecond latency to first byte (versus multi-hour) and performance characteristics of normal GCS.. so you could run a rare MapReduce job on it just the same as regular GCS :)
25. 145.▲
    by vgt 9y ago

    Check out Google's Coldline https://cloud.google.com/storage/archival/
26. 146.▲
    by vgt 9y ago

    I like your thinking, but one minor clarification. BigQuery's actually introduced Flat Rate [0] ( a year ago) and Committed Use Discounts [1] (Amazon RIs are similar) since that's kind of what some Enterprises want. These are opti
27. 147.▲
    Show HN: Creating an object detection application using TensorFlow(cloud.google.com)

    2 points by vgt 9y ago|0 comments
28. 148.▲
    Google cuts prices on cloud SSDs by up to 63%(venturebeat.com)

    5 points by vgt 9y ago|0 comments
29. 149.▲
    by vgt 9y ago

    I work for Google Cloud and want to make a note. This page is quoting Google Cloud's multi-region storage option, which is the most redundant storage option in the industry (but more expensive than, say, Regional GCS), which is what S3
30. 150.▲
    by vgt 9y ago

    Hey there, work at Google. Please feel free to ping me with your concerns. That's certainly not the experience we strive for.