vgt

101 ms·

1. 451.▲
   Understanding Google BigQuery's rapid scaling(cloud.google.com)

   3 points by vgt 11y ago|0 comments
2. 452.▲
   by vgt 11y ago

   Shameless plug: https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-...
3. 453.▲
   Google BigQuery Under the Hood – Borg, Jupiter, Colossus, and Dremel(cloud.google.com)

   2 points by vgt 11y ago|0 comments
4. 454.▲
   by vgt 11y ago

   Curious what Google lacks compared with Amazon and Azure, outside of having to raise default quotas?
5. 455.▲
   by vgt 11y ago

   yup, GCE gives you a "VPC" by default
6. 456.▲
   by vgt 11y ago

   I think both length of code is a side effect that results from the primary argument, and "cant do event time" is one of the symptoms. Neither is a primary argument in the blog post. The primary argument is demonstrated through col
7. 457.▲
   by vgt 11y ago

   Thanks! The original URL is a new developer-focused Big Data blog by Google and Google customers: https://cloud.google.com/blog/big-data/
8. 458.▲
   Dataflow/Beam and Spark: A Programming Model Comparison(cloud.google.com)

   109 points by vgt 11y ago|32 comments
9. 459.▲
   Compute Engine Now with 3 TB of Local SSD and 64 TB of Persistent Disk per VM(googlecloudplatform.blogspot.com)

   3 points by vgt 11y ago|0 comments
10. 460.▲
    by vgt 11y ago

    Because 2FA
11. 461.▲
    by vgt 11y ago

    It's a Batch+Stream unified processing model, an SDK. Idea is you can code up your pipeline in Dataflow and have your choice of where to run it - Spark, Flink, etc. Google Cloud Dataflow is a fully-managed service that executes Dataflo
12. 462.▲
    by vgt 11y ago

    This story doesn't get told enough. Most of Google Cloud is built to operate the same way with 1 user or 1m users. And in many cases, Google doesn't charge you for the "scaling vector", whereas AWS will, and will sometim
13. 463.▲
    by vgt 11y ago

    beauty of SDN :)
14. 464.▲
    by vgt 11y ago

    Google's SDN is seriously massive. Petabit of bisectional bandwidth: http://googlecloudplatform.blogspot.com/2015/06/A-Look-Insid... One major difference between Google and AWS is that Google will carry your
15. 465.▲
    by vgt 11y ago

    got it. thanks!
16. 466.▲
    by vgt 11y ago

    There are plenty of architectures that do exactly this. EMR-on-S3, Google Dataproc on GCS, Snowflake-on-S3, BigQuery-on-GCS, etc etc. The bigger point in the article is that these exact "take processing to the data" architectures
17. 467.▲
    by vgt 11y ago

    According to this page, you can get ~14Gbit networking on Google Cloud's n1-standard-8 http://googlecloudplatform.blogspot.com/2015/11/bringing-you...
18. 468.▲
    by vgt 11y ago

    Your statement is simply false. Here's a material answer - https://cloud.google.com/docs/google-cloud-platform-for-aws-... In short, Google has parity with AWS on many fronts, and exceeds AWS on many others. Only
19. 469.▲
    by vgt 11y ago

    Google has been the price leader in compute and storage for at least 2 years. Here's an example: http://www.rightscale.com/blog/cloud-cost-analysis/google-vs...
20. 470.▲
    by vgt 11y ago

    Bigtable is best thought of as an "event database". High reads, high writes, single index, accessible through the Hbase API. Cassandra and Hbase are similar technologies that are inspired by the original Bigtable paper. One big be
21. 471.▲
    by vgt 11y ago

    Take a look at PubSub + Google Cloud Dataflow combo.
22. 472.▲
    by vgt 11y ago

    One thing you gotta remember is, PubSub charges per volume, regardless of speed (in other words, scaling is free). AWS will charge you varying orders of magnitudes for varying scales, in addition to volume.
23. 473.▲
    by vgt 11y ago

    I like to think that BigQuery is far far ahead of Redshift in terms of performance + scale + manageability + cost (but i may be biased).
24. 474.▲
    by vgt 11y ago

    There's a free tier, but it's not free beyond that. Otherwise folks would be mining bitcoins using BigQuery UDFs :)
25. 475.▲
    by vgt 11y ago

    I suggest you check out BigQuery. Redshift is not truly "fully managed", and it's not really HA/Durable, as your experience indicates.
26. 476.▲
    by vgt 11y ago

    "we run our clusters at 50% utilization". This seems exceedingly wasteful.
27. 477.▲
    by vgt 11y ago

    I agree with you :)
28. 478.▲
    by vgt 11y ago

    Likewise, but I think you're getting into the philosophical, not the practical. You may choose to live in a single-CPU world for your database, but you're simply disqualifying yourself from a whole lot of interesting use cases. In
29. 479.▲
    by vgt 11y ago

    If you have a SQL query that takes 50,000 core-seconds, it's probably more useful to execute that query using 10,000 cores in 5 seconds rather than 10 cores and 5000 seconds, especially if cost is the same. Even better if you never hav
30. 480.▲
    by vgt 11y ago

    PubSub is not a product I work on, so apologize for lack of detail: - PubSub is used by Google internally to power everything from Android notifications to Hangouts messages. So it's certainly proven. - A lot of your questions are answ