fhoffa

12 ms·

1. 151.▲
   by fhoffa 11y ago

   Look... if you don't have problems that take hours to solve in your current environment, you haven't found big data problems (yet). And that's OK. Not everyone works with big data. But if one day you do, please ping me, it wi
2. 152.▲
   by fhoffa 11y ago

   Twitter noticed: https://twitter.com/_pwalsh/status/715054406073524224
3. 153.▲
   by fhoffa 11y ago

   Freebase data is being moved into Wikidata. I loaded Wikidata into BigQuery too! Fun with movies and cats: https://medium.com/google-cloud/oscars-2016-movies-that-got-... Implementation notes: https://lists.
4. 154.▲
   by fhoffa 11y ago

   > Especially in the sciences. Take a look at how Stanford is leveraging BigQuery for their genomics analysis: http://www.eventbrite.com/e/interactive-cloud-analytics-extr... > It manages that faster than just the
5. 155.▲
   by fhoffa 11y ago

   Hi Janne! I'm sure you'll enjoy a challenge. Let's talk about Wikidata. Can you download this 8 GB compressed file? https://dumps.wikimedia.org/wikidatawiki/entities/latest-all... I want to know the
6. 156.▲
   by fhoffa 11y ago

   2008. I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" ' 8 years later GMail has a billion users, and not going anywh
7. 157.▲
   by fhoffa 11y ago

   Some resources to get you started: - How to use the Hacker News dataset https://medium.com/google-cloud/big-data-stories-in-seconds-... - Discussion of the HN dataset announcement here https://news.ycombinat
8. 158.▲
   by fhoffa 11y ago

   Note that Google Cloud offers preemptible instances, up to 70% cheaper than the normal ones. https://cloud.google.com/preemptible-vms/ It would be interesting to see Quizlet thoughts on GCP preemptible VMs vs AWS spot
9. 159.▲
   Spotify Moving from Kafka to Google Cloud Pub/Sub, Part II: Messaging at ~1Gbps(labs.spotify.com)

   13 points by fhoffa 11y ago|4 comments
10. 160.▲
    [video] TensorFlow Tutorial: Distributed Systems for Training Neural Networks(research.microsoft.com)

    3 points by fhoffa 11y ago|0 comments
11. 161.▲
    by fhoffa 11y ago

    Note that this proposal is being back not only by Google, but also Cloudera, Data Artisans, Talend, Cask, PayPal, ... Some other posts on the announcement: http://googlecloudplatform.blogspot.com/2016/01/Dataflow-a
12. 162.▲
    by fhoffa 11y ago

    Nice! On https://reddit.com/r/bigquery , /u/omicron_n2 left queries to repeat the experiment on HN and on reddit comments too: - https://reddit.com/r/bigquery/comments/41py1v&#x2
13. 163.▲
    287M events/day and 1 engineer: How I built Quizlet's data pipeline(quizlet.com)

    2 points by fhoffa 11y ago|0 comments
14. 164.▲
    by fhoffa 11y ago

    Clunky ETL: Please correct me if I'm wrong, but what I saw about ETLs in the article "One example we encounter quite often is that Mixpanel stores timestamps in seconds, while Redshift expects timestamps in milliseconds." - t
15. 165.▲
    by fhoffa 11y ago

    Hi melted! I have to agree with you that BigQuery is probably faster, cheaper and more scalable than Redshift for this and other use cases. Specially given the "join multiple datasets" angle. But why would you say that ETL would b
16. 166.▲
    by fhoffa 11y ago

    I charted the peak times here: https://github.com/fhoffa/notebooks/blob/master/analyzing%20... (Python notebook - renders well on desktop, but GitHub might not show a nice rendering if you try it on mobi
17. 167.▲
    by fhoffa 11y ago

    I joined GitHub and Hacker News here - how many stars after a front page post? https://www.reddit.com/r/bigquery/comments/3qpyor/joining_ha...
18. 168.▲
    by fhoffa 11y ago

    Links: - Visualization: https://i.imgur.com/B5awmAL.png - GitHub Archive: http://www.githubarchive.org/ - Hacker News on BigQuery dataset: https://news.ycombinator.com/item?id=10440502 - Ot
19. 169.▲
    Tell HN: The Hacker News frontpage effect on a project GitHub stars

    16 points by fhoffa 11y ago|2 comments
20. 170.▲
    by fhoffa 11y ago

    and reddit too https://www.reddit.com/r/bigquery/comments/3cej2b/17_billion... and wikipedia, and weather, and... https://www.reddit.com/r/bigquery/wiki/datasets :)
21. 171.▲
    by fhoffa 11y ago

    Bad guess then! The answer is that the dataset must be very similar to the Algolia one, since both get their data from the same source: the Hacker News official API on Firebase. (but Algolia keeps it up-to-date in realtime, while I haven&#x
22. 172.▲
    by fhoffa 11y ago

    Btw, I really like your query. I modified it to get the story for up to 7 levels of recursion: SELECT p0.id, s.id, s.title, level FROM ( SELECT p0.id, p0.parent, p2.id, p3.id, p4.id, COALESCE(p7.parent, p6.parent, p5.parent, p4.pa
23. 173.▲
    by fhoffa 11y ago

    I guess it incorporates gravity - it's whichever way Hacker News tells its clients to rank the comments by.
24. 174.▲
    by fhoffa 11y ago

    You are right - I'll prepare a new release with that data. My oversight, sorry! :)
25. 175.▲
    by fhoffa 11y ago

    Parent comment was deleted, but given the grandparent comment I'll guess you are asking how is BigQuery different to Redshift? There is an ongoing conversation about this on reddit: https://www.reddit.com/r/bigdata
26. 176.▲
    by fhoffa 11y ago

    Hacker News chose to hide comment scores some time ago, but I still wanted to find a way to rank comments. The good news is that the API gives you a "kids" column that ranks comments in the order they should be displayed - that&#x
27. 177.▲
    by fhoffa 11y ago

    I didn't name the column "by", I just gave the field the name that the API uses for it. And to make everyone's lives easier (including mine), I copied the [by] column to an [author] column, so you can do a >select ...
28. 178.▲
    by fhoffa 11y ago

    Hi! I'm the one that loaded this dataset into BigQuery. Feel free to ask any questions :). The notebook with sample queries and visualizations: https://github.com/fhoffa/notebooks/blob/master/analyzi
29. 179.▲
    Hacker News dataset now shared on BigQuery: What are the best times to post?(github.com)

    1 points by fhoffa 11y ago|0 comments
30. 180.▲
    Google Cloud Datalab, an Interactive Tool for Exploring and Visualizing Data(techcrunch.com)

    3 points by fhoffa 11y ago|0 comments