7 ms·
Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery
- lima 9y agoI built a similar pipeline using Kafka and ClickHouse - it's amazing how easy it is nowadays to ingest and analyze billions of events a day using standard tools. ClickHouse can even ingest directly from Kafka (courtesy of Cloudflare - http://github.com/vavrusa http://github.com/vavrusa contributed it).
- SiempreViernes 9y agoDo you learn anything useful about humanity tho?
- pcarolan 9y agoFair comment. The title would imply there's a 'so what?' in the article, not just a 'how'.
- ishi 9y agoCan you elaborate a bit about the servers you used (hardware, cluster size, etc.)?
- xstartup 9y agoHere is an example Pipeline which supports loading data from an unbounded source to BigQuery in batches using load jobs (evading BigQuery's Streaming Insert cost) See: https://zero-master.github.io/posts/pub-sub-bigquery-beam/ https://zero-master.github.io/posts/pub-sub-bigquery-beam/
- cobookman 9y agoFYI dataflow's bigqueryio does not use stream inserts. Instead it batches the data into many load jobs
- oomkiller 9y agoThat is configurable, but for unbounded PCollections it appears the default is to use streaming inserts https://beam.apache.org/documentation/sdks/javadoc/2.4.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html https://beam.apache.org/documentation/sdks/javadoc/2.4.0/org...
- AWebOfBrown 9y agoAre you the author? OT but I'm amazed the author charged merely 100 euro for implementing that solution for the subject startup, even if they're cash-strapped. I'm not familiar with BigQuery, but I'm curious what a normal rate for solving that issue would look like.
- mentat 9y agoThat's shockingly cheap for the value.
- manigandham 9y agoYou can also skip pub/sub and/or use it to write files to cloud storage, then load from there by using a cloud function that will trigger the load job when a new object is created.
- ryanworl 9y agoFor anyone who wants to move this much data out of AWS and into another cloud provider, Kinesis Streams does not charge for bandwidth out. If you do the math, it works out to a very large savings over doing public internet data transfers if you are moving terabytes per day. The compute costs to copy data from Kafka to Kinesis should be minimal since you’re essentially just operating a pipe and not doing much actual compute, and this can operate easily on spot instances.
- nh2 9y agoCan you elaborate on that? I haven't been able to conclude that from what it says on the pricing page [1]: > Data transfer is free. AWS does not charge for data transfer from your data producers to Amazon Kinesis Data Streams, or from Amazon Kinesis Data Streams to your Amazon Kinesis Applications. That sounds like it's sonly free to Amazon Kinesis Applications (== inside AWS). And on [2] it says: > If you use Amazon EC2 for running your Amazon Kinesis Applications, you will be charged for Amazon EC2 resources in addition to Amazon Kinesis Data Streams costs. So that sounds like you will eventually pay the normal egress cost of EC2. [1]: https://aws.amazon.com/kinesis/data-streams/pricing/ https://aws.amazon.com/kinesis/data-streams/pricing/ [2]: https://aws.amazon.com/kinesis/data-streams/faqs/ https://aws.amazon.com/kinesis/data-streams/faqs/
- ryanworl 9y ago“If you use Amazon EC2 for running your Amazon Kinesis Applications” is the key there. You don’t have to use Amazon EC2. Amazon Kinesis Application does not imply EC2. They use on-prem examples in a few Re:Invent presentations. It is just an HTTP API like most everything else within Amazon. You will see other services spell out their bandwidth out charges much more clearly.
- davidbrent 9y agoVery interesting read, but as more of an analyst, I kept waiting for the ‘meaning.’ Maybe I missed it, but for anyone else who may be considering reading this, it is more about ‘giving business structure’ to analytic events. Good article none the less.
- hinkley 9y agoI see titles like this and my first thought is “more people need to watch Real Genius or watch it again.” “What do you think a phase conjugate tracking system is for, Kent?” Great. You made a system to track a billion people a day. You’re murdering privacy and then bragging about it. And bragging about it during a giant shitstorm caused by Facebook. The fuck is wrong with you?
- Arzh 9y agoI want to know how people use my website, fuck me right?
- worstscenario 9y agoExactly, fuck you.
- tzahola 9y agoAs soon as the IP packets leave your server, it’s not “your website” anymore.
- cheriot 9y agoWould you say as soon as the IP packets leave my device it's not "my data" anymore? It's simplified too far.
- hinkley 9y agoA hundred billion events a day isn’t “knowing how people use your website”. It’s mass scale surveillance. It’s stalking your customers.
- manigandham 9y agoIs there some magic scale threshold?
- hinkley 9y ago
- buremba 9y agoSounds like they over-engineered the solution. If you have ad-hoc use-case, BigQuery is great but it's quite expensive. If you just need to pre-calculate the metrics using SQL, Athena / Prestodb / Clickhouse / Redshift Spectrum might be much easier and cost-efficient.
- legendofneo 9y agoI agree with this statement if you consider BigQuery's on-demand pricing, but at a given scale the $40,000/month flat rate becomes competitive.
- vgt 9y agoBigQuery PM here. I'd love to genuinely understand why you have that impression. BigQuery's on-demand model charges you EXACTLY for what you consume. Meaning, your resource efficiency is 100% [0]. By contrast, typical "cluster pricing" technologies require you to pay for 100% of your cluster uptime. In private data centers, it's difficult to get above 30% average efficiency. BigQuery also takes care of all software, security, and hardware maintenance, including reprocessing data in our storage system for maximum performance and scaling your BigQuery "cluster" for you.[1] BigQuery has a perpetual free tier of 10GB of data stored and 1TB of data processed per month. Finally, BigQuery is the only technology we're aware of whose logical storage system doesn't charge you for loads - meaning we don't compromise your query capacity, nor do we bill you for loads. [0] https://cloud.google.com/blog/big-data/2016/02/visualizing-the-mechanics-of-on-demand-pricing-in-big-data-technologies https://cloud.google.com/blog/big-data/2016/02/visualizing-t... [1] https://cloud.google.com/blog/big-data/2016/08/google-bigquery-continues-to-define-what-it-means-to-be-fully-managed https://cloud.google.com/blog/big-data/2016/08/google-bigque...
- slap_shot 9y agoThe answer comes down to one line in your BigQuery Under the Hood article[0]: "The answer is very simple — BigQuery has this much hardware (and much much more) available to devote to your queries for seconds at a time. BigQuery is powered by multiple data centers, each with hundreds of thousands of cores, dozens of petabytes in storage capacity, and terabytes in networking bandwidth. The numbers above — 300 disks, 3000 cores, and 300 Gigabits of switching capacity — are small. Google can give you that much horsepower for 30 seconds because it has orders of magnitude more." Most companies are fine letting their warehouse go underutilized, or for queries to not be solved with such enormous resources, if it means capping their monthly data warehouse bill at a fixed number, say $9,000. BigQuery is an awesome piece of technology, but most publishers, ecommerce, and saas companies have teams of anlayts, engineers, and business folks pounding away at their warehouse all day. And it's fine if those queries aren't as fast BigQuery. I run an analytics companies and we load billions of events into our customers' warehouses each day. Many have evaluated BigQuery and they all came back with the same answer: too expensive. Most of them are big companies, but spending nowhere near the $40K they'd have to to cap their cost on BigQuery. And with the advent of Spectrum, they're even less likely to jump ship now. Since you're a PM, I'd be really interested to know if you guys are aware of this issue and if you're doing anything to offer a solution that competes with Redshift (fixed cost/resource). I ask this as someone who runs a ton of stuff on GCP, but we've just never found a way to make BigQuery cost effective for us. [0]https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde...
- throwaway66666 9y agoYou 'd be surprised how good modern hardware is. We are being hit with between 1.5 and 2 million requests per minute (steady traffic no spikes except increased usage in the weekends), and our analytics solution runs on just 2 main servers and costs $3k per month (total associated costs except human labour which is 1 engineer working on it part time now). We talked with a famous analytics company and they gave us a quote of 1 million yearly to work with us. So how did we get it down from 1m to 35k? We pretty much do the thing the article suggests (roll-ups section). We compute data hourly. Alongside with the hourly computation we also dump some extra data that can be used to compute numbers for the day (eg unique user ids, from unique user ids per hour), then from the day we get per week and then per month. We also have less moving pieces (our stack is way more traditional), and we manage our own hardware (key for keeping costs down) When you get data back from the system you only hit the pre-computed cache, no query touches the main system from the dashboards. We only allow queries running in a 30 minute window to run on the live system - to ensure that no crazy load is going to be built on top of it and we use that to mostly catch anomalies on the real time data. (our parsing time is good too, between 10 seconds and 1 minute compared to the 2-30 minutes the article gives). However this is the "You are alive but you 're not living" angst of analytics. All the data is there, but you cannot freely prod it for answers and patterns. If you want to get an answers about past data, you need to go through an overly complex process of raising a new cluster and ingesting old backups, multiple times, then waiting for a few days. It get's relatively expensive, slow and at times will demoralize you and make you back off from getting the answers you need. You could try keeping a smaller cluster that only gets % percent of the data (eg only 2%) for finding trends, drawing heatmaps etc and that one can run in realtime but your CEO will say that's a stupid idea to your face and it's realtime-all or nothing. You might say that's a situation you can live with provided the absolutely insane cost savings, but when the company goes at a nicer retreat for only a selective elite few that easily costs 20k, or runs an over the top kitsch open party party/recruiting that costs 80k, and you are being dragged into a meeting on a monday morning and confronted; Why did the "3k per month-3 billion requests per day" system cost 6k this month? (because we had multiple clusters in parallel computing historical data for the past 6 months that you asked for). You just get bitter you didn't give the analytics company the 1 million they asked for and be done with it.
- 9y ago
- asavinov 9y ago> Giving meaning to ... Ingesting such amounts of data is a challenge indeed. But problems will become much more complicated if it is necessary to perform complex analysis during data ingestion. Such analysis (not simply event pre-processing) can arise because of the following reasons: * It is physically not possible to store this amount of events. For example, assume you collect them from devices and sensors * It is necessary to make faster decisions, e.g., in mission critical applications * It can be more efficient to do some analytics before storing data (as opposed to first storing data persistently and then loading it again for analysis) Such analysis can be done by conventional tools like Spark Streaming (micro batch processing) or Kafka Streams (works only with Kafka). One novel approach is implemented in Bistro Streams [0] (I am an author). It is intended for general-purpose data processing including both batch and stream analytics but it radically differs from MapReduce, SQL and other set-oriented data processing frameworks. It represents data via functions and processes data via column operations rather than having only set operations. [0] Bistro: https://github.com/asavinov/bistro https://github.com/asavinov/bistro
- maxnevermind 9y ago"In digital advertising, ..." Stopped reading right there. Maybe a great article though. I started working on that thing called BigData not that long time ago but now realized that 50% of the job is advertisement, not fan of it at all, I want to like the end product or at least be neutral about it.
- GrandNewbien 9y agoCould you elaborate? There's plenty of big data tasks and roles that aren't even closely related to marketing.
- beepbeepbeep1 9y agoNot my experince. I'd say the majority of big data is unethical both in capture and use trying to sell you something or sell someone else something on you, a small amount is for general good. I wasn't involved on the project but a major Telecoms company i was at was prototyping scanning for mobile devices on their provided home routers. If they didn't recognise a mibile as being on their network they would then target you with mobile deals. Devs saw the issues with this, essentially scanning your home network from their routers and sending the data back to be processed. Managers found Apple's mac address rotation annoying and lost the point why Apple do it. Unfortunately besides quitting the job devs on the project don't have a say as they don't have the power or say in the final decisions at these companies. It's easy to say you would quit but it's harder when you have a mortgage, wife and kids for example.
- amorphid 9y agoMaybe you'd find Amino interesting. They use big data for healthcare savings. https://amino.com https://amino.com
- manigandham 9y agoThe vast majority of big data is used today to provide everything you depend on, like email, healthcare, banking, transportation, government services, etc.
- jwilliams 9y agoI've been working with all these tools for a while now (and a bunch more along the way). It's not the orthodox cloud-thinking, but you're often best off processing at the point of creation (or ingestion). Normalize the data as much as you can, (probably) compress[1], and send as close to the target as possible. If you grab the data, send elsewhere, transform <repeat>... it all gets slow and expensive pretty quickly. Also a headache to manage failures. This is especially true if you're using BigQuery. Stage your near-raw data into BigQuery and then use it's muscle as much as you can. A classic example here might be de-duplicating data. A painful prospect for many distributed systems, but pretty easy on the BigQuery side. This is all especially true for time-series data. With BigQuery time partitions you can keep the queries fast (and the costs reasonable). Also limits the range of technologies and languages you need to wrangle too. 1: Choosing your data format and compression approach can make a huge difference.