6 ms·

> Hive SQL, Spark SQL, Scala Spark, PySpark and Presto are widely used as different execution engines This makes me think they're doing something very very wro

> Hive SQL, Spark SQL, Scala Spark, PySpark and Presto are widely used as different execution engines This makes me think they're doing something very very wrong. AirBNB does not have data on the scale that would require these tools. They have 5.6 million listings, 150 million users, and 1 billion total person-stays. These numbers can easily be processed with Postgres or SQLite on single machines. Spark and Hive are for companies like Google and Facebook. https://www.thezebra.com/resources/home/airbnb-statistics/#infographic https://www.thezebra.com/resources/home/airbnb-statistics/#i...

rebelos 4y ago
Have you ever worked in data engineering? They're using these systems for event data, data generated through transformations (multiplicative effect on base size), data used for ML, etc.
- Dylan16807 4y ago
  How many events would there be per stay, in your estimation? And how many are actually important?
  - tuckerman 4y ago
    These events aren't just being generated per stay. A company like Airbnb will have events about logins, searches, site interactions, etc. You'll also be transforming the raw data and storing it again as higher level, materialized tables. Disclaimer: Worked at Airbnb (not on a data engineering or data infra team)
    - Dylan16807 4y ago
      So all unimportant data? I mean sure you can squeeze insights out of that but if a third of it disappeared overnight it wouldn't be a big deal. And even then anything short of obsessive mouse tracking won't be that much data. This isn't doing much to prove that the stuff in the article matters. Maybe it does but it's not self-evident and the criticism upthread makes sense. (Please note that I am not ignorantly saying the job is easy. I'm mostly wondering if it affects revenue and satisfaction by more than a tiny sliver to do the hard job with all these different big data engines as opposed to doing a much simpler job.)
      - cdavid 4y ago
        Search interactions data is one of the most valuable data in marketplaces. I never worked at Airbnb, but I worked at companies smaller than airbnb where improving ranking had many millions $ / year impact on revenue. > And even then anything short of obsessive mouse tracking won't be that much data. Consider tracking clicks and query / results. That's already 2 orders of magnitude more data than suggested by the OP, even under very conservative assumptions.
        Dylan16807 4y ago
        > That's already 2 orders of magnitude more data than suggested by the OP, even under very conservative assumptions. If we estimate a search input as 50 bytes and the results as 25 4-byte ID numbers, then multiply by 100 million, that's 15TB, one hard drive or a couple SSDs. And a hard drive full of clicks can fit a trillion. So even at 2-3 orders of magnitude over a billion, we're not looking at huge sums of data. And it's quite questionable whether you need special systems dedicated to keeping that click data pristine at all times.
        tuckerman 4y ago
        A search would contain a lot more than that. Structured events about search result pages will often contain a denormalized version of the results and how they were placed on the screen, experiment ids, information about the users session, standard tracking data for anti abuse. You might use these data to make statements in legal documents, financial filings, etc. and therefore you’d want a good story about why those data are trustworthy.
        cdavid 4y ago
        Even using your numbers, if you want to keep say only 3 months of data, we're talking about 1-2 PB already. Being able to query those data across different segments, aggregate into different dimensions is already quite far beyond what you can do w/ off the shelve postgresql or sqlite. And in general, in companies the size of airbnb, you don't control all the data sources to be super efficient, because that's organizationally impossible. So instead the data will be de normalized, etc. There is a reason most companies with those problems use systems like BQ, snowflake, and co. If it were possible to do with sqlite, a lot of them would do it.
  - tgsovlerkhgsel 4y ago
    Hundreds, if you consider all the web site interactions to book it?
joneholland 4y ago
This is what I love about Hackernews. Every now and then you see someone post with such confidence something that is so absolutely hilariously wrong.
- mywaifuismeta 4y ago
  It happens with technology sometimes, but all time the with finance. Having worked in hedge funds most of my career (as an engineer but I see enough of the business side) it's hilarious how clueless but confident people on HN are about anything that touches finance, trading, stocks, crypto, etc. Nothing wrong with being clueless, but the hilarious part is the confidence of the posters about what they are writing, which they probably got from Medium. If you don't know better you'd think they know what they're talking about.
rchiang 4y ago
Maybe some other numbers are more relevant? This blog post from April 2022 mentions 1000+ services: https://medium.com/airbnb-engineering/continuous-delivery-at-airbnb-6ac042bc7876 And this video from May 2017 mentions 1B daily events (for whatever they define as an event): https://youtu.be/70luTZU-D3E?t=102 It wouldn't surprise me if they're storing calls between microservices as "events" and they're likely logging a lot of both user data and internal services data, but that's purely a guess.
- abdouls 4y ago
  Interesting article (first one). It seems like more and more companies (looking at AWS and Netflix directly) seem to deploy ~1k micro services. I work in a team where we manage ~14 micro services (per environment - dev, staging and production - max ~42) and find it complicated to manage and monitor...
- deleted 4y ago
  [deleted]
cdavid 4y ago
They most likely do. Think about e.g. ranking in their search/recommendation engine. To be able to train ranking ML models, you would need to at least track the views, clicks, purchases, etc. done through their platform. For each search, you want to keep the query string and the results items ids. Let's say, very conservatively, they have 1000 qps on average. We're talking about 100 of millions of events a day.
didip 4y ago
This post has a strong HN energy. In data engineering, once data went through several transformations, the final event count can explode by a lot.

rebelos 4y ago

Have you ever worked in data engineering? They're using these systems for event data, data generated through transformations (multiplicative effect on base size), data used for ML, etc.

Dylan16807 4y ago
How many events would there be per stay, in your estimation? And how many are actually important?
- tuckerman 4y ago
  These events aren't just being generated per stay. A company like Airbnb will have events about logins, searches, site interactions, etc. You'll also be transforming the raw data and storing it again as higher level, materialized tables. Disclaimer: Worked at Airbnb (not on a data engineering or data infra team)
  - Dylan16807 4y ago
    So all unimportant data? I mean sure you can squeeze insights out of that but if a third of it disappeared overnight it wouldn't be a big deal. And even then anything short of obsessive mouse tracking won't be that much data. This isn't doing much to prove that the stuff in the article matters. Maybe it does but it's not self-evident and the criticism upthread makes sense. (Please note that I am not ignorantly saying the job is easy. I'm mostly wondering if it affects revenue and satisfaction by more than a tiny sliver to do the hard job with all these different big data engines as opposed to doing a much simpler job.)
    - cdavid 4y ago
      Search interactions data is one of the most valuable data in marketplaces. I never worked at Airbnb, but I worked at companies smaller than airbnb where improving ranking had many millions $ / year impact on revenue. > And even then anything short of obsessive mouse tracking won't be that much data. Consider tracking clicks and query / results. That's already 2 orders of magnitude more data than suggested by the OP, even under very conservative assumptions.
      - Dylan16807 4y ago
        > That's already 2 orders of magnitude more data than suggested by the OP, even under very conservative assumptions. If we estimate a search input as 50 bytes and the results as 25 4-byte ID numbers, then multiply by 100 million, that's 15TB, one hard drive or a couple SSDs. And a hard drive full of clicks can fit a trillion. So even at 2-3 orders of magnitude over a billion, we're not looking at huge sums of data. And it's quite questionable whether you need special systems dedicated to keeping that click data pristine at all times.
        tuckerman 4y ago
        A search would contain a lot more than that. Structured events about search result pages will often contain a denormalized version of the results and how they were placed on the screen, experiment ids, information about the users session, standard tracking data for anti abuse. You might use these data to make statements in legal documents, financial filings, etc. and therefore you’d want a good story about why those data are trustworthy.
        cdavid 4y ago
        Even using your numbers, if you want to keep say only 3 months of data, we're talking about 1-2 PB already. Being able to query those data across different segments, aggregate into different dimensions is already quite far beyond what you can do w/ off the shelve postgresql or sqlite. And in general, in companies the size of airbnb, you don't control all the data sources to be super efficient, because that's organizationally impossible. So instead the data will be de normalized, etc. There is a reason most companies with those problems use systems like BQ, snowflake, and co. If it were possible to do with sqlite, a lot of them would do it.
- tgsovlerkhgsel 4y ago
  Hundreds, if you consider all the web site interactions to book it?

joneholland 4y ago

This is what I love about Hackernews. Every now and then you see someone post with such confidence something that is so absolutely hilariously wrong.

mywaifuismeta 4y ago
It happens with technology sometimes, but all time the with finance. Having worked in hedge funds most of my career (as an engineer but I see enough of the business side) it's hilarious how clueless but confident people on HN are about anything that touches finance, trading, stocks, crypto, etc. Nothing wrong with being clueless, but the hilarious part is the confidence of the posters about what they are writing, which they probably got from Medium. If you don't know better you'd think they know what they're talking about.

rchiang 4y ago

Maybe some other numbers are more relevant? This blog post from April 2022 mentions 1000+ services: https://medium.com/airbnb-engineering/continuous-delivery-at-airbnb-6ac042bc7876 And this video from May 2017 mentions 1B daily events (for whatever they define as an event): https://youtu.be/70luTZU-D3E?t=102 It wouldn't surprise me if they're storing calls between microservices as "events" and they're likely logging a lot of both user data and internal services data, but that's purely a guess.

abdouls 4y ago
Interesting article (first one). It seems like more and more companies (looking at AWS and Netflix directly) seem to deploy ~1k micro services. I work in a team where we manage ~14 micro services (per environment - dev, staging and production - max ~42) and find it complicated to manage and monitor...
deleted 4y ago
[deleted]

cdavid 4y ago

They most likely do. Think about e.g. ranking in their search/recommendation engine. To be able to train ranking ML models, you would need to at least track the views, clicks, purchases, etc. done through their platform. For each search, you want to keep the query string and the results items ids. Let's say, very conservatively, they have 1000 qps on average. We're talking about 100 of millions of events a day.

didip 4y ago

This post has a strong HN energy. In data engineering, once data went through several transformations, the final event count can explode by a lot.