6 ms·

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying an

If anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that come to mind.

ljk 11y ago
part of it is probably recruiting
nedwin 11y ago
aesthetics
phunge 11y ago
They mention that it was originally paired with Druid. The data volume that Druid excels at (and that AirBnB must have) is orders of magnitude larger than what Tableau&Looker do well with. It's probably just built for bigger-than-SQL OLAP usecases.
- jsiegz 11y ago
  Looker scales with plenty of data for us at Snapchat, it's more about the underlying database than the BI tool.
  - MechanicalTwerk 11y ago
    Note that this user used to work at Looker.
    - jsiegz 11y ago
      yep i did.
- dwmintz 11y ago
  It's worth distinguishing between the tools that leave the data in your data warehouse (Caravel, Periscope, Mode, Looker (where I work)), and those that have their own data stores (Good Data, Qlikview, etc.) Tableau can connect directly to your datastore, but it's happier if it can operate on data that's stored locally in-memory. Anyway, the ones where bring your own database can scale as far as the database can bring you.
  - caravel 11y ago
    Note that Tableau doesn't play well with Presto which Airbnb uses extensively. No possibility of using the "live mode"
    - dwmintz 11y ago
      Not arguing that Airbnb should use Looker, but fwiw, Looker (where I work) does in fact connect to Presto fine.
- adrr 11y ago
  Looker performance comes from the underlying storage of data. You can store massive amounts of data on say a Red Shift cluster and still be performant. Most visualization tools that i know are directly tied to storage tier in terms of performance.
  - caravel 11y ago
    I don't think Looker supports Druid.
    - dwmintz 11y ago
      That's correct. We (I work at Looker) are written to leverage the analytical capabilities of SQL. So we support all kinds of SQL implementations, from totally standard (PG, MySQL, MS) to MPPs (Redshift, BigQuery) to SQL-on-Hadoop (Hive, Impala, Presto, Spark). But the JDBC -> Druid connectors that exist look pretty janky. So if someone builds a stable connector, I suspect we'd support it. But for the moment, no Druid.
andreasklinger 11y ago
I am pretty sure they used multiple (and then most likely at some point too many) of those tools
dwmintz 11y ago
Can't speak for Airbnb, but I'm not sure that any of the front-end clients that you mentioned (disclosure: I work at Looker), can talk to Druid. So if Airbnb already had a Druid warehouse in place, they may have decided it was easier to roll their own front-end than migrate to a different backend.
- caravel 11y ago
  Druid is definitely part of the equation. Larger, data-driven companies with significant engineering teams prefer not relying on 3rd party, closed-sourced vendors. That can represent a significant risk and a blockage for deeper integration with other internal applications when needed. Not that building always wins over buying, but the balance shifts relatively to the size of the company. Also, when using open source on the receiving end of the equation, you want to be a good citizen and contribute back to the ecosystem. It ties to pride, passion, and reflect a strong engineering culture, which can help with recruiting.
  - vgt 11y ago
    Your points resonate with me very strongly, but I do have a counter argument. I work on Google Cloud, where we have both closed-source - BigQuery, open source - Dataproc, and closed source that makes open source rock - Dataflow/Beam. There are merits to each. BigQuery is serverless and multi-tenant and can't really exist outside of Google Cloud due to its intrinsic dependency on low-level services that don't exist elsewhere (and for the same reasons we couldn't directly externalize Borg, choosing to create an OSS clone in Kubernetes). It is not unusual for me to hear from folks that they spent 6+ months building a performant and sizable, say Presto cluster. Then there's continuous management, tinkering, configuration, and optimization projects. I hear this from companies that one would consider sophisticated technologically. By contrast, 40% of all of BigQuery's Petabyte customers scale to these levels without ever talking to us. We just find them on consumption reports. On multiple occasions we've had "surprise" load tests of millions of rows per second streamed into BigQuery, and it just works. BigQuery is also HA out of the box at no additional cost, which is a great luxury. So sometimes if you need to scale analytics to Petabytes, the option is to just consume a managed and cost-effective service, or tinker with OSS, where there are significant operational tradeoffs. On the other hand, as you said, you build pride and culture. It's also far from an automatic shoo-in that OSS gives you better TCO (against the old closed-source guard, yes, but not so much BQ). Thus, the relationship between company size and value of technology can invert at higher levels. With all that, I'd love to see a Caravel-BQ plugin :) (PS. Kudos to Druid for introducing a Streaming ingest. BigQuery also sees value in Streaming ingest, GA-ing our own Streaming API in March of 2015, and Kudos to Airflow).
maerF0x0 11y ago
Few more for your list: Leftronic (service) or Grafana (package)
caravel 11y ago
Well none of the solutions mentionned are open source. Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can. More importantly, software for which we don't have control over the source is a risk. In this day and age anyone that cares enough should be able to push a bugfix/hotfix overnight. What if you'd have to wait for entire quarters or years for Tableau to parallelize their "live mode", or to get connectivity to Presto to work? What if you want to integrate a new type of visualization that isn't supported? What if you want to integrate with your anomaly detection framework or your A/B testing framework or other internal or external facing applications? Since this is a common need for most companies, it makes sense to have an open source solution that we can all use and collaborate on.
- timr 11y ago
  "Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying it from someone else is almost always cheaper than paying someone to maintaining it yourself, because expensive services are usually expensive for a good reason: they're niche, and finding someone with the expertise to build it is expensive. And having the source is for a product so that you can customize it is certainly a better answer, but it rarely happens, in practice. It's why we have gobs of open-source Apache-foundation products that nobody in their right mind wants to host in-house, unless they absolutely have to. Developers have a real, well-documented resistance to paying for things, and it sucks. Because in reality, most development of open-source tools happens when someone gets paid to maintain the tool. If they don't, the tool falls into disrepair. Open-source software isn't free -- it's just paid for by someone else.
  - caravel 11y ago
    There's also an in house cost in maintaining vendor software and integrating it. That's on top of the 20% usual maintenance fee. Building a community can also help distributing the costs. But more importantly it's a labour of collaboration instead of a client/vendor struggle.
ccozan 11y ago
It is even crowded than that on the commercial offering [0]. But having such a tool as Open Source benefits us all. Especially when trying to connect to obscure or non standard data sources. [0] https://en.wikipedia.org/wiki/Online_analytical_processing#Market_structure https://en.wikipedia.org/wiki/Online_analytical_processing#M...
tlrobinson 11y ago
To throw out an open source one: Metabase http://www.metabase.com http://www.metabase.com https://github.com/metabase/metabase https://github.com/metabase/metabase Disclaimer: I work on Metabase.
- gedrap 11y ago
  Not really related to your product, but I find the trend of using emojis in commit messages really distracting. It's probably okay-ish very very rarely for something huge, but commit log reading like slack chat... Em not sure about it.
hathym 11y ago
investor's money, baby!
ernestbro 11y ago
There is a study done by a market analyst (Wayne Eckerson) about Building vs Buying BI, he has some good insights based on surveys as to why some companies chose to build and some choose to buy 71% of those who chose to build the BI tools said they built because "We can customize the functionality better" 51% of those who buy say "Buying enables us to provide best-in-class BI functionality" The study: http://www.jaspersoft.com/sites/default/files/confirmation_files/embedded_bi_-_putting_reporting_and_analysis_everywhere.pdf http://www.jaspersoft.com/sites/default/files/confirmation_f...

ljk 11y ago

part of it is probably recruiting

nedwin 11y ago

aesthetics

phunge 11y ago

They mention that it was originally paired with Druid. The data volume that Druid excels at (and that AirBnB must have) is orders of magnitude larger than what Tableau&Looker do well with. It's probably just built for bigger-than-SQL OLAP usecases.

jsiegz 11y ago
Looker scales with plenty of data for us at Snapchat, it's more about the underlying database than the BI tool.
- MechanicalTwerk 11y ago
  Note that this user used to work at Looker.
  - jsiegz 11y ago
    yep i did.
dwmintz 11y ago
It's worth distinguishing between the tools that leave the data in your data warehouse (Caravel, Periscope, Mode, Looker (where I work)), and those that have their own data stores (Good Data, Qlikview, etc.) Tableau can connect directly to your datastore, but it's happier if it can operate on data that's stored locally in-memory. Anyway, the ones where bring your own database can scale as far as the database can bring you.
- caravel 11y ago
  Note that Tableau doesn't play well with Presto which Airbnb uses extensively. No possibility of using the "live mode"
  - dwmintz 11y ago
    Not arguing that Airbnb should use Looker, but fwiw, Looker (where I work) does in fact connect to Presto fine.
adrr 11y ago
Looker performance comes from the underlying storage of data. You can store massive amounts of data on say a Red Shift cluster and still be performant. Most visualization tools that i know are directly tied to storage tier in terms of performance.
- caravel 11y ago
  I don't think Looker supports Druid.
  - dwmintz 11y ago
    That's correct. We (I work at Looker) are written to leverage the analytical capabilities of SQL. So we support all kinds of SQL implementations, from totally standard (PG, MySQL, MS) to MPPs (Redshift, BigQuery) to SQL-on-Hadoop (Hive, Impala, Presto, Spark). But the JDBC -> Druid connectors that exist look pretty janky. So if someone builds a stable connector, I suspect we'd support it. But for the moment, no Druid.

andreasklinger 11y ago

I am pretty sure they used multiple (and then most likely at some point too many) of those tools

dwmintz 11y ago

Can't speak for Airbnb, but I'm not sure that any of the front-end clients that you mentioned (disclosure: I work at Looker), can talk to Druid. So if Airbnb already had a Druid warehouse in place, they may have decided it was easier to roll their own front-end than migrate to a different backend.

caravel 11y ago
Druid is definitely part of the equation. Larger, data-driven companies with significant engineering teams prefer not relying on 3rd party, closed-sourced vendors. That can represent a significant risk and a blockage for deeper integration with other internal applications when needed. Not that building always wins over buying, but the balance shifts relatively to the size of the company. Also, when using open source on the receiving end of the equation, you want to be a good citizen and contribute back to the ecosystem. It ties to pride, passion, and reflect a strong engineering culture, which can help with recruiting.
- vgt 11y ago
  Your points resonate with me very strongly, but I do have a counter argument. I work on Google Cloud, where we have both closed-source - BigQuery, open source - Dataproc, and closed source that makes open source rock - Dataflow/Beam. There are merits to each. BigQuery is serverless and multi-tenant and can't really exist outside of Google Cloud due to its intrinsic dependency on low-level services that don't exist elsewhere (and for the same reasons we couldn't directly externalize Borg, choosing to create an OSS clone in Kubernetes). It is not unusual for me to hear from folks that they spent 6+ months building a performant and sizable, say Presto cluster. Then there's continuous management, tinkering, configuration, and optimization projects. I hear this from companies that one would consider sophisticated technologically. By contrast, 40% of all of BigQuery's Petabyte customers scale to these levels without ever talking to us. We just find them on consumption reports. On multiple occasions we've had "surprise" load tests of millions of rows per second streamed into BigQuery, and it just works. BigQuery is also HA out of the box at no additional cost, which is a great luxury. So sometimes if you need to scale analytics to Petabytes, the option is to just consume a managed and cost-effective service, or tinker with OSS, where there are significant operational tradeoffs. On the other hand, as you said, you build pride and culture. It's also far from an automatic shoo-in that OSS gives you better TCO (against the old closed-source guard, yes, but not so much BQ). Thus, the relationship between company size and value of technology can invert at higher levels. With all that, I'd love to see a Caravel-BQ plugin :) (PS. Kudos to Druid for introducing a Streaming ingest. BigQuery also sees value in Streaming ingest, GA-ing our own Streaming API in March of 2015, and Kudos to Airflow).

maerF0x0 11y ago

Few more for your list: Leftronic (service) or Grafana (package)

caravel 11y ago

Well none of the solutions mentionned are open source. Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can. More importantly, software for which we don't have control over the source is a risk. In this day and age anyone that cares enough should be able to push a bugfix/hotfix overnight. What if you'd have to wait for entire quarters or years for Tableau to parallelize their "live mode", or to get connectivity to Presto to work? What if you want to integrate a new type of visualization that isn't supported? What if you want to integrate with your anomaly detection framework or your A/B testing framework or other internal or external facing applications? Since this is a common need for most companies, it makes sense to have an open source solution that we can all use and collaborate on.

timr 11y ago
"Free as in beer is one incentive as licenses are not cheap, and vendors know when they have you locked down and tend to milk everything they can." Free as in beer is never the answer. A project like this takes multiple engineer-years to build and maintain. That's hundreds of thousands of dollars, at least. How much is a site license? Are you sure? Have you negotiated the rate? Even for "expensive" services, buying it from someone else is almost always cheaper than paying someone to maintaining it yourself, because expensive services are usually expensive for a good reason: they're niche, and finding someone with the expertise to build it is expensive. And having the source is for a product so that you can customize it is certainly a better answer, but it rarely happens, in practice. It's why we have gobs of open-source Apache-foundation products that nobody in their right mind wants to host in-house, unless they absolutely have to. Developers have a real, well-documented resistance to paying for things, and it sucks. Because in reality, most development of open-source tools happens when someone gets paid to maintain the tool. If they don't, the tool falls into disrepair. Open-source software isn't free -- it's just paid for by someone else.
- caravel 11y ago
  There's also an in house cost in maintaining vendor software and integrating it. That's on top of the 20% usual maintenance fee. Building a community can also help distributing the costs. But more importantly it's a labour of collaboration instead of a client/vendor struggle.

ccozan 11y ago

It is even crowded than that on the commercial offering [0]. But having such a tool as Open Source benefits us all. Especially when trying to connect to obscure or non standard data sources. [0] https://en.wikipedia.org/wiki/Online_analytical_processing#Market_structure https://en.wikipedia.org/wiki/Online_analytical_processing#M...

tlrobinson 11y ago

To throw out an open source one: Metabase http://www.metabase.com http://www.metabase.com https://github.com/metabase/metabase https://github.com/metabase/metabase Disclaimer: I work on Metabase.

gedrap 11y ago
Not really related to your product, but I find the trend of using emojis in commit messages really distracting. It's probably okay-ish very very rarely for something huge, but commit log reading like slack chat... Em not sure about it.

hathym 11y ago

investor's money, baby!

ernestbro 11y ago

There is a study done by a market analyst (Wayne Eckerson) about Building vs Buying BI, he has some good insights based on surveys as to why some companies chose to build and some choose to buy 71% of those who chose to build the BI tools said they built because "We can customize the functionality better" 51% of those who buy say "Buying enables us to provide best-in-class BI functionality" The study: http://www.jaspersoft.com/sites/default/files/confirmation_files/embedded_bi_-_putting_reporting_and_analysis_everywhere.pdf http://www.jaspersoft.com/sites/default/files/confirmation_f...