15 ms·
Airbnb open-sources Caravel: data exploration and visualization platform
- jedisct1 11y agoWasn't Panoramix already opensource?
- caravel 11y agoIt was (Panoramix got renamed to Caravel), it's just officially supported, maintained an grown by Airbnb now.
- deleted 11y ago[deleted]
- cauthon 11y agoWhat's the name of this style of plot? https://camo.githubusercontent.com/c22acad6c1302c5da3236cb8eea816013a866006/687474703a2f2f692e696d6775722e636f6d2f44326b5a4c37712e706e67 https://camo.githubusercontent.com/c22acad6c1302c5da3236cb8e...
- RobPfeifer 11y agoSankey Diagrams. Useful for complex funnels!
- bduerst 11y agoWe always just called them flow diagrams. I didn't know they had a specific name.
- deleted 11y ago[deleted]
- divideby0 11y agoI believe it's called a Sankey diagram: http://bl.ocks.org/d3noob/5028304 http://bl.ocks.org/d3noob/5028304
- devy 11y agoI believe it's called "Sankey Diagram" as denoted in the dropdown menu on the upper top left. Here is the original demo[1] from Mike Bostock, D3's author. [1]: https://bost.ocks.org/mike/sankey/ https://bost.ocks.org/mike/sankey/
- nthitz 11y agoThough they have been around much longer than D3 has... https://en.wikipedia.org/wiki/Sankey_diagram https://en.wikipedia.org/wiki/Sankey_diagram
- simonsarris 11y agoSankey: http://gojs.net/latest/samples/sankey.html http://gojs.net/latest/samples/sankey.html
- kayhi 11y agoWhere is the data from? It's a great overview of CO2 emissions, etc.
- mooneater 11y agoI really need a great data explorer/dashboard for my postgres-based systems. I was going to use shiny but this looks really nice -- I hope the docs can be built out very soon. Can anyone comment on other competing products? In the commercial space, I like Looker but its too pricey.
- jonbishop 11y agoHey! I run marketing for Periscope Data, a data explorer/dashboard product. We have a lot of customers using postgres and get compared to Looker a lot. We focus on optimizing for the analyst whereas tools like Looker focus on business users. We have a lot of features for business users, but chart creation is all SQL based. Our site is here: https://www.periscopedata.com/ https://www.periscopedata.com/ and if you have any questions, shoot me an email at jon@periscopedata.com.
- javiercr 11y agoTake a look to http://www.metabase.com/ http://www.metabase.com/ (open source, made by Expa, Uber's co-founder incubator).
- dacort 11y agoSecond this recommendation. Very easy to get up and running - may not be able to handle some complex use cases, but for the basics it's fantastic.
- johnieeboy 11y agoseems a little bit sparse on the documentation or am I missing something?
- caravel 11y agoWe'll be providing short user training videos very soon.
- teej 11y agoAs someone who is the core audience for this tool can I say that I strongly prefer clear documentation over videos? Videos are way too hard to maintain and end up being stale the minute after you post them in fast-moving projects. I can't text-search a video and I can't be linked directly to an answer in a StackOverflow response. Written documentation is vastly superior to videos in my opinion.
- glwtta 11y agoseconded. videos are not useful as documentation.
- shostack 11y agoTo add--you can also skim text MUCH more quickly for the piece you want compared to video. I absolutely loathe video for any analytics-related documentation. It rarely adds any real value over text outside of live webinars where I can ask questions.
- dacort 11y agoWould very much appreciate that as well as walk-through docs. Got it up and running easily enough, and connected to Redshift. But seemed like creating a new "slice" required custom JSON params to define it. Unless I missed something? edit: yep, missed something. Can "explore" a table by clicking it's link in the table listing.
- robroy72 11y agoIs there an API for building viz in code similar to something like Bokeh, or is it end user viz design only?
- educar 11y agoIs this like piwik?
- teej 11y agoPiwik is more of a Google Analytics replacement. It's a package that contains a data visualization platform, a data storage engine, and a data emitter (website tag) all in one. This is just a data visualization platform. You need to bring your own data store and data.
- don_draper 11y agoThe code is so clean and simple. This is great PR for the company. I want to work there.
- umhan35 11y agoNot from the number of lines of code. Some of them exceeds 1000.
- vermontdevil 11y ago# of lines does not indicate good or poor coding
- ultimoo 11y agoI think so too! I have done a fair bit of Ruby a few years ago but I'm new to python CRUD apps and trying to improve my knowledge here. Is defining all models in the same file[1] conventional in python apps? Rails used to have separate files for each model. And most Ruby apps that I have seen advocate the one-class-one-file convention. [1] https://github.com/airbnb/caravel/blob/master/caravel/models.py https://github.com/airbnb/caravel/blob/master/caravel/models...
- Keats 11y agoIt depends how many models but usually 1 app models == 1 file (taking the app structure from django) unless they are expected to grow. For example you have a comment app that could contain several models: Comment, Thread, Report, etc those can be in the same file. To continue on the django example, I would personally prefer having a models folder in the comments app and one file per model as some can get really big. I also do 1 file / model in Flask, minus some specific cases where it just makes sense to have them in the same file
- tedmiston 11y agoAll models in one models.py file is common for Flask and Django. If you use multiple apps within one Django project or the equivalent in Flask (Blueprints), that extends to one models.py per app (where a "project" is a collection of "apps"). Sometimes you'll see one file model per (with a models/__init__.py that imports them for use). While I think it keep dependency imports for each model very cleanly separated, you end up having a lot of redundancy importing the same basic pieces in every model file.
- twakefield 11y agoIf anyone involved with this is around, I'm curious why Airbnb would build something like this - cost, performance, features, all of the above? Data querying and visualization is a pretty crowded field with a lot of commercial options to choose from[1]. I'm not knocking Caravel (it looks amazing) just curious why build vs buy in this case. [1] Tableau, Looker, Periscope, Chartio, Qlikview, Gooddata are just some that come to mind.
- ljk 11y agopart of it is probably recruiting
- nedwin 11y agoaesthetics
- phunge 11y agoThey mention that it was originally paired with Druid. The data volume that Druid excels at (and that AirBnB must have) is orders of magnitude larger than what Tableau&Looker do well with. It's probably just built for bigger-than-SQL OLAP usecases.
- jsiegz 11y agoLooker scales with plenty of data for us at Snapchat, it's more about the underlying database than the BI tool.
- MechanicalTwerk 11y agoNote that this user used to work at Looker.
- jsiegz 11y agoyep i did.
- dwmintz 11y ago
- polartx 11y agowhat are the supported data sources? I saw SQL tables and I imagine flat files, can you write calls to web service endpoints?
- wesd 11y agoFrom the readme file: Database Support Caravel was originally designed on top of Druid.io, but quickly broadened its scope to support other databases through the use of SqlAlchemy, a Python ORM that is compatible with most common databases[1]. [1]http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html
- tedmiston 11y agoSpecifically, SQLAlchemy includes dialects out of the box for: Firebird, Microsoft SQL Server, MySQL, Oracle, PostgreSQL, SQLite, Sybase. http://docs.sqlalchemy.org/en/rel_1_0/dialects/index.html http://docs.sqlalchemy.org/en/rel_1_0/dialects/index.html
- infinite8s 11y agoAnd has adapters for Hive, Presto, Redshift and Google BigQuery.
- vhhuhhfryuhgfh 11y agoAny pictures anywhere?
- polskibus 11y agoI'm wondering - what's the effort required to build such a BI tool (tables, charts, maps) these days assuming reusing open source components and focusing on SQL-speaking datastores? Could a small team of experienced devs accomplish such a feat in a year?
- markovbling 11y agoAwesome - great work! A tutorial on how to link it to a mysql database would be greatly appreciated :)
- tedmiston 11y agoThey're using SQLAlchemy as a database abstraction layer which supports MySQL out of the box. So, you just need to set the config param SQLALCHEMY_DATABASE_URI like this: https://github.com/airbnb/caravel/blob/1b4e750b2aa111445703dca07fe929f7ffde38c9/caravel/config.py#L31 https://github.com/airbnb/caravel/blob/1b4e750b2aa111445703d... The configuration guide explains it further: https://github.com/airbnb/caravel/blob/master/docs/installation.rst#configuration https://github.com/airbnb/caravel/blob/master/docs/installat...
- tedmiston 11y agoI really like the Python style you guys have adopted. Grouping imports into: standard lib, third party, local is a strong pattern that I don't see done consistently in many repos. Likewise with your use of wrapping long imports with ()s and a single tab. Any chance of sharing your Python style guide? My startup is Python based (Django and Flask) and would really appreciate it!
- kevinastone 11y agoThe import ordering is pretty common: https://google.github.io/styleguide/pyguide.html?showone=Imports_formatting#Imports_formatting https://google.github.io/styleguide/pyguide.html?showone=Imp...
- flashman 11y agoHaving some trouble getting it working on Windows, which I see you don't currently support (I need to create caravel_config.py to get past the fabmanager installation step). This looks really interesting but I might wait until someone has posted Windows instructions.
- robroy72 11y agoI got it to work on windows using the Anaconda Python 2.7 installation; keep in mind that the caravel commands in the docs have to be run from your install dir; e.g. change dir to <yourPythonInstallDir>\Lib\site-packages\caravel\bin then run as python caravel db upgrade
- dschiptsov 11y agoIt is not Java, but Python. What a surprise!
- arikfr 11y agoHave you considered using Re:dash[1] before writing your own tool? [1] https://github.com/getredash/redash https://github.com/getredash/redash
- gorkemcetin 11y agoRedash is around for 2.5 years only and probably AirBNB engs thought it was too early to have a look at.
- kfk 11y agoHey, this is very interesting, I will take a look. I am trying to shift a whole $700m division and then hopefully $3b segment on a new workflow paradigm for data (automatically refreshed dashboards instead of sending around excel files, focus on building models and not data pasting in spreadsheets) and unfortunately for now Tableau is my only option. I feel very uncomfortable going with a closed solution since I know that we will have lots of edge cases and that being able to do your own coding is in the end the best way to deal with those. License cost is also incredibly high, we are talking $200 per user per year at minimum, that means 200 to 400 thousand per year for a big organization between 1000 and 200 users.
- dwmintz 11y agoI'm a huge proponent of the idea of centralizing the data model. That's the core idea behind Looker (where I just came to work after 3 years as a customer), and I agree it's a hugely powerful change from the world of everybody-in-their-own spreadsheet. On your other point, though, to echo the build vs. buy discussion from above, I think it's a bit misleading to say "oh, we'll just use an open-source solution and that'll be cheaper." Because if open source means a couple of internal developers and an analyst, that's easily $300k+/year in salaries that you might not spend if you were using a vendor. Anyway, given your particular statement of the problem you're facing, I'd humbly suggest you take a look at Looker. The data modeling layer that's core to Looker is meant to solve EXACTLY that problem, by leaving your data where it lives and then embedding your business logic in the layer that sits between end users and the data.
- uberneo 11y agoIts Apache licensed .. so it means can i use it directly in my company replacing the existing commercial products like Tableau / Looker?
- gavin6252 11y agoGreat looking product! Has anyone figured out how to join tables yet, or do you need to define views in your sql database?