5 ms·

Velox: Meta's Unified Execution Engine [pdf]

99 points by luu 3y ago

pvg 3y ago
A thread from late 2022: https://news.ycombinator.com/item?id=32673873 https://news.ycombinator.com/item?id=32673873
jauntywundrkind 3y ago
Python's Substrait seems like the biggest/most-used competitor-ish out there. I'd love some compare & contrast; my sense is that Substrait has a smaller ambition, more wants to be a language for talking about execution rather than a full on optimization/execution engine. https://github.com/substrait-io/substrait https://github.com/substrait-io/substrait . (Edit: ah, there's a recent talk discussing PyVelox trying to get Substrait integration. https://www.youtube.com/watch?v=l_kHxkGkNRg#t=18m22s https://www.youtube.com/watch?v=l_kHxkGkNRg#t=18m22s . However there's also discussion about the un-maintainedness of some of the current Substrait work here; unclear status. https://github.com/facebookincubator/velox/issues/8895 https://github.com/facebookincubator/velox/issues/8895) We can also see from the Apache Arrow DataFusion discussion that they too see themselves as a bit of a Velox competitor. https://github.com/apache/arrow-datafusion/discussions/6441 https://github.com/apache/arrow-datafusion/discussions/6441 It's cool to see this space mature. I like that even Velox sees that Apache Arrow (underlying Apache Arrow DataFusion too) is industry standard tech that they ought work with. https://engineering.fb.com/2024/02/20/developer-tools/velox-apache-arrow-15-composable-data-management/ https://engineering.fb.com/2024/02/20/developer-tools/velox-... Theres a solid Influx post talks to some of how they are composing the assorted technologies to build they next gen 3.0, which I find helpful for getting a sense of how all the pieces of a modern high-performance data engine slot together. https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/ https://www.influxdata.com/blog/flight-datafusion-arrow-parq...
- kristjansson 3y ago
  I think you're right - Substrait wants to sit above something like Velox. The closest comparison is probably Databricks Photon[1], but that's proprietary. [1]: https://www.databricks.com/product/photon https://www.databricks.com/product/photon
- threeseed 3y ago
  A lot of it is just trying to build a more modular Spark. Which is nice and all but for most companies they want something integrated so a lot of these projects despite promises haven't really seen that much traction. Especially when every cloud provider has a fully supported Spark platform available.
zX41ZdbW 3y ago
Many ideas look like they were influenced by ClickHouse, and some are direct copies. I'm surprised they didn't provide references to ClickHouse, where the implementations are proven in production in the first place.
- pedroerp 3y ago
  Vectorization was never invented by Clickhouse; they all root at the work done by CWI in MonetDB, then VectorWise, then more recently in DuckDB (and Velox). Velox also does not claim to have invented any of these techniques; the novelty claimed by the project is to do so in a modular way so that it can be reused while building any other engines (engine-agnostic), following any SQL dialect (dialect-agnostic). Could you list the "copied" ideas you are referring to?
- gaogao 3y ago
  Could you be specific about which ideas you think were influenced by ClickHouse specifically and not Presto or DuckDB or Spark?
- deleted 3y ago
  [deleted]
redskyluan 3y ago
Velox could be competitor of datafusion. It is more focus on execution engine and could be great to integrate to other high performance databases. Database will be split into pieces and rebuild!
- sakras 3y ago
  Yes this has been an up-and-coming theme in the data science world. Arrow for the data format, Ibis for the API, Acero/Velox/DataFusion/DuckDB/Polars for execution, Substrait for the query plan representation, etc.
  - soulbadguy 3y ago
    Isn't spark already providing a bundled and somewhat functional version of all of this ?
    - sakras 3y ago
      Right, so I think the idea is to "unbundle" this so that you can compose your own data analytics engine.
- redskyluan 3y ago
  By the way, we are looking for talents of velox, data fusion, ducdb, clickhouse to build the world's fastest vector db milvus(mainly on search side). Contact me if you have this background!
sgt101 3y ago
I wonder how many of this sort of FAANG project really get used where they are built. I went for an interview at a FAANG years ago to work on a very big consumer product (when it was in relative infancy) and expected to find a hyper tech data backend to use... they told me that they were using mySQL. I didn't get the job so maybe they were just joking around with me - but the general despair that they evinced about their data situation makes me wonder!
- bezosdontpipme 3y ago
  I can neither confirm nor deny that S3’s global bucket database is actually just MySql (with a lil bit of special sauce)
  - sgt101 3y ago
    tbh my general response to all data questions is "use postgres". It does happen that someone comes back with a good reason why that would be a bad idea, but it's not frequent! mySQL == Oracle now... so bad on theological grounds.
    - astrange 3y ago
      You can use MariaDB.
  - nonrandomstring 3y ago
    And why ever not? It's a perfectly good solution, no? What the GP alludes to is interesting though - mythologising of organisations, brands and names. Spend enough time with "famous" people, "big names", centres of power and prominence and you quickly see everyone is just ordinary dudes doing ordinary things with ordinary gear. But for some reason there's fuck loads of money and attention, and sometimes cloying paranoia and adulation floating around. Sure, right out on the periphery are a noble few who play with particle accelerators, spaceships and bunker supercomputers. But then, that's just a day job too. True genius/exceptionalism is rare and found in the unexpected places. The rest is conjured out of thin air by marketing and PR people, the press, and commentators. They are the ones who need the big legend.
sakras 3y ago
My general take is that while the idea of composability is good, the implementations of these things are just frankly not of high quality. Velox/Acero in particular are all plagued by what I've come to call "Java syndrome", where everything is written as idiomatic Java but with C++ syntax. Virtual methods, std::shared_ptr galore (in lieu of garbage collection), random heap allocations, etc. As a result these systems tend to be bloated and significantly slower than they need to be. DuckDB is good though, and I predict its quality of implementation will keep "monolithic databases" relevant for a while longer.
- pjmlp 3y ago
  Basically 1990's C++ before Java was invented, as proven in any C++ GUI framework that has survived to our days. I really take issue with people calling "Java code" to what used to be quite common C++ code from CFront 2.0 until C++11 came to be. Java is the outcome of C++'s programing practices before 1996, with a flavour of Objective-C semantics on top. To pretend those communities aren't responsible for those practices in first place is not being honest where they came from.
- rfoo 3y ago
  Acero is indeed plagued by "Java syndrome". However honestly Velox looks better, it's not as good as ClickHouse but working on it does not leave a "bruh Apache Arrow bruh shared_ptr everywhere bruh" bad taste in my mouth. And it does not use Apache Arrow C++, it implemented its own with Arrow compatible memory layout.
- yencabulator 3y ago
  > DuckDB is good though, and I predict its quality of implementation DuckDB has segfaulted a lot for me. That just simply bodes horribly poorly on the future of a C++ codebase. Datafusion has been a pretty pleasant experience.
HermitX 3y ago
To the best of my knowledge, Meta has significantly reduced its investment in the Velox project. Apart from Meta, I'm not aware of any other major company that really uses Velox in a production environment. Frankly speaking, Velox may have already missed the window of opportunity for rapid development. If you're looking for a vectorized execution engine, you could consider ClickHouse (www.clickhouse.com) or StarRocks (www.starrocks.io). If your data analysis scenarios require more multi-table join operations, StarRocks is clearly a better choice.
- pedroerp 3y ago
  This thread clearly has folks from Clickhouse trying to talk down the project. Could you provide any data or evidence to support your claim?
- kgp7 3y ago
  This isnt really true, Meta if anything has doubled down on Velox.
  - kamikaz1k 3y ago
    Curious if you’re both right. Could it be because of the orgs involved? Which teams are using it? What products?
    - kgp7 3y ago
      Presto is actively switching to use Velox as the backend ( https://github.com/prestodb/presto/tree/master/presto-native-execution https://github.com/prestodb/presto/tree/master/presto-native... ) . It is also being used extensively internally, again the paper describes these and their usages have grown , not reduced.

99 points by luu 3y ago

pvg 3y ago

A thread from late 2022: https://news.ycombinator.com/item?id=32673873 https://news.ycombinator.com/item?id=32673873

Python's Substrait seems like the biggest/most-used competitor-ish out there. I'd love some compare & contrast; my sense is that Substrait has a smaller ambition, more wants to be a language for talking about execution rather than a full on optimization/execution engine. https://github.com/substrait-io/substrait https://github.com/substrait-io/substrait . (Edit: ah, there's a recent talk discussing PyVelox trying to get Substrait integration. https://www.youtube.com/watch?v=l_kHxkGkNRg#t=18m22s https://www.youtube.com/watch?v=l_kHxkGkNRg#t=18m22s . However there's also discussion about the un-maintainedness of some of the current Substrait work here; unclear status. https://github.com/facebookincubator/velox/issues/8895 https://github.com/facebookincubator/velox/issues/8895) We can also see from the Apache Arrow DataFusion discussion that they too see themselves as a bit of a Velox competitor. https://github.com/apache/arrow-datafusion/discussions/6441 https://github.com/apache/arrow-datafusion/discussions/6441 It's cool to see this space mature. I like that even Velox sees that Apache Arrow (underlying Apache Arrow DataFusion too) is industry standard tech that they ought work with. https://engineering.fb.com/2024/02/20/developer-tools/velox-apache-arrow-15-composable-data-management/ https://engineering.fb.com/2024/02/20/developer-tools/velox-... Theres a solid Influx post talks to some of how they are composing the assorted technologies to build they next gen 3.0, which I find helpful for getting a sense of how all the pieces of a modern high-performance data engine slot together. https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/ https://www.influxdata.com/blog/flight-datafusion-arrow-parq...

kristjansson 3y ago
I think you're right - Substrait wants to sit above something like Velox. The closest comparison is probably Databricks Photon[1], but that's proprietary. [1]: https://www.databricks.com/product/photon https://www.databricks.com/product/photon
threeseed 3y ago
A lot of it is just trying to build a more modular Spark. Which is nice and all but for most companies they want something integrated so a lot of these projects despite promises haven't really seen that much traction. Especially when every cloud provider has a fully supported Spark platform available.

zX41ZdbW 3y ago

Many ideas look like they were influenced by ClickHouse, and some are direct copies. I'm surprised they didn't provide references to ClickHouse, where the implementations are proven in production in the first place.

pedroerp 3y ago
Vectorization was never invented by Clickhouse; they all root at the work done by CWI in MonetDB, then VectorWise, then more recently in DuckDB (and Velox). Velox also does not claim to have invented any of these techniques; the novelty claimed by the project is to do so in a modular way so that it can be reused while building any other engines (engine-agnostic), following any SQL dialect (dialect-agnostic). Could you list the "copied" ideas you are referring to?
gaogao 3y ago
Could you be specific about which ideas you think were influenced by ClickHouse specifically and not Presto or DuckDB or Spark?
deleted 3y ago
[deleted]

redskyluan 3y ago

Velox could be competitor of datafusion. It is more focus on execution engine and could be great to integrate to other high performance databases. Database will be split into pieces and rebuild!

sakras 3y ago
Yes this has been an up-and-coming theme in the data science world. Arrow for the data format, Ibis for the API, Acero/Velox/DataFusion/DuckDB/Polars for execution, Substrait for the query plan representation, etc.
- soulbadguy 3y ago
  Isn't spark already providing a bundled and somewhat functional version of all of this ?
  - sakras 3y ago
    Right, so I think the idea is to "unbundle" this so that you can compose your own data analytics engine.
redskyluan 3y ago
By the way, we are looking for talents of velox, data fusion, ducdb, clickhouse to build the world's fastest vector db milvus(mainly on search side). Contact me if you have this background!

sgt101 3y ago

I wonder how many of this sort of FAANG project really get used where they are built. I went for an interview at a FAANG years ago to work on a very big consumer product (when it was in relative infancy) and expected to find a hyper tech data backend to use... they told me that they were using mySQL. I didn't get the job so maybe they were just joking around with me - but the general despair that they evinced about their data situation makes me wonder!

bezosdontpipme 3y ago
I can neither confirm nor deny that S3’s global bucket database is actually just MySql (with a lil bit of special sauce)
- sgt101 3y ago
  tbh my general response to all data questions is "use postgres". It does happen that someone comes back with a good reason why that would be a bad idea, but it's not frequent! mySQL == Oracle now... so bad on theological grounds.
  - astrange 3y ago
    You can use MariaDB.
- nonrandomstring 3y ago
  And why ever not? It's a perfectly good solution, no? What the GP alludes to is interesting though - mythologising of organisations, brands and names. Spend enough time with "famous" people, "big names", centres of power and prominence and you quickly see everyone is just ordinary dudes doing ordinary things with ordinary gear. But for some reason there's fuck loads of money and attention, and sometimes cloying paranoia and adulation floating around. Sure, right out on the periphery are a noble few who play with particle accelerators, spaceships and bunker supercomputers. But then, that's just a day job too. True genius/exceptionalism is rare and found in the unexpected places. The rest is conjured out of thin air by marketing and PR people, the press, and commentators. They are the ones who need the big legend.

sakras 3y ago

My general take is that while the idea of composability is good, the implementations of these things are just frankly not of high quality. Velox/Acero in particular are all plagued by what I've come to call "Java syndrome", where everything is written as idiomatic Java but with C++ syntax. Virtual methods, std::shared_ptr galore (in lieu of garbage collection), random heap allocations, etc. As a result these systems tend to be bloated and significantly slower than they need to be. DuckDB is good though, and I predict its quality of implementation will keep "monolithic databases" relevant for a while longer.

pjmlp 3y ago
Basically 1990's C++ before Java was invented, as proven in any C++ GUI framework that has survived to our days. I really take issue with people calling "Java code" to what used to be quite common C++ code from CFront 2.0 until C++11 came to be. Java is the outcome of C++'s programing practices before 1996, with a flavour of Objective-C semantics on top. To pretend those communities aren't responsible for those practices in first place is not being honest where they came from.
rfoo 3y ago
Acero is indeed plagued by "Java syndrome". However honestly Velox looks better, it's not as good as ClickHouse but working on it does not leave a "bruh Apache Arrow bruh shared_ptr everywhere bruh" bad taste in my mouth. And it does not use Apache Arrow C++, it implemented its own with Arrow compatible memory layout.
yencabulator 3y ago
> DuckDB is good though, and I predict its quality of implementation DuckDB has segfaulted a lot for me. That just simply bodes horribly poorly on the future of a C++ codebase. Datafusion has been a pretty pleasant experience.

HermitX 3y ago

To the best of my knowledge, Meta has significantly reduced its investment in the Velox project. Apart from Meta, I'm not aware of any other major company that really uses Velox in a production environment. Frankly speaking, Velox may have already missed the window of opportunity for rapid development. If you're looking for a vectorized execution engine, you could consider ClickHouse (www.clickhouse.com) or StarRocks (www.starrocks.io). If your data analysis scenarios require more multi-table join operations, StarRocks is clearly a better choice.

pedroerp 3y ago
This thread clearly has folks from Clickhouse trying to talk down the project. Could you provide any data or evidence to support your claim?
kgp7 3y ago
This isnt really true, Meta if anything has doubled down on Velox.
- kamikaz1k 3y ago
  Curious if you’re both right. Could it be because of the orgs involved? Which teams are using it? What products?
  - kgp7 3y ago
    Presto is actively switching to use Velox as the backend ( https://github.com/prestodb/presto/tree/master/presto-native-execution https://github.com/prestodb/presto/tree/master/presto-native... ) . It is also being used extensively internally, again the paper describes these and their usages have grown , not reduced.