19 ms·

Velox: An open-source unified execution engine

151 points by polyrand 4y ago

coding123 4y ago
it's airflow with more specifics around transportable data structures? instead of junky xcoms?
- gregwebs 4y ago
  Airflow has always been focused on workflows and it’s tasks don’t have to query data in an external system. You could have code running in airflow that runs a query that is executed by Velox.
injidup 4y ago
- pavlov 4y ago
  If that's the case, the domain is short for "Fine Bodies" presumably.
- dang 4y ago
  Please don't do this here.
  - injidup 4y ago
    Ok! Point taken. Sorry!
debarshri 4y ago
It sounds very similar to apache beam. You can actually create runners for various data management systems [1] [1] https://beam.apache.org/documentation/runners/ https://beam.apache.org/documentation/runners/
- KptMarchewa 4y ago
  Sounds like beam is something you can use on top - beam is more query planner that can translate it's plans to several other engines. Velox executes those plans.
gigatexal 4y ago
Cool that it’s being integrated into presto.
- KptMarchewa 4y ago
  Note that it's "fb presto" rather than more popular and widely developed Trino fork.
  - stevenmih 4y ago
    Just a clarification about "fb presto", there isn't a "fb presto", its Linux Foundation Presto, which has its own unique innovations and used reliably at all scales, including places like Meta, Uber, Alibaba, and more recently ByteDance. Linux Foundation Presto isn't controlled by one vendor, it has 10 member orgs overseeing the project, following Linux project practices, and doing community outreach like PrestoCon. See for yourself @ prestodb.io (Disclaimer: I'm Ahana CEO, providing a Cloud Managed Service for Presto-based SQL Lakehouses, and member of Linux Foundation's Presto Foundation)
mborch 4y ago
Interesting to see how Databricks reacts to this given that they have their own Project Lightspeed (replacing the Spark execution engine).
tinco 4y ago
"In common usage scenarios, Velox takes a fully optimized query plan as input and performs the described computation. Considering Velox does not provide a SQL parser, a dataframe layer, or a query optimizer, it is usually not meant to be used directly by end-users; rather, it is mostly used by developers integrating and optimizing their compute engines." So the way you use it is that you describe some computation over your data as a query plan, and you implement a dataframe layer so Velox knows how to retrieve data from your database, and then Velox will efficiently execute the query plan? But it doesn't even optimize, so the problem it solves is that these systems like Spark and Presto don't efficiently execute optimized queries? This world is very far removed from me, does anyone have a concrete example of how Velox might help them? Why is Velox better than both Presto worker and Spark engine. Aren't those core components of the system?
- scott_s 4y ago
  Disclaimer: I work at Meta. I don't work on Velox, but my work intersects with Velox in multiple ways. The short answer is consistent semantics. We have a large data warehouse, and several different engines that query that data warehouse. We want, as much as possible, consistent semantics across all of our engines for our users. That is, as much as possible, we want the same query to produce the same results on different surfaces. The value of that is so that users can draft a query on one surface, and use that same query elsewhere with confidence that they will get the same results. If we can consolidate on the same execution engine inside of our query engines, we can achieve that. Minor quibble on terminology: "Velox knows how to retrieve data from your database". I would instead say that Velox knows how to retrieve data from your storage. Velox is deeply integrated into the query engine, and the combination of the query engine and the storage is "the database." In large data warehouses, we've already separated storage from compute to achieve scalability. If this is all too abstract, think of this way: your query engine (such as Presto) is like a full computer system, while Velox is like the processor. Processors, by themselves, are not useful. They need to be attached to a motherboard which has RAM and connections to hard drives, GPUs and other external devices. Your query engine is like the computer system that contains that motherboard and all the components connected to it. There's enormous value in having multiple computer systems with different capabilities, but using the same kind of processor: you get consistent behavior when the capabilities are the same. Velox is that processor, ready to be plugged into different query engines.
marginalia_nu 4y ago
* [...] unified execution engine * accelerating data management systems * [...] streamlining their development * [...] consolidate and unify data management systems Can someone translate this to English? I can see and recognize the individual meanings of the words, but I don't understand what they're trying to say.
- bjackman 4y ago
  > Meta’s infrastructure plays an important role in supporting our products and services.
  - boesboes 4y ago
    lol, that's basically the definition of infrastructure
- jsty 4y ago
  The section titled "An Overview of Velox" gets into the meat of it - you give it some data and an optimised plan of the operators you want applied (expression, evaluation, aggregation, etc.) - Velox then executes that plan as efficiently as possible given the available compute resources. That way multiple top-level systems like analytics databases, dataframe implementations, etc. can all share the same underlying execution engine.
  - picardo 4y ago
    So it's like an operating system for cloud workloads?
    - aseipp 4y ago
      No, it's close to the core internals of an OLAP database. The "operators" it can execute in question are things like filter, join, aggregate, group by, projection (select), things of that nature. It makes sure to use available resources like SIMD and multithreading to do that efficiently. If you built a SQL parser -- and also the glue to create query plans from that -- you could attach it to Velox to do all that on some data source, for example. But you'd still need a storage layer (disk, s3) and also some kind of higher layer if you wanted to use multiple computers for a complete database. The query execution engine is a critical component, however.
nspattak 4y ago
could this be a name conflict with this https://www.thermofisher.com/order/catalog/product/VELOX https://www.thermofisher.com/order/catalog/product/VELOX ?
YetAnotherNick 4y ago
Is it same as YARN?
- cmollis 4y ago
  I think it's more like Databricks' photon.. a rewrite of the execution engine that can be plugged into existing spark deployments.
polskibus 4y ago
I see this as a continued effort of middleware being rewritten in C++, Rust and Go to replace Java - seems like common wisdom "Java can be as fast as C" has finally been abolished as this situation progresses (Kubernetes and other newer cloud middleware written in Go instead of Java, etc.)
- zaphar 4y ago
  I think it's a little more nuanced than that. Java was fast enough combined with security and safety guarantees to make it worth it in a total calculation. But Go and Rust are both faster with at least as good if not better safety guarantees so you are seeing a move away. Language choice is never wholly about speed. Java filled a niche of fast enough with certain guarantees and affordances to allow it to fill certain roles. C++ has gained a lot of affordances as well to improves safety as long as your can enforce their usage in the product so it's starting to eat into Java's market share as well. But I think long term Rust and Go will absorb more of what you would have done in Java than C++ will.
- liuliu 4y ago
  Containerization also helps the trend. Previously, you manage JVM runtime separately in your cluster and the deployment becomes simpler (just deploy the jars). Without containerization, maintaining separate runtime dependencies for compiled language is not as smooth. But containerization changed the equation. Now you deploy runtime dependencies along side your binary (if needed). It is actually a liability to deploy JVM runtime along side your jars now (actually, can you deploy java container images with JVM on the machine?)
- benreesman 4y ago
  I broadly agree, though I think you might be under-counting the degree to which accumulated ecosystem value drives this phenomenon, especially amongst a group of languages not known for strong mutual interoperability. Java is fucking fast, I imagine Cliff Click would like a word with someone arguing golang outperforms a well-tuned JVM. But the language isn’t aging the best and interop with non-JVM languages is, also not the best. golang seems pretty optimized for polyglot SOA-type setups where a comparatively modest amount of existing code compared to say C++ isn’t a real drawback because you’re hitting that stuff over the network anyways. Rust is probably the better language in C++ niches at this point when you don’t have a big C++ ecosystem investment and can do a more “greenfield”-type project. That can become a holy war and I’ll leave my position at: “there are good reasons to choose both”.
lioeters 4y ago
https://github.com/facebookincubator/velox https://github.com/facebookincubator/velox
rajko_rad 4y ago
Anyone know how this compares to Photon by Databricks? That’s probably the benchmark + arch comp I’d like to see… https://www.databricks.com/product/photon https://www.databricks.com/product/photon
deleted 4y ago
[deleted]
politician 4y ago
> Velox leverages numerous runtime optimizations, such as filter and conjunct reordering, key normalization for array and hash-based aggregations and joins, dynamic filter pushdown, and adaptive column prefetching. That's a strong set of capabilities. I'm excited to see where this goes -- this could catalyze a Cambrian explosion of data systems that offload execution to Velox.
liminal 4y ago
So this is an Apache Arrow database engine integrated into other databases? My main takeaway is that it's great to see more projects standardizing on Arrow and pushing it further down the stack.
_gabe_ 4y ago
> Ultimately, this fragmentation results in systems with different feature sets and inconsistent semantics — reducing the productivity of data users that need to interact with multiple engines to finish tasks. > In order to address these challenges and to create a stronger, more efficient data infrastructure for our own products and the world, Meta has created and open sourced Velox. Maybe I'm missing something here, but it sounds like a lot of separate services got created that solve the same or similar problem in slightly different ways. These services became hard to use because they were fragmented. So the solution is to keep all the services and build a complex service as a middle man? Why not unify the good parts of all the services into one central service? Then deprecate and transition off all the old fragmented ones? I understand that it's really hard to coordinate all of this and properly transition, but isn't the alternative of having to maintain many slightly different services (and now a complex middle man) more detrimental long term?
- entropicdrifter 4y ago
  >Why not unify the good parts of all the services into one central service? https://xkcd.com/927/ https://xkcd.com/927/
whoevercares 4y ago
Is this similar to Arrow datafusion but in C++? Tbh I think every hot new dataframe or analytics db has such components. The basic idea is not too different from the textbook at first glance.
whoevercares 4y ago
This seems like analogous to LLVM, looks like we could (finally) build various front ends for analytics on tops of this?

151 points by polyrand 4y ago

coding123 4y ago

it's airflow with more specifics around transportable data structures? instead of junky xcoms?

gregwebs 4y ago
Airflow has always been focused on workflows and it’s tasks don’t have to query data in an external system. You could have code running in airflow that runs a query that is executed by Velox.

injidup 4y ago

pavlov 4y ago
If that's the case, the domain is short for "Fine Bodies" presumably.
dang 4y ago
Please don't do this here.
- injidup 4y ago
  Ok! Point taken. Sorry!

debarshri 4y ago

It sounds very similar to apache beam. You can actually create runners for various data management systems [1] [1] https://beam.apache.org/documentation/runners/ https://beam.apache.org/documentation/runners/

KptMarchewa 4y ago
Sounds like beam is something you can use on top - beam is more query planner that can translate it's plans to several other engines. Velox executes those plans.

gigatexal 4y ago

Cool that it’s being integrated into presto.

KptMarchewa 4y ago
Note that it's "fb presto" rather than more popular and widely developed Trino fork.
- stevenmih 4y ago
  Just a clarification about "fb presto", there isn't a "fb presto", its Linux Foundation Presto, which has its own unique innovations and used reliably at all scales, including places like Meta, Uber, Alibaba, and more recently ByteDance. Linux Foundation Presto isn't controlled by one vendor, it has 10 member orgs overseeing the project, following Linux project practices, and doing community outreach like PrestoCon. See for yourself @ prestodb.io (Disclaimer: I'm Ahana CEO, providing a Cloud Managed Service for Presto-based SQL Lakehouses, and member of Linux Foundation's Presto Foundation)

mborch 4y ago

Interesting to see how Databricks reacts to this given that they have their own Project Lightspeed (replacing the Spark execution engine).

tinco 4y ago

"In common usage scenarios, Velox takes a fully optimized query plan as input and performs the described computation. Considering Velox does not provide a SQL parser, a dataframe layer, or a query optimizer, it is usually not meant to be used directly by end-users; rather, it is mostly used by developers integrating and optimizing their compute engines." So the way you use it is that you describe some computation over your data as a query plan, and you implement a dataframe layer so Velox knows how to retrieve data from your database, and then Velox will efficiently execute the query plan? But it doesn't even optimize, so the problem it solves is that these systems like Spark and Presto don't efficiently execute optimized queries? This world is very far removed from me, does anyone have a concrete example of how Velox might help them? Why is Velox better than both Presto worker and Spark engine. Aren't those core components of the system?

scott_s 4y ago
Disclaimer: I work at Meta. I don't work on Velox, but my work intersects with Velox in multiple ways. The short answer is consistent semantics. We have a large data warehouse, and several different engines that query that data warehouse. We want, as much as possible, consistent semantics across all of our engines for our users. That is, as much as possible, we want the same query to produce the same results on different surfaces. The value of that is so that users can draft a query on one surface, and use that same query elsewhere with confidence that they will get the same results. If we can consolidate on the same execution engine inside of our query engines, we can achieve that. Minor quibble on terminology: "Velox knows how to retrieve data from your database". I would instead say that Velox knows how to retrieve data from your storage. Velox is deeply integrated into the query engine, and the combination of the query engine and the storage is "the database." In large data warehouses, we've already separated storage from compute to achieve scalability. If this is all too abstract, think of this way: your query engine (such as Presto) is like a full computer system, while Velox is like the processor. Processors, by themselves, are not useful. They need to be attached to a motherboard which has RAM and connections to hard drives, GPUs and other external devices. Your query engine is like the computer system that contains that motherboard and all the components connected to it. There's enormous value in having multiple computer systems with different capabilities, but using the same kind of processor: you get consistent behavior when the capabilities are the same. Velox is that processor, ready to be plugged into different query engines.

marginalia_nu 4y ago

* [...] unified execution engine * accelerating data management systems * [...] streamlining their development * [...] consolidate and unify data management systems Can someone translate this to English? I can see and recognize the individual meanings of the words, but I don't understand what they're trying to say.

bjackman 4y ago
> Meta’s infrastructure plays an important role in supporting our products and services.
- boesboes 4y ago
  lol, that's basically the definition of infrastructure
jsty 4y ago
The section titled "An Overview of Velox" gets into the meat of it - you give it some data and an optimised plan of the operators you want applied (expression, evaluation, aggregation, etc.) - Velox then executes that plan as efficiently as possible given the available compute resources. That way multiple top-level systems like analytics databases, dataframe implementations, etc. can all share the same underlying execution engine.
- picardo 4y ago
  So it's like an operating system for cloud workloads?
  - aseipp 4y ago
    No, it's close to the core internals of an OLAP database. The "operators" it can execute in question are things like filter, join, aggregate, group by, projection (select), things of that nature. It makes sure to use available resources like SIMD and multithreading to do that efficiently. If you built a SQL parser -- and also the glue to create query plans from that -- you could attach it to Velox to do all that on some data source, for example. But you'd still need a storage layer (disk, s3) and also some kind of higher layer if you wanted to use multiple computers for a complete database. The query execution engine is a critical component, however.

nspattak 4y ago

could this be a name conflict with this https://www.thermofisher.com/order/catalog/product/VELOX https://www.thermofisher.com/order/catalog/product/VELOX ?

YetAnotherNick 4y ago

Is it same as YARN?

cmollis 4y ago
I think it's more like Databricks' photon.. a rewrite of the execution engine that can be plugged into existing spark deployments.

polskibus 4y ago

I see this as a continued effort of middleware being rewritten in C++, Rust and Go to replace Java - seems like common wisdom "Java can be as fast as C" has finally been abolished as this situation progresses (Kubernetes and other newer cloud middleware written in Go instead of Java, etc.)

zaphar 4y ago
I think it's a little more nuanced than that. Java was fast enough combined with security and safety guarantees to make it worth it in a total calculation. But Go and Rust are both faster with at least as good if not better safety guarantees so you are seeing a move away. Language choice is never wholly about speed. Java filled a niche of fast enough with certain guarantees and affordances to allow it to fill certain roles. C++ has gained a lot of affordances as well to improves safety as long as your can enforce their usage in the product so it's starting to eat into Java's market share as well. But I think long term Rust and Go will absorb more of what you would have done in Java than C++ will.
liuliu 4y ago
Containerization also helps the trend. Previously, you manage JVM runtime separately in your cluster and the deployment becomes simpler (just deploy the jars). Without containerization, maintaining separate runtime dependencies for compiled language is not as smooth. But containerization changed the equation. Now you deploy runtime dependencies along side your binary (if needed). It is actually a liability to deploy JVM runtime along side your jars now (actually, can you deploy java container images with JVM on the machine?)
benreesman 4y ago
I broadly agree, though I think you might be under-counting the degree to which accumulated ecosystem value drives this phenomenon, especially amongst a group of languages not known for strong mutual interoperability. Java is fucking fast, I imagine Cliff Click would like a word with someone arguing golang outperforms a well-tuned JVM. But the language isn’t aging the best and interop with non-JVM languages is, also not the best. golang seems pretty optimized for polyglot SOA-type setups where a comparatively modest amount of existing code compared to say C++ isn’t a real drawback because you’re hitting that stuff over the network anyways. Rust is probably the better language in C++ niches at this point when you don’t have a big C++ ecosystem investment and can do a more “greenfield”-type project. That can become a holy war and I’ll leave my position at: “there are good reasons to choose both”.

lioeters 4y ago

https://github.com/facebookincubator/velox https://github.com/facebookincubator/velox

rajko_rad 4y ago

Anyone know how this compares to Photon by Databricks? That’s probably the benchmark + arch comp I’d like to see… https://www.databricks.com/product/photon https://www.databricks.com/product/photon

deleted 4y ago

[deleted]

politician 4y ago

> Velox leverages numerous runtime optimizations, such as filter and conjunct reordering, key normalization for array and hash-based aggregations and joins, dynamic filter pushdown, and adaptive column prefetching. That's a strong set of capabilities. I'm excited to see where this goes -- this could catalyze a Cambrian explosion of data systems that offload execution to Velox.

liminal 4y ago

So this is an Apache Arrow database engine integrated into other databases? My main takeaway is that it's great to see more projects standardizing on Arrow and pushing it further down the stack.

_gabe_ 4y ago

> Ultimately, this fragmentation results in systems with different feature sets and inconsistent semantics — reducing the productivity of data users that need to interact with multiple engines to finish tasks. > In order to address these challenges and to create a stronger, more efficient data infrastructure for our own products and the world, Meta has created and open sourced Velox. Maybe I'm missing something here, but it sounds like a lot of separate services got created that solve the same or similar problem in slightly different ways. These services became hard to use because they were fragmented. So the solution is to keep all the services and build a complex service as a middle man? Why not unify the good parts of all the services into one central service? Then deprecate and transition off all the old fragmented ones? I understand that it's really hard to coordinate all of this and properly transition, but isn't the alternative of having to maintain many slightly different services (and now a complex middle man) more detrimental long term?

entropicdrifter 4y ago
>Why not unify the good parts of all the services into one central service? https://xkcd.com/927/ https://xkcd.com/927/

whoevercares 4y ago

Is this similar to Arrow datafusion but in C++? Tbh I think every hot new dataframe or analytics db has such components. The basic idea is not too different from the textbook at first glance.

whoevercares 4y ago

This seems like analogous to LLVM, looks like we could (finally) build various front ends for analytics on tops of this?