4 ms·
* [...] unified execution engine * accelerating data management systems * [...] streamlining their development * [...] consolidate and unify data management
by marginalia_nu 4y ago
* [...] unified execution engine
* accelerating data management systems
* [...] streamlining their development
* [...] consolidate and unify data management systems
Can someone translate this to English? I can see and recognize the individual meanings of the words, but I don't understand what they're trying to say.
- bjackman 4y ago> Meta’s infrastructure plays an important role in supporting our products and services.
- boesboes 4y agolol, that's basically the definition of infrastructure
- jsty 4y agoThe section titled "An Overview of Velox" gets into the meat of it - you give it some data and an optimised plan of the operators you want applied (expression, evaluation, aggregation, etc.) - Velox then executes that plan as efficiently as possible given the available compute resources. That way multiple top-level systems like analytics databases, dataframe implementations, etc. can all share the same underlying execution engine.
- picardo 4y agoSo it's like an operating system for cloud workloads?
- aseipp 4y agoNo, it's close to the core internals of an OLAP database. The "operators" it can execute in question are things like filter, join, aggregate, group by, projection (select), things of that nature. It makes sure to use available resources like SIMD and multithreading to do that efficiently. If you built a SQL parser -- and also the glue to create query plans from that -- you could attach it to Velox to do all that on some data source, for example. But you'd still need a storage layer (disk, s3) and also some kind of higher layer if you wanted to use multiple computers for a complete database. The query execution engine is a critical component, however.
- a-dub 4y agospark and presto have big farms of workers. it's a replacement for the part that owns the memory where the computations happen on the workers. it has a handful of fast support libraries and i/o libraries but can also make use of spark and presto. it is probably faster to start up and has less fixed overhead (beyond the space used by the work area) than the usual runtimes because it is written in c++. it also is probably a better bridge for dl type workloads. you can think of it as similar to a "python kernel" but distributed and language agnostic for "big data" type jobs.