4 ms·
At my company, we experimented with Hive and found it to be too slow. Once you decide that every row of data will be materialized into a java hash table instanc
by MichaelSalib 14y ago
At my company, we experimented with Hive and found it to be too slow. Once you decide that every row of data will be materialized into a java hash table instance, there are real performance limitations that follow from that representation. We decided to use HDFS and Hadoop but built our own query language called Trecul; it uses LLVM to jit compile dataflow code into native code. Picking off a single field to filter on is a single machine instruction for us.
The code is open source (albeit with rough edges) at https://github.com/akamai-tech/trecul/ https://github.com/akamai-tech/trecul/
- alexatkeplar 14y agoThanks for sharing Michael - Trecul looks to be the first OSS release from Akamai (at least on GitHub)? While we're on the subject of high-performance alternatives to Hive for event analysis, we've been watching the development of Ben Johnson's behavioural db, called Sky, with interest: https://github.com/skylandlabs/sky https://github.com/skylandlabs/sky
- benbjohnson 14y agoThanks for the shout out, Alex. Michael- Trecul looks cool. I'm doing something similar with Sky. I'm building an LLVM-based query language on it called EQL (Event Query Language) that's optimized for evented data. It does a lot of function rewriting to optimize loops and avoid heap allocations so it can crank through tens of millions of events per second. It's not finished yet but it should be done in the next couple weeks.