4 ms·
From my understanding of the video, the queries are generated by the analysts via a frontend-generated or hand-written SQL query. The SQL query is parsed by Pos
by lcampbell 13y ago
From my understanding of the video, the queries are generated by the analysts via a frontend-generated or hand-written SQL query. The SQL query is parsed by PostgreSQL which forwards it via FDW[1] to Multicorn[2]. Their custom data storage and processing backend implements the API expected by Multicorn (e.g., you implement the multicorn.ForeignDataWrapper interface); this is where they transform from the parsed, serialized SQL into their custom DSL (the metaprogramming bit) which compiles to LLVM.
--
[1] http://wiki.postgresql.org/wiki/Foreign_data_wrappers http://wiki.postgresql.org/wiki/Foreign_data_wrappers
[2] http://multicorn.org/ http://multicorn.org/
- dialtone 13y agoYeah, that's pretty much how it works. The frontend is anything that supports PostgreSQL as a database. Right now we use Tableau but we also used to have a custom WebUI on top of this service, it was very functional-inspired. Unfortunately Ville is on vacation right now, otherwise he'd be glad to dive more into the details of how that piece worked.
- iskander 13y agoWhy did you guys choose to compile through Numba rather than directly to LLVM or C?
- vtuulos 13y agoNumba was just the fastest way to get it working. LLVM(py) is very low-level. C is still an option but Numba made interfacing with a no-brainer.
- vtuulos 13y agoAuthor here: Fortunately HN works even in the Finnish countryside. I am happy to answer any questions.
- dev360 13y agoVery inspiring talk! Is it possible to deal with continuous data in those matrices or is it more oriented around discrete values?
- vtuulos 13y agoThanks! Our approach supports both discrete and continuous values. It is mainly optimized for the use case where you want to aggregate continuous variables over discrete filters.
- iskander 13y agoAny interest in also trying Parakeet (https://github.com/iskandr/parakeet https://github.com/iskandr/parakeet) for the backend? I'm curious to see how the performance would compare with Numba. I also have a semi-usable Builder API which constructs typed functions at a higher-level than llvmpy.