23 ms·

Cool use of Numba. Does anyone know if there's more information available about their query language and what kinds of Python expressions they dynamically gener

Cool use of Numba. Does anyone know if there's more information available about their query language and what kinds of Python expressions they dynamically generate?

lcampbell 13y ago
From my understanding of the video, the queries are generated by the analysts via a frontend-generated or hand-written SQL query. The SQL query is parsed by PostgreSQL which forwards it via FDW[1] to Multicorn[2]. Their custom data storage and processing backend implements the API expected by Multicorn (e.g., you implement the multicorn.ForeignDataWrapper interface); this is where they transform from the parsed, serialized SQL into their custom DSL (the metaprogramming bit) which compiles to LLVM. -- [1] http://wiki.postgresql.org/wiki/Foreign_data_wrappers http://wiki.postgresql.org/wiki/Foreign_data_wrappers [2] http://multicorn.org/ http://multicorn.org/
- dialtone 13y ago
  Yeah, that's pretty much how it works. The frontend is anything that supports PostgreSQL as a database. Right now we use Tableau but we also used to have a custom WebUI on top of this service, it was very functional-inspired. Unfortunately Ville is on vacation right now, otherwise he'd be glad to dive more into the details of how that piece worked.
  - iskander 13y ago
    Why did you guys choose to compile through Numba rather than directly to LLVM or C?
    - vtuulos 13y ago
      Numba was just the fastest way to get it working. LLVM(py) is very low-level. C is still an option but Numba made interfacing with a no-brainer.
  - vtuulos 13y ago
    Author here: Fortunately HN works even in the Finnish countryside. I am happy to answer any questions.
    - dev360 13y ago
      Very inspiring talk! Is it possible to deal with continuous data in those matrices or is it more oriented around discrete values?
      - vtuulos 13y ago
        Thanks! Our approach supports both discrete and continuous values. It is mainly optimized for the use case where you want to aggregate continuous variables over discrete filters.
    - iskander 13y ago
      Any interest in also trying Parakeet (https://github.com/iskandr/parakeet https://github.com/iskandr/parakeet) for the backend? I'm curious to see how the performance would compare with Numba. I also have a semi-usable Builder API which constructs typed functions at a higher-level than llvmpy.
vtuulos 13y ago
Sorry, not yet. I hope to be able to make more information available soon. The query language is very straightforward. More interestingly, this approach makes it easy to implement various algorithms for machine learning / data mining, at least compared to MapReduce.

Cool use of Numba. Does anyone know if there's more information available about their query language and what kinds of Python expressions they dynamically generate?

lcampbell 13y ago

From my understanding of the video, the queries are generated by the analysts via a frontend-generated or hand-written SQL query. The SQL query is parsed by PostgreSQL which forwards it via FDW[1] to Multicorn[2]. Their custom data storage and processing backend implements the API expected by Multicorn (e.g., you implement the multicorn.ForeignDataWrapper interface); this is where they transform from the parsed, serialized SQL into their custom DSL (the metaprogramming bit) which compiles to LLVM. -- [1] http://wiki.postgresql.org/wiki/Foreign_data_wrappers http://wiki.postgresql.org/wiki/Foreign_data_wrappers [2] http://multicorn.org/ http://multicorn.org/

dialtone 13y ago
Yeah, that's pretty much how it works. The frontend is anything that supports PostgreSQL as a database. Right now we use Tableau but we also used to have a custom WebUI on top of this service, it was very functional-inspired. Unfortunately Ville is on vacation right now, otherwise he'd be glad to dive more into the details of how that piece worked.
- iskander 13y ago
  Why did you guys choose to compile through Numba rather than directly to LLVM or C?
  - vtuulos 13y ago
    Numba was just the fastest way to get it working. LLVM(py) is very low-level. C is still an option but Numba made interfacing with a no-brainer.
- vtuulos 13y ago
  Author here: Fortunately HN works even in the Finnish countryside. I am happy to answer any questions.
  - dev360 13y ago
    Very inspiring talk! Is it possible to deal with continuous data in those matrices or is it more oriented around discrete values?
    - vtuulos 13y ago
      Thanks! Our approach supports both discrete and continuous values. It is mainly optimized for the use case where you want to aggregate continuous variables over discrete filters.
  - iskander 13y ago
    Any interest in also trying Parakeet (https://github.com/iskandr/parakeet https://github.com/iskandr/parakeet) for the backend? I'm curious to see how the performance would compare with Numba. I also have a semi-usable Builder API which constructs typed functions at a higher-level than llvmpy.

vtuulos 13y ago

Sorry, not yet. I hope to be able to make more information available soon. The query language is very straightforward. More interestingly, this approach makes it easy to implement various algorithms for machine learning / data mining, at least compared to MapReduce.