5 ms·
The numpy style interface might be interesting. The problem is that it would probably kill performance to marshal into a Python process. Unless there's some way
by pauldix 13y ago
The numpy style interface might be interesting. The problem is that it would probably kill performance to marshal into a Python process. Unless there's some way to embed numpy?
Supporting custom functions is definitely something we want to do.
- yummyfajitas 13y agoWhy would it kill performance? All you need to do is this: double* values = (double*)malloc(sizeof(double)*num_of_values); (If you are unfamiliar with numpy, it's just a python wrapper around raw blocks of memory.)
- waterlion 13y agoYou still have to get the bytes out of the server and into the Python process.
- yummyfajitas 13y agoAssuming Python is not running in the same process as InfluxDB, amend my code snippet: double* data = shmget(key, sizeof(double)*num_data_pts, whatever_flag); But directly embedding a Python interpreter is pretty easy in C, so I imagine it should also be fairly easy in Go as well. (Of course, given that my code snippets are C, you can probably deduce I've never written any Go, so take my comments on how easy it is with a grain of salt.)
- waterlion 13y agoGo interop with C is very doable (although I've never tried it). Not sure about shared memory WRT garbage collection though.
- lotyrin 13y agoA route to this would be providing an FFI for loadable modules that provide custom procedures or functions. Folks that wanted to could then write Python/Numpy code for these and use Numba+LLVM to compile them. It could be quite performant and avoid having to marshal data or do IPC, could possibly avoid even copying data in some cases (Numpy/Numba have pretty robust support for structures coming over an FFI, not sure about FFI in Go.)