3 ms·
it is theoretically possible that a library like numpy or pandas could do this directly. That is, if you take each of these transformation operations and stor
by zzzeek 7y ago
it is theoretically possible that a library like numpy or pandas could do this directly. That is, if you take each of these transformation operations and store them, but not actually run them until needed, that is, when someone accesses the individual array values, you only need to make one copy and there is no impact on the programmer to have to worry about this.
I was doing something similar for SQLAlchemy recently. SQLAlchemy is all about capturing Python operators and operands into data structures that represent intent which can be invoked later.
edit: here is a demonstration: https://gist.github.com/zzzeek/caa4a7ed94f326fbbc031acecb9d7a44 https://gist.github.com/zzzeek/caa4a7ed94f326fbbc031acecb9d7...
edit edit: it is actually possible with numpy by using the Dask library: https://dask.org/ https://dask.org/
- masklinn 7y agoThe problem with the library doing this is you end up with the same issue as in Haskell: you accumulate thunks and both memory use and location of CPU hotspots becomes much harder to predict and troubleshoot.
- zzzeek 7y agoso...turn off the "thunking" feature while you are debugging.
- JeremyBanks 7y agoThat won't help when you're debugging the performance implications of thunking.
- zzzeek 7y agocompare memory / callcounts / time spent with thunking turned on vs. off would give a good idea of it. i can say for my side of things thunking is a super huge performance gain as it allows for quick gathering of expression intent and then the computation side on the other side of a cache.
- vmchale 7y agoYou can use Futhark; it compiles to PyOpenCL and then can be called by python fairly fluently. Accelerate is a nice example of an array library with a JIT backend.