4 ms·
>how inefficient the boundary crossing is For 99.99% of the programs that people write, the modern M.2 NVME hard drives are plenty fast, and thats the laziest
by ActorNightly 1y ago
>how inefficient the boundary crossing is
For 99.99% of the programs that people write, the modern M.2 NVME hard drives are plenty fast, and thats the laziest way to load data into a C extension or process.
Then there is unix pipes which are sufficiently fast.
Then there is shared memory, which basically involves no loading.
As with Python, all depends on the setup.
- zahlman 1y agoThe problem isn't loading the data, but marshalling it (i.e, transforming it into a data structure that makes sense for the faster language to operate on, and back again). Or if you don't transform (or the data is special-cased enough that no transformation makes sense) then the available optimizations become much more limited.
- ActorNightly 1y agoThats all just design. Nothing having to do with particular language.
- jononor 1y agoThere are several datastructures for numeric data that do not need marshalling, and are suitable for very efficient interoperetion between Python and C/C++/Rust etc. Examples include array.array (in standard library), numpy.array, and PyArrow.