4 ms·
Awesome work. I tried something similar a while back, but I gave up once I realized the memory requirements for Ruby numeric primitives make it near impossible
by fny 5y ago
Awesome work. I tried something similar a while back, but I gave up once I realized the memory requirements for Ruby numeric primitives make it near impossible to massage large data sets in Ruby.
Pandas and numpy skirt this with custom numeric types. I've seen some efforts to build pandas clones in Ruby, but none have come close to the performance needed to handle a few gigs of data.
- rgrieselhuber 5y agoThere is also Dask in the Python world for when Pandas can’t handle the data size.
- richard_todd 5y agoI think they were adding something like python’s buffer protocol to ruby3, which should pave the way for something like numpy if there is enough demand. I’m referencing https://bugs.ruby-lang.org/issues/14722 https://bugs.ruby-lang.org/issues/14722
- nitrogen 5y agoNumo::NArray has proven useful to me for working with audio in Ruby, though I haven't tested it with GBs of data in a single array.