4 ms·
Not parent, but I do data analysis where more memory is always appreciated. Essentially all tooling now works with bigger than memory datasets, but you eat a pe
by fbdab103 3y ago
Not parent, but I do data analysis where more memory is always appreciated. Essentially all tooling now works with bigger than memory datasets, but you eat a performance cost as data gets paged in/out to disk. When you can load a 50GB dataframe directly into pandas, it lets you do things the dumb way rather than having to spend extra brain cycles figuring out how to compartmentalize the problem to stay within your RAM budget.
- saltcured 3y agoSame for me. Back around 2015, I had 64 GB in a workstation because I had to develop and test image analysis tools. These would be working with 3D and 4D dense arrays and rather than paging in/out from disk it was a matter of loading chunks in/out from RAM to the GPU's 12 GB of VRAM and assembling larger results to eventually serialize back to disk. When not working on these tools, it was all just buffer cache for Linux while the actual processes were easily living in 8 GB or so. Even today, I'd comfortably live in 16 GB on a laptop, except once in a while miss the option to just do sloppy large allocations for a single task instead of worrying about chunking, IO, and careful access order.