6 ms·
I did an assignment on the Blue Brain Project once. It's an incredibly complex project with lofty ambitions. One thing that blew my mind was how limited they w
by daoist_shaman 4y ago
I did an assignment on the Blue Brain Project once. It's an incredibly complex project with lofty ambitions.
One thing that blew my mind was how limited they were by modern storage technology. Storing high-fidelity neuronal cross-section images consumed more space than what was available at the time. Each scan was crazy massive, something like a petabyte or an exabyte.
To store the entire human brain in this fashion would consume more storage space than that which exists on Earth.
- robertk 4y agoCan they trade off storage for compute by applying ML based compression JIT?
- bsedlm 4y agoprobably not cuz I'd imagine they want lossless compression (given as they don't understand the data well enough to do meaningfull (lossless) compression)
- bsedlm 4y agoI meant: meaningfull (lossy) compression
- whimsicalism 4y agoYou can get very good lossless compression with ML models, it is just not very fast to decompress.
- davidatbu 4y agoAre you referring to the work of Fabrice Bellard? I thought he only showed the proof of concept for text data, using language models, only?
- whimsicalism 4y agoI am referring to the work by Fabian Mentzer and related approaches, which covers both lossless and lossy compression.
- davidatbu 4y agoJust saw this. Thanks!
- bsedlm 4y agohttps://bjlkeng.github.io/posts/lossless-compression-with-latent-variable-models-using-bits-back-coding/ https://bjlkeng.github.io/posts/lossless-compression-with-la... ?
- whimsicalism 4y agoWas thinking of work like https://arxiv.org/pdf/1811.12817.pdf https://arxiv.org/pdf/1811.12817.pdf although the compression improvements vs classical methods are much more impressive for lossy compression
- slavik81 4y agoI've been curious if video codecs could be repurposed to store voxel data. Most approaches seem to use image stacks, but there's inherently going to be a lot of redundancy from slice to slice. I'm sure a purpose-built format could be better with enough effort, but there's a lot of work put into video compression, so it seems like there might be a way to leverage that for some cheap improvements to 3D data storage. The storage solution that I've seen in practice has just been a folder of TIFs, so it seems like there's lots of room for improvement.