4 ms·
I've been digging into Common Lisp again lately. I'm really enjoying some parts: CLIM is weird but interesting, and the fact that I'm basically developing insid
by floren 4y ago
I've been digging into Common Lisp again lately. I'm really enjoying some parts: CLIM is weird but interesting, and the fact that I'm basically developing inside a debugger makes testing and iterating on stuff pretty straightforward. On the other hand, library documentation frequently feels more like the programmer was making notes to himself rather than illustrating how the code might actually be used. I was also pretty shocked when, after using a quick shell script to generate a file containing a big list (~80MB on disk, 800k items) defined in s-expressions, reading it into SBCL exhausted the heap. After telling SBCL it could allocate two gigabytes I was able to read the file, but doing anything with it was a sure-fire way to run out of memory again. I would never think twice about reading an 80MB file into a Go program!
I sort of feel like I'm walking through an abandoned alien city and wondering what size they were, how many limbs they had, and if we share the same 5 senses or not.
- armitron 4y agoLisp won't save you from yourself. I bet a Go program using an equivalent representation (assumption being you naively used READ) would exhibit similar memory consumption. However, that doesn't mean that you don't have other options: - Use an array - Load the entire file in a memory buffer and skip the reader - Use mmap (for extra points, you can keep the entire file outside the Lisp heap but still manipulate it in Lisp)
- fiddlerwoaroof 4y agoYou can run into similar issues in the JVM if `-Xmx#` is too small. This has to do with the way sbcl’s decided to manage the heap and other implementations won’t have this issue. But, I typically start sbcl with an 8GB heap and basically never run into this when consuming large files. But, also, once you hit a couple megabytes or so, it’s almost always better IMO to consume a file incrementally rather than all at once.
- perihelions 4y agoSBCL strings store 4 bytes per character (the raw Unicode codepoints, like UTF-32). It's not ideal.
- winrid 4y agoReading an 80mb JSON file in Node wouldn't be an issue either, so that's surprising.
- aidenn0 4y agoAlso there's no problem reading an 80MB JSON file in CL. I suspect gp was using cl:load which is for loading source files. SBCL definitely struggles with very large source files; I never bothered debugging that since I don't need 80mb source files.
- floren 4y agoI was calling read. I suspect the real problem is explained by another commenter, who points out that 80 MB of ASCII text will become 240MB of UTF32 in the hands of SBCL, not to mention all the additional words of memory required for cons cells.