4 ms·
0:"path" /read lines "path"0:("line1";"line2") /write lines 1:"path" /read bytes "path"1:"content" /writ
by ngnk 7y ago
0:"path" /read lines
"path"0:("line1";"line2") /write lines
1:"path" /read bytes
"path"1:"content" /write bytes
- 7thaccount 7y agoThanks ngnk, but what if I can't read the entire file into memory (Ex: 20 GB file)? Are there any streaming words?
- ngnk 7y agoreading is done with mmap. it returns instantly and then loads memory pages from disk only when you use them.
- 7thaccount 7y agoSo it uses memory mapped files for all operations? So whether the file is 1 MB or 1 TB, it'll read what it can via streaming? Sorry, not knowledgeable here. Sounds neat though.
- ngnk 7y ago1:x uses mmap, so it would return instantly no matter how large a file you give it 0:x uses 1:x and then it splits the content into lines. unfortunately splitting requires copying, so you'd be limited by the amount of ram (let's ignore "swap"). i don't use mmap for writing/amendment yet. i'll be working on it. the way modern hardware works is like this: every process run by the os has its own view of the (typically 48-bit) address space. a process can request from the os that a part of that address space be "backed" by a certain file. this means that every time the process touches (i.e. tries to read or write) a virgin memory page there (usually page=4k, always aligned), the os will be automatically notified and will make sure to fill it in with actual content from the file, before the process even knows. from that point on, the page will occupy physical ram. if the os is low on memory later, it may decide to free up the page and return it to its previous state. in effect, data from disk (or any disk-like storage) can be "streamed" while your program uses ordinary array indexing. the word "streaming" though implies reading from start to end in order (which is additionally sped up by prefetch, but that's a different story..); memory mapping is more general - it allows random access.
- bakul 7y agoAssuming you are terminating strings with a NUL, for 0: it may be advantageous to R/W mmap with copy-on-write (i.e. MAP_PRIVATE). Then you can overwrite LF or CRLF with a NUL.
- 7thaccount 7y agoI'd love to use this on Windows at some point as your language matures. Is there any legal reason you'd be worried? It seems like ok and Kona are meant to exist strictly as toys.
- ngnk 7y agoianal. to the best of my knowledge the project is clean from a legal perspective. ofc, you don't have to be doing something wrong to become a target of copyright trolling.
- kick 7y agoYeah, the reason oK was a toy was because John Earnest works for a company that has their own fork of the K3 source, if I remember correctly, which puts him in an area that won't end with one of Kx's harsh lawsuits getting dismissed immediately. (Not that if he made a high-performance interpreter it'd be a bad thing legally: the reason GNU's programming guidelines are so archaic is partially because everyone implementing GNU early-on had seen the UNIX source code yet they could get around getting sued by writing esoterically.)
- ngnk 7y agoregarding windows - see https://news.ycombinator.com/item?id=22010059 https://news.ycombinator.com/item?id=22010059
- 7thaccount 7y agoSo just to make sure I understand: If I want to run on Windows, I just put those "c" code files in a directory and run the Microsoft C compiler against it?