4 ms·

For file sizes where parsing speed really makes a difference, loading the entire file into memory doesn't seem feasible. Maybe some sort of a hybrid approach (

• atwebb 11y ago
  I think there's a utility that does something similar (windows based, assuming it's available elsewhere), I remember it having a "crass" name, you'd think it would make it easier to remember, maybe F.U.C.K.?
• victorNicollet 11y ago
  In a REPL environment, for a 1GB file, this can be the difference between "wait three seconds" and "let me get my coffee".
• bane 11y ago
  And if you're going to load it, you may as well parse it on the way in.
  • victorNicollet 11y ago
    A streaming CSV parser is very difficult to get right, once you get past a certain level of complexity. Sometimes, you are not lucky enough to have perfect control over the encoding, number format and date format of the input, so you need to look ahead at a value sample to try and find out what those are. Sometimes, you cannot even assume that the software that produced the file didn't mangle the quotes around fields, and you have to detect that you are 40960 bytes into a quoted field, decide that it's probably an error, and backtrack. If you have enough memory to load the entire file, you will save a lot of time by giving up on streaming processing.
    • maxhou 11y ago
      if you use mmap(), there is virtualy no difference between the streaming/non streaming parsing code.