3 ms·

Sorry for not looking into it myself, but what is the format of your index file (I'm assuming the index is stored as a file somewhere). Could other tools read f

Sorry for not looking into it myself, but what is the format of your index file (I'm assuming the index is stored as a file somewhere). Could other tools read from that too?

burntsushi 12y ago
It's extremely simple. Even if you don't know Rust, this code should be readable: https://github.com/BurntSushi/rust-csv/blob/master/src/index/mod.rs#L52-L69 https://github.com/BurntSushi/rust-csv/blob/master/src/index... Basically, it writes a 64 bit byte offset (big endian) for the start of each record. The very last byte is then the total number of records. (I forget why I'm doing this, since the count should be computable from the size of the index file.) The "start of each record" means "if you point a CSV parser at this byte offset, then it will successfully read a record." So when you want to read record `i`, seek the index to byte offset `i * 8`, read 8 bytes (big endian) into a 64 bit integer, then seek to that byte offset in your CSV data. If the CSV data changes, then you're in trouble. One easy way to handle it (which isn't bulletproof) is to check the last modified times of the index and the CSV file and regenerate the index (or refuse to use it with an error).
- breck 12y ago
  For an append to a CSV file it shouldn't be too hard to update the index. Remove the last byte in your current format and just parse the additional lines and append to your index. And I'm sure there's a technique out there to handle the update case. I'm not too familiar with hard disk operations so I wouldn't know it. If you insert a byte at position 2 in a 1GB file will the whole file get rewritten? Or are there techniques out there to store updates to a char array without having to rewrite the whole thing?
  - burntsushi 12y ago
    I'm pretty sure you're barking up a long and complex tree. The first problem (for xsv) is that it doesn't control the csv data. So it doesn't know what changes were made. It could have even become an invalid csv file. The second problem is that insertions in sorted sequences are hard and are one of the fundamental problems in a database implementation: http://en.wikipedia.org/wiki/B-tree#The_database_problem http://en.wikipedia.org/wiki/B-tree#The_database_problem
    - breck 12y ago
      Agreed. It's complex but a common situation so I knew there'd be a lot of well tested approaches.

Sorry for not looking into it myself, but what is the format of your index file (I'm assuming the index is stored as a file somewhere). Could other tools read from that too?

burntsushi 12y ago

It's extremely simple. Even if you don't know Rust, this code should be readable: https://github.com/BurntSushi/rust-csv/blob/master/src/index/mod.rs#L52-L69 https://github.com/BurntSushi/rust-csv/blob/master/src/index... Basically, it writes a 64 bit byte offset (big endian) for the start of each record. The very last byte is then the total number of records. (I forget why I'm doing this, since the count should be computable from the size of the index file.) The "start of each record" means "if you point a CSV parser at this byte offset, then it will successfully read a record." So when you want to read record `i`, seek the index to byte offset `i * 8`, read 8 bytes (big endian) into a 64 bit integer, then seek to that byte offset in your CSV data. If the CSV data changes, then you're in trouble. One easy way to handle it (which isn't bulletproof) is to check the last modified times of the index and the CSV file and regenerate the index (or refuse to use it with an error).

breck 12y ago
For an append to a CSV file it shouldn't be too hard to update the index. Remove the last byte in your current format and just parse the additional lines and append to your index. And I'm sure there's a technique out there to handle the update case. I'm not too familiar with hard disk operations so I wouldn't know it. If you insert a byte at position 2 in a 1GB file will the whole file get rewritten? Or are there techniques out there to store updates to a char array without having to rewrite the whole thing?
- burntsushi 12y ago
  I'm pretty sure you're barking up a long and complex tree. The first problem (for xsv) is that it doesn't control the csv data. So it doesn't know what changes were made. It could have even become an invalid csv file. The second problem is that insertions in sorted sequences are hard and are one of the fundamental problems in a database implementation: http://en.wikipedia.org/wiki/B-tree#The_database_problem http://en.wikipedia.org/wiki/B-tree#The_database_problem
  - breck 12y ago
    Agreed. It's complex but a common situation so I knew there'd be a lot of well tested approaches.