5 ms·
Whats wrong with this: 1) scan to find 300m line offset N 2) shift file from N to 0 using reads+writes 3) truncate last N bytes You can print out the current
by emerged 6y ago
Whats wrong with this:
1) scan to find 300m line offset N
2) shift file from N to 0 using reads+writes
3) truncate last N bytes
You can print out the current offsets in case of a crash you need to resume from. You read exactly the size of the file once and write it exactly once, and it's safe.
- kreetx 6y agoBeaten to the punch :)
- ComodoHacker 6y agoIt's #1 answer. Nothing wrong except it isn't safe. Depending on type of crash you might not have a chance to see your printed offset.
- emerged 6y agoYou can miss a printed offset and it still works. The file data exists in two places while writing the offset.
- tzs 6y agoThe old fashioned way to deal with making sure you see your printed offsets is to actually print your offsets. You'd go to the machine room and run the command on the system console, which would typically be a hardcopy terminal like a DECwriter. The trendy modern way, of course, would be to use the cloud. Upload the offsets to the cloud as they are produced, pausing until you get confirmation after each upload. A more clever and/or foolhardy way is to put the crash recovery markers in the file itself. Place evenly spaced markers throughout the file, saving the data that they overwrite in a separate file. Then do your in-place file shifting. Finally, replace the markers with the saved data.
- anon946 6y agoIf you print the offset after a write has safely completed, then it should still work. The internal copy is idempotent, up to the number of bytes in the first 300M lines. In other words, assuming reasonable line lengths, you can safely re-do a copy operation. So if you crash, you may not have printed out the last offset, but you have recorded a recent offset. You can then just restart from that recent offset.
- jrochkind1 6y agoThat is basically the answer in the OP, do you suspect there's something wrong with it?