4 ms·
That does seem simpler. I've run into variants of this problem where each thread/process can potentially produce GBs of data. For that use case performance wou
by davvid 10y ago
That does seem simpler. I've run into variants of this problem where each thread/process can potentially produce GBs of data. For that use case performance would suffer if combining were done as a separate step.
- kbenson 10y agoActually, since the data is sequential, I think you can write a very quick and optimized combining process. E.g. Open each file and buffer a small amount, such as 4k. 1) Read the first record of each buffer 2) Display the first record of the buffers 3) Read the next record from the buffer you just consumed from, and re-buffer another 4k from it if needed. 4) Go to 2. Go ahead and change how much you want to buffer from each file depending on what seems to work well and to take advantage of sequential read times if using media that uses that to it's advantage (spinning disks), but I don't see that being very slow. Worst case, you are naively sorting X records each time to find the first, but since you are always consuming them in order, you can easily keep track of the current order and do a dingle comparison per loop after the first.