4 ms·
I don't really have any explanatory sources at hand, just the source code here: https://github.com/potree/PotreeServer/blob/redo/src/RegionsFilter.js https://gi
by mschuetz 8y ago
I don't really have any explanatory sources at hand, just the source code here: https://github.com/potree/PotreeServer/blob/redo/src/RegionsFilter.js https://github.com/potree/PotreeServer/blob/redo/src/Regions...
Points of interest:
Schedule files to be loaded. This will usually load around 1000 files:
https://github.com/potree/PotreeServer/blob/redo/src/RegionsFilter.js#L283 https://github.com/potree/PotreeServer/blob/redo/src/Regions...
Whenever a file is loaded, iterate through the points in it, do the filtering and write the results to the output file:
https://github.com/potree/PotreeServer/blob/redo/src/RegionsFilter.js#L286 https://github.com/potree/PotreeServer/blob/redo/src/Regions...
Wait until all files were loaded and processed:
https://github.com/potree/PotreeServer/blob/redo/src/RegionsFilter.js#L399 https://github.com/potree/PotreeServer/blob/redo/src/Regions...
Wait until the output was written to disk:
https://github.com/potree/PotreeServer/blob/redo/src/RegionsFilter.js#L401 https://github.com/potree/PotreeServer/blob/redo/src/Regions...
What happens is that ~1000 files will be loaded in the background, points are filtered in the main thread and even while some files are still being loaded, we already start writing the results to the output file.
> I tried to process simple CSV files in a very straight forward (but async) way and got reading 200mb CSV and just splitting it to columns (with simple split by comma) takes ~10 seconds.
CSV may be much bigger challenge since it's ASCII data. That always tends to take multiple times longer.
- orf 8y agoThe project is private.
- mschuetz 8y agoWhoops, my bad. ^^ Guess I wanted to keep it private till it's actually finished.
- flukus 8y agoIf file buffering is the bottleneck have you tried using tools like cat to buffer the input? Buffering input and output is the sort of thing you get for free in unix.