3 ms·
This starts from my frustration when opening large CSV files, but later evolved to a log/data analyzer that loads arbitrary format in constant time (O(1)). htt
by SdtEE 5mo ago
This starts from my frustration when opening large CSV files, but later evolved to a log/data analyzer that loads arbitrary format in constant time (O(1)).
https://github.com/Verticalysis/Hitomi https://github.com/Verticalysis/Hitomi
The secret: I engineered an incremental combinatorial parser capable of processing customized format from a steam. Any inputs, including file or the stdout from a command, are first chunked and then fed to the pipeline. The UI is ready when the first small chunk is processed.
Other highlights:
2-mode filter, one with a convenient UI and the other is based on an extensible DSL for complex cases;
Timeline mode scrollbar, a secret weapon for log or time series analysis;
Column widths fit to content automatically;
Native code, no web bloat;
Cross-platform (currently Windows and Linux, MacOS WIP).
If you are tired of all the quirks Excel have when working with CSV files, you'll gonna love it!
- klntsky 5mo agoDoes it update the parser incrementally?
- SdtEE 5mo agoI'm not sure what do you mean by "update". If you mean the state of the parser - of course, and I did some trick to reduce the supposedly infinite states for arbitrary format to a fixed set of 4. If you mean by the parser program it self -No. The parser was developed with Dart, a statically compiled language, with no ability to modify the program itself.