mattewong

17 ms·

1. 121.▲
   Show HN: SQL on CSV/TSV/etc + native/WASM + worlds fastest CSV parser(github.com)

   10 points by mattewong 5y ago|2 comments
2. 122.▲
   by mattewong 5y ago

   There is a lot of talk these days about "big" data, but we see a lot of "small data" use cases-- say, 1,000 to 1mm rows-- where repetitive but non-automated tasks hamper everyday work for armies of analysts. Many pieces
3. 123.▲
   Show HN:in-browser CSV/XLSX stratification & validation w covid & loans examples(zsvhub.com)

   1 points by mattewong 5y ago|1 comments
4. 124.▲
   by mattewong 5y ago

   Thank you for the ideas / links / mentions! I had not seen them and will check them out. Always looking for improvements!
5. 125.▲
   by mattewong 5y ago

   Correction! https://zsvhub.com/playground/ (needs the ending slash) which is powered by a web version of the SIMD parser ( https://github.com/liquidaty/zsv ). BTW some quick performance notes on the
6. 126.▲
   by mattewong 5y ago

   Saw it :) > done efficiently with very few branches, though, using mostly SIMD and some scalar operations on masks [...] first phase basically looking for a comma or newline followed by a quote, then looking for the next unescaped quote
7. 127.▲
   by mattewong 5y ago

   Huh? your comment proves exactly what my prior post said, which is that you end up with 3 equal values that were each represented, in the input, in different ways. Looks like csv.reader + dialect=excel is doing exactly that.
8. 128.▲
   by mattewong 5y ago

   As promised: https://github.com/liquidaty/zsv
9. 129.▲
   by mattewong 5y ago

   If their data comes from a controlled bubble and they need not assume "real-world" data, then CLMUL might do nicely but best case it would likely only be a marginal improvement (and even then I would be willing to bet, not at an
10. 130.▲
    by mattewong 5y ago

    What I can tell you is that if you save a file with 'AA"A,"AA""A","A"A"A' (excluding the surrounding single-quotes) and then double-click to open in Excel, you get 3 cells with the exact sam
11. 131.▲
    Show HN: Full-featured+fast SIMD CSV lib, extensible utility & web playground(github.com)

    5 points by mattewong 5y ago|2 comments
12. 132.▲
    by mattewong 5y ago

    In the spirit of giving, we give you ZSV/lib, a fully-baked SIMD-powered CSV parser and extensible standalone utility. This is the initial alpha release, and includes select, sql, count, desc(rice), pretty, flatten, 2json, 2tsv, serial
13. 133.▲
    by mattewong 5y ago

    Even with the CLMUL trick, CSV parsing does not play nice. It can be made to work for JSON parsing because you can make more assumptions. With CSV, it only works smoothly if you are willing to accept a subset of what most spreadsheet progra
14. 134.▲
    by mattewong 5y ago

    Stay tuned for a SIMD powered CSV parser library and standalone utility about to drop this weekend. Alpha, but test showing it to be faster than anything else we could get our hands on