3 ms·

With multi-thread and some tricks, csv parsing should be able to reach SSD speed (~2GiB/s read): https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-of-t

by liuliu 3y ago

With multi-thread and some tricks, csv parsing should be able to reach SSD speed (~2GiB/s read): https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-of-the-nvme-storage/ https://liuliu.me/eyes/loading-csv-file-at-the-speed-limit-o... At that point, I think the speed is limited by boxing JS objects.

leeoniya 3y ago
nice, is this something you've tried in WASM or WebGL/WebGPU? at some point it makes sense to simply recommend using WASM SQLite (or DuckDB) to load the data for OLAP stuff. im not gonna encourage anyone to use JS on multi-GB files. even a few hundred MB becomes questionable ;)
- texodus 3y ago
  The CSV parser for [Perspective](https://github.com/finos/perspective https://github.com/finos/perspective) uses the [Apache Arrow](https://github.com/apache/arrow https://github.com/apache/arrow) C++ CSV parser compiled to WASM. It's not currently multi-threaded but this is possible as well to my understanding.
  - leeoniya 3y ago
    what i learned from benchmarking Dekkai, is that simply "being in WASM" is a poor indicator of actual performance if you ever need to pull the values across that boundary into JS...especially strings. i'll take a look and see if i can incorporate Perspective's parser into the bench, thanks!
tomcam 3y ago
What does it mean to box an object?
- deleted 3y ago
  [deleted]
- davefol 3y ago
  I believe allocating memory for objects on the heap. https://stackoverflow.com/a/80113/10654749 https://stackoverflow.com/a/80113/10654749
  - tomcam 3y ago
    I think it’s more specific, something along the lines of turning a reference (pointer-allocated) object to a value-based object? But I’m woefully ignorant on this.
    - liuliu 3y ago
      Oh, by "boxing object", I meant to create the in-memory object that can be used by JavaScript runtime, preferably native objects. For example, in JSC case, something like https://developer.apple.com/documentation/javascriptcore/jsvalue?language=objc https://developer.apple.com/documentation/javascriptcore/jsv....
      - tomcam 3y ago
        Ah, thanks

by liuliu 3y ago

leeoniya 3y ago

nice, is this something you've tried in WASM or WebGL/WebGPU? at some point it makes sense to simply recommend using WASM SQLite (or DuckDB) to load the data for OLAP stuff. im not gonna encourage anyone to use JS on multi-GB files. even a few hundred MB becomes questionable ;)

texodus 3y ago
The CSV parser for [Perspective](https://github.com/finos/perspective https://github.com/finos/perspective) uses the [Apache Arrow](https://github.com/apache/arrow https://github.com/apache/arrow) C++ CSV parser compiled to WASM. It's not currently multi-threaded but this is possible as well to my understanding.
- leeoniya 3y ago
  what i learned from benchmarking Dekkai, is that simply "being in WASM" is a poor indicator of actual performance if you ever need to pull the values across that boundary into JS...especially strings. i'll take a look and see if i can incorporate Perspective's parser into the bench, thanks!

tomcam 3y ago

What does it mean to box an object?

deleted 3y ago
[deleted]
davefol 3y ago
I believe allocating memory for objects on the heap. https://stackoverflow.com/a/80113/10654749 https://stackoverflow.com/a/80113/10654749
- tomcam 3y ago
  I think it’s more specific, something along the lines of turning a reference (pointer-allocated) object to a value-based object? But I’m woefully ignorant on this.
  - liuliu 3y ago
    Oh, by "boxing object", I meant to create the in-memory object that can be used by JavaScript runtime, preferably native objects. For example, in JSC case, something like https://developer.apple.com/documentation/javascriptcore/jsvalue?language=objc https://developer.apple.com/documentation/javascriptcore/jsv....
    - tomcam 3y ago
      Ah, thanks