4 ms·
This is impressive, but... "A fast reader exploits the capabilities of the storage system"... the graphs show that their storage system is doing 4.00 GB/sec
by justinsaccount 10y ago
This is impressive, but...
"A fast reader exploits the capabilities of the storage system"...
the graphs show that their storage system is doing 4.00 GB/sec
I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD?
I tried running a quick test but only succeeded in OOMing my 8G laptop.
Even just doing
import paratext
it = paratext.load_csv_as_iterator("/dev/shm/tmp/c.log", expand=True, forget=True)
x = it.next()
Starts eating up all my ram after about a minute of spinning the cpu... so I think they have a slightly different definition of an iterator as everyone else.
Compared to
cut -d , -f 5 < c.log > /dev/null
which runs in a few seconds, or a slightly more domain specific and optimized version of 'cut'[1] that runs even faster (300-500MB/s on a single core depending on which fields you want)
$ du -hs c.log ;wc -l c.log
2.1G c.log
16197412 c.log
I also wonder if that is 2.5 GB/s per core.
https://github.com/BurntSushi/rust-csv https://github.com/BurntSushi/rust-csv does 241 MB/s in raw mode, so I find it a little hard to believe that this is 10x faster... unless that is while maxing out multiple cores.
[1] https://github.com/bro/bro-aux/blob/master/bro-cut/bro-cut.c https://github.com/bro/bro-aux/blob/master/bro-cut/bro-cut.c
- JoshTriplett 10y ago> I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD? One high-end PCIe SSD can manage that; two easily could. A high-end NAS might, too.
- ygra 10y agoDoes cut work with quoted fields correctly? My understanding was that it was just a dumb line tokenizer and CSV is a little more complex than that, e.g., CSV rows can span more than one line when fields contain line breaks.
- rspeer 10y agoWhich means an effective thing to do when you control the data pipeline is to ban tabs and line-breaks from your values, and then use `cut` on tab-separated files.
- Twirrim 10y agoOr just not use CSV as a data transfer format :D
- rspeer 10y agoI think you're missing the point. Unix tools like `cut` are really effective ways to deal with plain text data, who cares if it's CSV.
- ygra 10y agoSure, if you can redefine your data format, you can take shortcuts. But otherwise comparing cut to a CSV parser is just a different way of saying »if it doesn't have to be correct, I can make it as fast as I want«.
- gkst 10y agoNo, it doesn't and therefore is no viable option for many real-world datasets.
- omgtehlion 10y ago> 4.00 GB/sec > I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD? Consumer grade NVMe disks currently achiveve 2GB/s. It is easy to use a couple of those, or just one professional SSD.
- IanCal 10y ago> Starts eating up all my ram after about a minute of spinning the cpu... so I think they have a slightly different definition of an iterator as everyone else. I don't think so, the difference is you're expecting it to read one row at a time and it's actually reading one column at a time. load_csv_as_iterator is iterating over the columns.
- burntsushi 10y agoFrom looking at their source briefly, I can make two observations: Firstly, their core parsing routine is a state machine not unlike the one found in rust-csv: https://github.com/wiseio/paratext/blob/master/src/csv/rowbased_worker.hpp#L48 https://github.com/wiseio/paratext/blob/master/src/csv/rowba... Secondly, they do indeed appear to be achieving high parsing speed using coarsely grained parallelism. You can see the code that chunks up the CSV data here: https://github.com/wiseio/paratext/blob/master/src/generic/chunker.hpp https://github.com/wiseio/paratext/blob/master/src/generic/c... It's pretty clever!
- Ygg2 10y agoCan this be used to speed up rust-csv?
- burntsushi 10y agoIt feels like something that should be built on top of rust-csv.
- Ygg2 10y agoI thought this was base ability of the parser to parse CSV in chunks.