2 ms·
Best of three, on an i7 8GB ram: Loading a 156mb csv file in kdb 32-bit free version, single thread: \t trade:`sym`time`ex`cond`size`price!("STCCXH";",")0
by geocar 10y ago
Best of three, on an i7 8GB ram:
Loading a 156mb csv file in kdb 32-bit free version, single thread:
\t trade:`sym`time`ex`cond`size`price!("STCCXH";",")0:`t.csv
1850
in paratext, 64-bit:
>>> timeit.timeit('paratext.load_csv_to_dict("t.csv",num_threads=4)', setup="import paratext", number=1)
3.1176819801330566
No improvement.
Loading a 1.5GB csv file in kdb:
\t quote:`sym`time`ex`bid`bsize`ask`asize`mode!("STCHXHXC";",")0:`q.csv
14135
in paratext:
>>> timeit.timeit('paratext.load_csv_to_dict("q.csv",num_threads=4)', setup="import paratext", number=1)
12.962939977645874
Not too shabby! An almost 10% improvement over KDB by turning my fans on and burning my lap!
However, I think they should probably make their parser faster before they waste heat trying to make slow code finish sooner.