3 ms·
Did the authors benchmark against kdb? e.g., run a single k interpeter on each CPU, then divide and conquer Isn't it faster to do parsing in memory and avoid
by uudecode 10y ago
Did the authors benchmark against kdb?
e.g., run a single k interpeter on each CPU, then divide and conquer
Isn't it faster to do parsing in memory and avoid I/O wherever possible?
- geocar 10y agoBest of three, on an i7 8GB ram: Loading a 156mb csv file in kdb 32-bit free version, single thread: \t trade:`sym`time`ex`cond`size`price!("STCCXH";",")0:`t.csv 1850 in paratext, 64-bit: >>> timeit.timeit('paratext.load_csv_to_dict("t.csv",num_threads=4)', setup="import paratext", number=1) 3.1176819801330566 No improvement. Loading a 1.5GB csv file in kdb: \t quote:`sym`time`ex`bid`bsize`ask`asize`mode!("STCHXHXC";",")0:`q.csv 14135 in paratext: >>> timeit.timeit('paratext.load_csv_to_dict("q.csv",num_threads=4)', setup="import paratext", number=1) 12.962939977645874 Not too shabby! An almost 10% improvement over KDB by turning my fans on and burning my lap! However, I think they should probably make their parser faster before they waste heat trying to make slow code finish sooner.