5 ms·
Why not just use data.table? The solution would be much less error prone and most likely much quicker as well.
by nojito 6y ago
Why not just use data.table?
The solution would be much less error prone and most likely much quicker as well.
- ketanmaheshwari 6y agoThank you for the comment. One reason to use awk was that I wanted to see how far I could go with Awk. I will checkout data.table. Does it offer any kind of parallelism?
- nojito 6y agoMany common functions in data.table are parallelized under the hood by OpenMP.
- pwdisswordfish2 6y agoWhy not just use kdb+?
- nojito 6y agodata.table is faster and open source.
- pwdisswordfish2 6y ago"data.table is faster" How do you know this, i.e., can you share a benchmark -- the test data, not the results -- that you ran?
- nojito 6y agoBecause I process ~1.5TB a day and I was exploring multiple options. Data.table was the clear winner for my use case.
- pwdisswordfish2 6y agoBut you cannot share the details of the pericular use case so someone else can test your claims? "After "exploring" many options, kdb+ was the clear winner for my use case. I process an average of 1.6TB per day." See how silly that sounds without any details or credibility? Does "clear winner" mean it was faster? How does a person know if the recommender even tried both solutions? What if someone else's use case is dfferent from the recommender's? How could someone verify that one solution was faster than the other? The easiest way is to provide sample data and a processing task and let people try the two solutions for themselves.