27 ms·
Running Awk in parallel to process 256M records
- tarun_anand 6y agoAmazing work. Keep it up.
- gautamcgoel 6y agoYour system had 512-core Xeons? Did you mean that you had 5 12-core xeons? Or 512 cores total?
- ketanmaheshwari 6y ago512 Intel Xeon CPUs configured over 16 nodes. This one: http://www.comnetco.com/sgi-uv300-the-most-powerful-in-memory-supercomputer http://www.comnetco.com/sgi-uv300-the-most-powerful-in-memor...
- MR4D 6y ago#ComputeEnvy
- gautamcgoel 6y agoWow. I didn't realize you could run so many CPUs in one address space. This thing is basically one huge computer. Would love to open up Gnome System Monitor and see 512 cores!
- cesaref 6y agoCore counts has gone up quite considerably in the last few years, certainly in a compute farm context. 10 years ago, you'd be looking at 2U servers having 8 cores (dual quad core) as high density. These days, 2U servers can pack 4 sockets, and processors are in the high 20 something cores, so if you've got deep pockets you can get over 100 cores in 2U.
- jerven 6y agoAMD Epyc can give you 128/256 cores in a 2 socket system. Not that expensive.
- tobias2014 6y agoI'm sure that with tools like MPI-Bash [1] and more generally libcircle [2] many embarrassingly parallelizable problems can easily be tackled with standard *nix tools. [1] https://github.com/lanl/MPI-Bash https://github.com/lanl/MPI-Bash [2] http://hpc.github.io/libcircle/ http://hpc.github.io/libcircle/
- pdimitar 6y agoI have lately used a shell `for` loop that only emits part of the iterated values (not all are eligible for further processing) and then fed the loop directly to the `parallel` tool via the pipe operator. The results were indeed embarrassingly parallel. I am a fan of some languages that seem better equipped to utilise our modern many-core machines and I'd still write a longer-living system with better guarantees in those languages -- but many people ignore shell goodness at their own peril.
- mturmon 6y agoBig agree. Gnu parallel, xargs, and make -j are all very useful basic tools for embarrassingly parallel workloads. I've been developing a simulation software that does Monte Carlo over realizations of a simulated universe, and xargs and (later) parallel were super useful for parts of the workload. All the parallel job instances run the same simulation code, but for a different simulated universe, all controlled by a random number seed, so you can generate an ensemble of simulations with basically: head -NumberOfSimsWanted seeds.txt | xargs simulation.py
- pdimitar 6y agoYep! Even as a fan of more parallel-friendly programming languages I still think having single-core algorithms applied to different input parameters in parallel is a very viable alternative approach if your problem space allows for it. I quite disliked the journey. The UNIX shell legacy shenanigans are real and can be a pain. But you can still bear a lot of fruit without going to the more arcane corners of it.
- flyinghamster 6y ago
- hidiegomariani 6y agothis somewhat reminds me of taco bell programming http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
- ketanmaheshwari 6y agoExcellent! Didn't know there is a name for this. Agreed the work is a case of Taco bell programming!
- Scarbutt 6y agoThere is also https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- sydney6 6y agoIs this Hunter S. Thompson in the picture?
- gen220 6y agoThank you! I was hunting down this link for a coworker a few months ago, but I forgot the keyword “taco bell”, and Google fu was failed me... I love these kind of “counterculture” uses of UNIX tools to solve hard problems. The boundary of “where to stop using xargs, awk, and grep and start using Python” is pretty blurry for a surprising number of tasks, especially if you’re willing to invest in mastering those tools!
- kyuudou 6y agoOh my word thank you so much for linking this.
- deleted 6y ago[deleted]
- nojito 6y agoWhy not just use data.table? The solution would be much less error prone and most likely much quicker as well.
- ketanmaheshwari 6y agoThank you for the comment. One reason to use awk was that I wanted to see how far I could go with Awk. I will checkout data.table. Does it offer any kind of parallelism?
- nojito 6y agoMany common functions in data.table are parallelized under the hood by OpenMP.
- pwdisswordfish2 6y agoWhy not just use kdb+?
- nojito 6y agodata.table is faster and open source.
- pwdisswordfish2 6y ago"data.table is faster" How do you know this, i.e., can you share a benchmark -- the test data, not the results -- that you ran?
- nojito 6y agoBecause I process ~1.5TB a day and I was exploring multiple options. Data.table was the clear winner for my use case.
- pwdisswordfish2 6y ago
- svnpenn 6y ago!($1 in a) && FILENAME ~ /aminer/ { print } This uses a regular expression. As regex is not actually needed in this case, you might be able to get better performance with something like this: !($1 in a) && index(FILENAME, "aminer") != 0 { print }
- ineedasername 6y agoDefinitely an under-appreciated tool. Very useful for one-off tasks that would take a fair bit longer to code in something like python.
- co_dh 6y agoI like the idea of using AWK for this. But you can give kdb/q a try. 250M rows is nothing for kdb, and it seemed that you can afford the license.
- lenkite 6y agokdb/q is amazing. I have seen some experienced folks make jaw dropping data computations at the drop of a hat on gigabyte sized data. Makes everything else look laughably puny. I suspect if it was open source, it would probably be the most popular big-data storage and computing platform.
- labelbias 6y agoHmm... 256M records is still not that much for postgres or mysql so it shouldn't be too much for kdb/q. Several gb of data also shouldn't be too much.
- greggyb 6y agokdb/q, or many other mature data platforms. 250M records isn't big for a lot of data platforms. I've seen plenty of solutions in traditional RDBMSs that would easily scale an order of magnitude larger than this, if not several. Moving into anything analytically focused, like an OLAP engine, or just a columnstore in an RDBMS easily gives you another order of magnitude or few.
- skanga 6y agoTry mawk if you can. I find that it does even faster.
- ketanmaheshwari 6y agoIndeed I did try mawk and found it to be faster. However, when I try setting locale "LC_ALL=C"; the performance of both awk and mawk were almost the same. I also let it at awk in favor of portability -- mawk is not available on most systems and needs to be installed.
- tannhaeuser 6y agoIt's odd that TFA has this focus on performance but doesn't mention which awk implementation has been used; at least I haven't found any mention of it. There are 3-4 implementations in mainstream use: nawk (the one true awk, an ancient version of which is installed on Mac OS by default), mawk (installed on eg. Ubuntu by default), gawk (on RH by default last I checked), or busybox-awk. Tip: mawk is much faster than the others and to get performance out of gawk you should use LANG=C (and also because of crashes with complex regexpes in Unicode locales in some versions of gawk 3 and 4).
- ketanmaheshwari 6y agoThank you for your comment. I should have clarified that I use gawk version 4.0.2. I mean to update the post but somehow am unable to push to repo or login to my github. Edit: Done.
- tannhaeuser 6y agoYeah just read in another answer of yours that for your scripts probably regexp performance dominates the results as there's not much procedural code to make a difference.
- mfontani 6y agoThey seem to be currently having a few problems with log-in: https://www.githubstatus.com/incidents/q3cfsrp1qb6l https://www.githubstatus.com/incidents/q3cfsrp1qb6l
- mjcohen 6y agogawk has been my go-to text processing program for many years. I have written a number of multi-thousand line programs in it. I always use the lint option. Catches many of my errors. One of these had to read a 300,000,000 byte file into a single string so it could be searched. The file was in 32-byte lines. At first, I read each line in and appended it to the result string, but that took way to long since the result string was reallocated each time it was appended to. So I read in about 1000 of the 32-byte lines, appending them to a local string. This 32000-byte string was then appended to the result, so this only was done 10000 times. Worked fine.
- gh123man 6y agoSlightly off topic, but as a swift developer (https://swift.org/ https://swift.org/) the usage of swift/T in this project really confused me. Is swift/T in any way related to Apple's swift language? The naming conflict makes googling the differences fairly challenging.
- ketanmaheshwari 6y agoBoth Swift's are different. I was aware of potential confusion which is why I did not use Swift in the title and mention explicitly in the blog that this is not Apple Swift. Fun story: Apple contacted Swift/T team before launching Apple Swift and made a mention of it in their page.
- gh123man 6y agoCool! Thanks for the explanation. That answers my biggest question of who came first.
- tetha 6y agoHm. I'm fully aware that I'm currently turning into a bearded DBA. And I may be just misreading the article and I probably don't understand the article. But, I started being somewhat confused by something: > Fortunately, I had access to a large-memory (24 T) SGI system with 512-core Intel Xeon (2.5GHz) CPUs. All the IO is memory (/dev/shm) bound ie. the data is read from and written to /dev/shm. > The total data size is 329GB. At first glance, that's an awful lot of hardware for a ... decently sized but not awfully large dataset. We're dealing with datasets that size at 32G or 64G of RAM, just a wee bit less. The article presents a lot more AWK knowledge than I have. I'm impressed by that. I acknowledge that. But I'd probably put all of that into a postgres instance, compute indexes and rely on automated query optimization and parallelization from there. Maybe tinker with pgstorm to offload huge index operations to a GPU. A lot of the shown scripting would be done by postgres, the parallelization is done automatically based on indexes, while eliminating the string serializations. I do agree with the underlying sentiment of "We don't need hadoop". I'm impressed that AWK goes so far. I'd still recommend postgres in this case as a first solution. Maybe I just work with too many silly people at the moment.
- VHRanger 6y agoIf you're dealing with such a static single-machine dataset, why not go for SQLite instead of PostGres? There'd be much less setup overhead
- FDSGSG 6y agoSpending minutes on these tasks on hardware like this is pretty silly. awk is fine if these are just one-off scripts where development time is the priority, otherwise you're wasting tons of compute time. Querying things like these on such a small dataset should take seconds, not minutes.
- ketanmaheshwari 6y agoThank you for your comment. Most of the solutions indeed take less than a minute. Solution to problem 1 & 3 took 25 sec and that to 5 took 26. The solution that took 9 minutes involved processing the abstract in each record. The abstracts are quite sizeable on some of these publications. Processing millions of them took time. The solution that took 48 minutes involved a nested loop effectively reaching the iteration count for 216 years times 256M records which comes to about 55B. Hope this clarifies things a bit but I am not claiming this to be the most optimized solution. I am sure there is scope for refinements -- this was my take on it.
- Upvoter33 6y agoI've always wanted to build a parallel awk. And call it pawk. And have an O'Reilly book about it. With a chicken on the cover. pawk, pawk, pawk! This is a true story, sadly.
- snorrah 6y agoQ. How much rep / user score do I need before I can downvote? I ask because I can only upvote and this comment should be downvoted.
- ketanmaheshwari 6y agoI too wanted to do almost exactly the same. I want to port awk to GPUs by hacking its code and adding OpenCL tags at right places. Someday!
- elmolino89 6y agoName is already taken by python awk substitute: https://github.com/alecthomas/pawk https://github.com/alecthomas/pawk
- winrid 6y agoand here I am working on a big distributed system that has to handle 200k records a day (and hardly does successfully). sigh.
- nmz 6y agoYou couldn't have used FS="\036" or "\r"?
- _wldu 6y agoTurning json data into tabular data using jq was pretty neat. So many json apis in use today yet still a need for csv and excel docs.
- nojito 6y agoExcel can natively connect to and parse json nowadays.
- schmichael 6y agohttps://mobile.twitter.com/awkdb https://mobile.twitter.com/awkdb was a joke account made in frustration by a coworker trying to operate a Hadoop cluster almost a decade ago. Maybe it's time to hand over the account...
- Zeebrommer 6y agoI am often impressed by the things that can be done with these old-school UNIX tools. I'm trying to learn a few of them, and the most difficult part are these very implicit syntax constructions. How is the naive observer to know that in bash `$(something)` is command substitution, but in a Makefile `$(something)` is just a normal variable? With `awk`, `sed` and friends it gets even worse of course. Is the proper answer 'just learn it'? Are these tools one of these things (like musical instruments or painting) where the initial learning phase is tedious and frustrating, but the potential is basically limitless?
- avar 6y agoSome of it you pickup or remember from the context of the file you're looking at, but you really should take the time to read the manuals for these tools from cover to cover at some point if you're making extensive use of them. In the case of GNU bash & make: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.... & https://www.gnu.org/software/make/manual/make.html https://www.gnu.org/software/make/manual/make.html
- MaxBarraclough 6y agoShoutout to the lesser-known GNU Recutils. Not sure how well it would fare in this particular instance, but it seems relevant. https://www.gnu.org/software/recutils/ https://www.gnu.org/software/recutils/ https://en.wikipedia.org/wiki/Recfiles https://en.wikipedia.org/wiki/Recfiles
- flatfilefan 6y agoGNU Parallel + AWK = even less code to write.
- eatYourFood 6y agoWow - great insight......
- arendtio 6y agoFirst, I think it is great that you found a tool that suits your needs. A few weeks ago I was mangling some data too (just about 17 million records) and would like to contribute my experience. My tools of choice were awk, R, and Go (in that order). Sometimes I could calculate something within a few seconds with awk. But for various calculations, R proved to be a lot faster. At some point, I reached a problem where the simple R implementation I borrowed from Stack Overflow (which was supposed to be much faster than the other posted solutions) did not satisfy my expectations and I spend 4 hours writing an implementation in Go which was a magnitude faster (I think it was about 20 minutes vs. 20 seconds). So my advice is to broaden your toolset. When you reach the point where a single execution of your awk program takes 48 minutes, it might be worth considering using another tool. However, that doesn't mean awk isn't a good tool, I still use it for simple things, as writing 2 lines in awk is much faster than writing 30 in Go for the same task.