6 ms·
Playing with Go: Embarrassingly Parallel Scripts
- jgrahamc 14y agoYes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddresses, _ := net.LookupIP(d) ip := "" if len(ipAddresses) > 0 { ip = ipAddresses[0].String() } fmt.Println("Mapping: ", d, "->", ip) done <- true }(domain) count++ } for i := 0; i < count; i++ { <- done } }
- dadkins 14y agoThere's a library for that pattern: http://golang.org/pkg/sync/#WaitGroup http://golang.org/pkg/sync/#WaitGroup Instead of counting and using a done channel, import "sync" ... var wg sync.WaitGroup for ... { wg.Add(1) go dowork() } wg.Wait()
- jameskilton 14y agoNice. Yeah I'm very new to Go so I know almost nothing about the stdlib libraries. Thanks for the pointer.
- darkhelmetlive 14y agoTooting my own horn here, but I'm working on a book covering the Go Standard Library. Still in progress, and not at the sync package yet, but it's coming along. Check it out if you feel inclined. http://thestandardlibrary.com/go.html http://thestandardlibrary.com/go.html
- lsb 14y agoYou know what'd be cool? To take arbitrary code in a language, pattern match on the implementation in that language of each std lib function, and actively recommend substitutes for duplicated code.
- jlgreco 14y agoThat would be very cool. I wonder how hard it would be though. At least in Go I know that the standard lib contains lots of duplicate code (primarily so that things that should be small don't require larger things as dependencies. I think the time package's String() functions use reimplemented fmt package functionality for example, since fmt is a much larger dependency than time should have.)
- skybrian 14y agoYes, but outside the Go standard libraries, adding a dependency on a standard library isn't a big deal and won't add a cycle.
- jlgreco 14y agoYes. What I mean though is that if you are reimplementing, say, fmt.Printf, such a suggestion system might correctly suggest you use fmt.Printf instead, but also suggest you can use func (m Month) String() string from time, or something equally silly. Since the standard libs in Go duplicate code, you would have to be careful that your suggestion system isn't picking up false positives. I think the idea has a lot of promise though.
- stock_toaster 14y agoLooks interesting (thanks). Will certainly check it out.
- zmj 14y agoI'm embarrassed how many times I've implemented the above without thinking to look for a standard lib solution. Thanks!
- 0xABADC0DA 14y agoEven so why would you write that boilerplate code out each time? Something like this would work even better: result = src.asyncMap { |e| dowork(e) }; Except Google Go returns several values instead of tuples, so you can't just collect all the results as-is. And with no generics it would be annoying to actually use e and the result, since they would need casts. Too bad.
- burntsushi 14y agoWhy am I not surprised to see you trolling HN too? What's all too sweet is that you've brought your anti-Go zealotry here too! Joy. > Except Google Go returns several values instead of tuples, so you can't just collect all the results as-is. This is false. Functions in Go do not have to return multiple values. Therefore, you can "collect all the results as-is". > And with no generics it would be annoying to actually use e and the result, since they would need casts. Too bad. Actually, it wouldn't be annoying, because you wouldn't use a general purpose map like you've shown. You'd use code shown in the parent.
- 0xABADC0DA 14y agoIf you think you can do asyncMap in Google Go, without casts and without manually collecting multiple return values, by all means show us the code. I would find that really interesting.
- burntsushi 14y agoI didn't claim I could. Take your trolling elsewhere.
- y0ghur7_xxx 14y agoI know it's about Go, but bash is awesome for stuff like this: while read line; do host $line | head -n1 | awk '{print $1 " -> " $4}' & done < domains.txt
- etrain 14y agoI realize it's a one-off, but you could potentially mix lines of output with a script like that. If you gnu parallel (xargs++) installed - turn that second line into a script (and replace $line with $1). cat domains.txt | parallel -n 1 -P 50 script.sh
- icebraining 14y agoNotice that with parallel, you don't need the -n 1, since that's the default.
- etrain 14y agoAh, didn't realize that - throwback to my xargs days! Thanks for the tip.
- crusso 14y agoThanks for the note on parallel, it looks to be an excellently useful tool. I can find a lot of ways to save time with it. FYI: On OSX, there is a Homebrew formula (brew install parallel).
- ole-tange 14y agoIf you are looking for which domain point to 1.2.3.4: cat domains.txt | parallel -P 100 --tag host | grep 1.2.3.4 Somewhat slower runtime than the go-solution, but may be faster to write.
- philsnow 14y agoYou don't need to spawn N awks, <domains.txt while read line; do host $line | head -n1 & done | awk '{print $1 " -> " $4}'
- drivebyacct2 14y agoSo many ignored errors. :(
- jameskilton 14y agoIt's a one-off script, I didn't really care about errors. If this was something run regularly inside of a bigger application yes I'd have full error handling.
- ImJasonH 14y agoBut tiny throwaway scripts get built up into huge applications all the time, sometimes by other people who don't have the mental TODO to go back and handle errors.
- stusmall 14y agoThis is one of the things I love about go. If he wants the return value but doesn't want to address the errors, he has to actively do it. It makes you think twice when you are putting a _ for an error return. And now to another programmer coming in to maintain it, they stick out like a sore thumb.
- saraid216 14y agoI know zero Go (it's a little far down on the to-learn list), and maybe he edited it, but I went back and read the GP's code and didn't see anything that looked like ignoring errors. Is it the two ", _" things?
- stusmall 14y agoSo for this line of code: file_in, _ := ioutil.ReadFile("domains.txt") It says call ReadFile. In go a function can return multiple values. ReadFile returns an byte array and an error value. Normally you'd check to see if err is nil, if it is then no error happened. If it isn't you can check it out for information about the error and handle it. A unique feature about go is that declaring a variable and never using it is an error and not a warning. Actually there aren't even compiler warnings. Either it is right or wrong. This means if he had called: file_in, err := ioutil.ReadFile("domains.txt") but never checked err it would not build. So to get the byte array but not the error you use the _ symbol to tell it to throw away that return value. This is what I was on about in that you have to actively ignore error handling if you want a return value.
- hannibalhorn 14y agoGo is great for stuff like this, especially when part of an actual system. That said, if this was just an adhoc job (to figure out which domains point to a specific IP address) you can just use "xargs -P" or GNU parallel and it becomes a pretty basic shell script, along the lines of: cat domains.txt | xargs -P 1000 -n 1 host
- mistermcgruff 14y agoxargs -P is a godsend for one-off jobs needing parallelism
- skeltoac 14y agoUntil last week I didn't know that xargs could invoke commands in parallel. xargs -n1 -P8 dig A <hosts.txt | grep -v ';' | grep $TARGET_IP | sed 's/\.\s.*//' >hosts-matched.txt
- scurry 14y agoSo what's the difference between xargs and parallel? I thought the point of parallel was that it was xargs with the addition of running things in parallel. But if xargs can do that already, is there any reason to use one over the other?
- prodigal_erik 14y agoFor what it's worth, GNU Parallel seems to be fairly new; at work I have an ubuntu distro from this year and it's not in the package repo yet. xargs is POSIX so you can expect it everywhere, though no parallel option is specified (merely encouraged).
- eggoa 14y agoIt's mostly about how they handle special characters. http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel http://www.gnu.org/software/parallel/man.html#differences_be...
- hannibalhorn 14y ago
- lazyjones 14y agoTo be fair to languages without such great parallelism support: you can do this using asynchronous/event-loop-based code because the parallelism will be limited by the nameserver anyway (the calling code does almost nothing, it mostly waits for the net / the nameserver).
- chimeracoder 14y ago> To be fair to languages without such great parallelism support: you can do this using asynchronous/event-loop-based code Well you can do with callbacks anything that you can do with channels and goroutines. Go's primary appeal is that it makes concurrent[1] code easy to reason about, not that it enables you to do anything that you "couldn't do" otherwise. Continuations are just GOTOs, and just like GOTOs, some people love them and some people hate them, but even people who like them can find them difficult in large doses. Goroutines and channels are nice, because they fit the structure of imperative code, whereas callbacks sort of resemble imperative code but "inside out". [1] Note that I didn't say parallel!
- j2labs 14y agoThis kind of a thing is also very easy in Python if you use Gevent. I have used Gevent a lot over the last couple years and see a lot of similarities in Go's concurrency, which I think is great. Concurrency doesn't have to be about insane looking code!
- d0mine 14y agoWith concurrent.futures http://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor http://docs.python.org/3/library/concurrent.futures.html#con... it could look like: from concurrent.futures import ThreadPoolExecutor as Pool from socket import getaddrinfo def lookup(domain): try: result = getaddrinfo(domain, 80) exception Exception as e: print("error %s -> %s" % (domain, e)) else: print("done %s -> %s" % (domain, result)) nconcurrent = 20 with open('domains.txt') as file, Pool(nconcurrent) as pool: for domain in (line.strip() for line in file): pool.submit(lookup, domain) To run multiple processes instead of threads, change the import to ProcessPoolExecutor. To support multiprocessing.Pool (for Python 2 where concurrent.futures is not in stdlib), replace pool.submit() with pool.apply_async() and use contextlib.closing() around the Pool().
- jnazario 14y agoin the DNS case asynchronous event handling would be super easy to do. in python asyncore with something like dpkt to construct and read DNS lookups works like a champ, as does twisted. i did a simple async DNS resolver in pure python (asyncore, dpkt) and can sustain thousands of lookups a second. GNU adns also has bindings in various languages. you can get Go's parallelisms via CSP (e.g. python-csp, ruby-csp) and replace a lot of fragile threading/parallel code with it. i've been doing that in lieu of learning Go (i know i know .. i'm lazy) and been very pleased. anyhow, many ways to skin cats. those are just two or three.
- tptacek 14y agoYou wouldn't do DNS lookups asynchronously in Go to begin with. Modeling concurrency of any sort in Go the way you would with an event loop is usually a code smell.
- jnazario 14y agoagreed, and i should have been more clear. the author's blog post states that one of the reasons he explored Go was that his initial sketch of a solution in his language of choice (ruby) was sequential. my point was that you can get performance in ruby with asynchronous operations, and that you don't need to go parallel for something like this. then again if it was a matter of "well, i had a problem to solve and i had a desire to explore another language, so solving it in that new language was a way to explore" then the point is moot. however agreed 100% or more on the "code smell" of doing an event loop in Go.
- blablabla123 14y agoNot sure if I understand your point, can you explain a bit more please? Are you saying event loops with select {} are considered code smell?
- tptacek 14y agoIf I had to simultaneously generalize the idea and make it specific enough to explain it further, I'd say fiddly callback state machines are a code smell in Go.
- dkhenry 14y agoScala is really great for this too. The downside is the spin up time for the JVM, but the upside is that if you use SBT's script launcher it will compile and cache the script transparently for you , you can still pull in _any_ JVM dependency and you can run it just like a shell script. I needed to test for a port being open in parallel and it was a cake walk to use NIO's SocketSelector to do it reactivly.
- danneu 14y agoMy attempt with Ruby and Celluloid. https://gist.github.com/a803d86234e8d1fc5496 https://gist.github.com/a803d86234e8d1fc5496 I also include a list of 100 domains in a domains.txt if anyone wants to try for themselves. require "socket" require "celluloid" class IPGetter include Celluloid def get(url) Socket.getaddrinfo(url, "http")[0][2] end end pool = IPGetter.pool(size: 100) ips = {} File.open("domains.txt").each_line do |line| line.chomp! ips[line] = pool.future.get(line) end ips.each do |url, ip_future| puts "#{url} => #{ip_future.value}" end
- ericmoritz 14y agoOr Erlang https://gist.github.com/bb01a85404e6b445dcb3#file_resolve_domains_plist.erl https://gist.github.com/bb01a85404e6b445dcb3#file_resolve_do... % -*- erlang -*- %%! -smp enable worker(Hostname) -> {ok, IP} = inet:getaddr(Hostname, inet), io:format( "~s => ~s~n", [ip_to_string(IP)] ). ip_to_string({N1,N2,N3,N4}) -> io_lib:format( "~w.~w.~w.~w", [N1,N2,N3,N4] ). main([DomainFile]) -> {ok, Bin} = file:read_file(DomainFile), String = binary_to_list(Bin), Domains = string:tokens(String, "\n"), plists:foreach( fun(Domain) -> worker(Domain) end, Domains ). This uses https://github.com/eveel/plists/ https://github.com/eveel/plists/
- geoka9 14y agoTo be fair to Ruby, it releases the GVL when a thread is blocked on IO; so for this kind of task it wouldn't be an issue, probably.