3 ms·
Is there any way you can control the number of native threads in go ? I wrote a small go example that used net/smtp to bombard myself with emails. Just a loop t
by noselasd 14y ago
Is there any way you can control the number of native threads in go ? I wrote a small go example that used net/smtp to bombard myself with emails. Just a loop that fired off a goroutine, which each sent an email.
It was quite disturbing that firing off several thousand goroutines resulted in several thousand native threads - resulting the system trashing. If I inserted a very small pause between creating the go routines, this did not occur and just 1 or 2 native threads were created. I tried the runtime.GOMAXPROCS, to no awail. Is there any way of controlling this kind of behavior ?
- jesstaa 14y agoIt's likely this is related to DNS lookups. Doing a lookup on a modern operating system requires calling in to some C library. The call in to the C code means that the thread can no longer be used to schedule goroutines, the runtime doesn't know how long the C call will take so it creates a new thread to keep running goroutines. For calls in to C that are very quick this can lead to a lot of thread creation. You can do one of: 1. limit thread creation by using your operating systems limiting features. eg ulimit 2. rebuild the standard lib with DISABLE_CGO=1 to avoid the calls in to C if that's possible in your environment. 3. batch all your dns lookups in to a single goroutine. 4. make changes to the operating system so provide async dns lookups.
- noselasd 14y agoThanks, I'll do some testing with these points later on. Though it does sound like go should batch up gethostbyname/getaddrinfo C calls on one or a few threads. Having DNS lookups be the cause of thousands of threads to be created, up till the point where go routines fails is rather ridiculous for a language that's supposed to be good at concurrency imho.
- jesstaa 14y agoBatching the C calls means that you all lookups are delayed by other lookups. This isn't very good for concurrency especially since it's reliant on network traffic. If you had a few lookups in the queue, you could end up having goroutines waiting a few seconds or for lots of lookups this could balloon out to minutes for a lookup. This is the trade off. In general most programs aren't making thousands of concurrent DNS lookups.
- dillonmacc 14y agoi just use http://www.unlocktheinbox.com/dnslookup/mx/ http://www.unlocktheinbox.com/dnslookup/mx/ or some other online dns lookup tools.