4 ms·
It's likely this is related to DNS lookups. Doing a lookup on a modern operating system requires calling in to some C library. The call in to the C code means t
by jesstaa 14y ago
It's likely this is related to DNS lookups.
Doing a lookup on a modern operating system requires calling in to some C library. The call in to the C code means that the thread can no longer be used to schedule goroutines, the runtime doesn't know how long the C call will take so it creates a new thread to keep running goroutines.
For calls in to C that are very quick this can lead to a lot of thread creation.
You can do one of:
1. limit thread creation by using your operating systems limiting features. eg ulimit
2. rebuild the standard lib with DISABLE_CGO=1 to avoid the calls in to C if that's possible in your environment.
3. batch all your dns lookups in to a single goroutine.
4. make changes to the operating system so provide async dns lookups.
- noselasd 14y agoThanks, I'll do some testing with these points later on. Though it does sound like go should batch up gethostbyname/getaddrinfo C calls on one or a few threads. Having DNS lookups be the cause of thousands of threads to be created, up till the point where go routines fails is rather ridiculous for a language that's supposed to be good at concurrency imho.
- jesstaa 14y agoBatching the C calls means that you all lookups are delayed by other lookups. This isn't very good for concurrency especially since it's reliant on network traffic. If you had a few lookups in the queue, you could end up having goroutines waiting a few seconds or for lots of lookups this could balloon out to minutes for a lookup. This is the trade off. In general most programs aren't making thousands of concurrent DNS lookups.
- dillonmacc 14y agoi just use http://www.unlocktheinbox.com/dnslookup/mx/ http://www.unlocktheinbox.com/dnslookup/mx/ or some other online dns lookup tools.