2 ms·
Not all the same decisions - we didn't have forking model at the core except maybe some stuff from gunicorn, but RPC and DNS woes did happen in geometric fashio
by p_l 6y ago
Not all the same decisions - we didn't have forking model at the core except maybe some stuff from gunicorn, but RPC and DNS woes did happen in geometric fashion.
And that was with barely few services and maybe total less of 20 VMs. Some things were triggered by me putting a load-balancer in front of services (It's the first project where I used Thrift, and the project that left me with strong reasons to not use it ever again), so we actually ended up with several cases where developers went "it must be load balancer" then we would finally track it down to stupid django logging configured to use MySQL as log store, with MySQL conked out by colocated Redis server because the new service which asked for it didn't specify expected usage. That was fun. So was "we DoS'd our DNS system and thus everything died because kerberos couldn't resolve reverse queries.
- karmakaze 6y agoSimilarly, I've seen DNS problems at scale with coarse-grained (not micro) services that were in Java/Scala (also using Thrift) with threads. Gave up on Thrift due to the poor/inconsistent language support even between Java<->Scala.