3 ms·
I remember reading around 2002-2004 about a sysadmin managing a very large "supernode" p2p server who was able to fine-tune its Linux kernel, and to recompile a
by mrb 15y ago
I remember reading around 2002-2004 about a sysadmin managing a very large "supernode" p2p server who was able to fine-tune its Linux kernel, and to recompile an optimized version of the p2p app (to allocate data structures as small as possible for each client) to support up to one million concurrent TCP connections. It wasn't a test system, it was a production server routinely reaching this many connection at its daily peak.
If it was possible in 2002-2004, I am not impressed that it is still possible in 2011.
One of the optimizations was to reduce the per-connection TCP buffers (net.ipv4.tcp_{mem,rmem,wmem}) to only allocate one physical memory page (4kB) per client, so that one million concurrent TCP connections would only need 4GB RAM. His machine had barely more than 4GB RAM (6 or 8? can't remember), which was a lot of RAM at the time.
I cannot find a link to my story though...
- gtani 15y ago(Not your link but) some pointers on tuning erlang/OTP servers: http://www.erlang-consulting.com/thesis/tcp_optimisation/tcp_optimisation.html http://www.erlang-consulting.com/thesis/tcp_optimisation/tcp... http://www.trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles http://www.trapexit.org/Building_a_Non-blocking_TCP_server_u... http://groups.google.com/group/erlang-programming/browse_thread/thread/f47daa85ca45ed71/ http://groups.google.com/group/erlang-programming/browse_thr... (this erl mailing list thread is pretty typical, if you put up code, describe your app, hardware, network, database/external dependencies, etc, you'll get a ton of good advice about killing off bottlenecks. Another example http://groups.google.com/group/erlang-programming/browse_frm/thread/1931368998000836/ http://groups.google.com/group/erlang-programming/browse_frm...
- mmaunder 15y agoAgree with this being underwhelming. Maybe they'll hire someone who understands that Erlang is using kqueue to make this possible. Running netstat|grep like this on a high concurrency server takes a long time to run. I've never found a faster way to get real-time stats on our busy servers and would be interested if anyone else has.
- derobert 15y agoYou might be able to parse /proc/net/tcp yourself faster, especially for just counting by state.
- X-Istence 15y ago/proc/net/tcp doesn't exist on FreeBSD...
- thomasknowles 15y agoActually I would run ss over the netstat for such high concurrency connection as netstat hits /proc/net/tcp and it's quite slow. However, you need to have the tcp_diag module loaded otherwise it falls back to /proc/net/tcp.
- X-Istence 15y agoYou may not have read the entire article, but he clearly explains that their tech stack is FreeBSD + Erlang not Linux + Erlang. There is no /proc/net/tcp on FreeBSD. Hell, there is no /proc unless it is specifically mounted by the administrator, but tools certainly don't use it to get data.
- thomasknowles 15y agoI read it, I just presume people most users are linux based in here.
- adient 15y agoYou may have tried but generally add -n to netstat to speed it up by just displaying IPs instead of trying to resolve them. I've never seen a system with so many concurrent connections hands on so there could be some other bottleneck.
- jen_h 15y agonetstat -s |grep "connections established" is pretty fast and does the job if you're not filtering for certain ports or IPs. But running plain old netstat on a system with so many connections is a really bad idea. I almost fired myself once over a crufty cron job I left running...