7 ms·
Ask HN: What is an acceptable latency for a web load balancer?
We use an external company for network management. They've put a Citrix ADC load balancer in front of our web servers, which is also terminating TLS. I'm not sure if it was always like this (before I joined the company) but downloading a 1000 bytes file via HTTP takes ~15ms while the same file via HTTPS takes at least 110ms. The external company ignores this and claims that SSL termination is expensive. I can understand a few ms, but clearly 100ms is rubbish.
What do you think folks?
- danpalmer 6y ago100ms for setting up an initial TLS connection is about what I'd expect. It depends on the key size, ciphers, etc, but up to 300ms is typical. However, this should only be done at the beginning of the connection. After this the client will have a symmetric encryption key that is much faster to use. Their load balancer should be caching these sessions so that subsequent connections don't need to re-negotiate a session key. If this 110ms is only on the first request, and a cache miss on the sessions, then I'd say that's probably something you should be expecting. If it's after the TLS session has been set up, or on a cache hit, that sounds bad. It also could be that their session cache isn't large enough and is forgetting sessions too soon, causing more TLS negotiation than may be necessary.
- axismundi 6y agoThis is happening for every single request.
- jeromegn 6y agoTheir server and/or your client might not be setup for session reuse. Definitely a thing to check.
- axismundi 6y agoThank you. I will make sure this is enabled.
- dharmab 6y agoMake sure you’re reusing connections. A lot of languages and libraries create a new connection for every request unless you explicitly manage a session or connection pool.
- axaxs 6y agoAlso, make sure the adc is reusing connections. It's a setting on the vserver.
- Matthias247 6y agoMost of the latency of a TLS handshake (which equals the minimum latency for the first response on a HTTPS connection) comes from round trips. You need 1 for TCP connection establishment. And 1 to 2 (depending on the TLS version) for the TLS handshake. 300ms would be rather on the high side. RTTs with todays infrastructure are often 30-50ms if you hit an Edge server (opposed to a server around the world). So you should end up up at around 100ms for the complete TLS connection establishment.
- gravypod 6y agoMost people don't care about latency but no, 110ms is not what I'd expect in this case. Take a look at these files hosted by google: https://developers.google.com/speed/libraries https://developers.google.com/speed/libraries d3.js on my network is ~200kb and downloads in 20ms with ssl.
- tleb_ 6y ago95-130ms on mine, it just depends.
- corobo 6y agoI think if you're quibbling over 100ms you need to be paying more so that people listen to the issue you're having and work with you. I'm not saying this to minimise like "oh it's only 100ms" I mean there are providers out there that will work with you on this. They're not cheap. If you pay enough you can reshape the internet - I once had an issue with a backup process and ended up having two ISPs start peering with each other to fix it.
- iso1210 6y agoSo many people in the industry don't understand when I say that a 100ms outage is a problem, it's depressing.
- danpalmer 6y agoThis depends what "industry" you're talking about. If you're talking about "tech", I'd say 100ms doesn't matter for the vast majority of use-cases. Bear in mind that Australia/New Zealand have a ~250ms round trip latency to most web services, 4G can easily have a 300ms latency to Google, and developing countries are often in a much worse position. Depending on <100ms for anything is a non-starter for most of the web. If you're a service provider several layers removed from a human, then there may be enough between you and the user that it's critical that you respond in <100ms, but this is not most of the industry. More generally, I find this sort of comment to be similar to the sort of comment that asks why anyone would write a web service in a language as slow as Ruby or Python when we have languages like Go and Rust. They ignore the fact that there are huge, successful companies running on Ruby, Python, and other "slow" languages, all for very good reasons.
- smolder 6y ago100ms becomes significant if you're doing teleconferencing or other real time interaction. If we somehow improved latency and jitter on the internet generally, I think that'd do more to improve its usefulness than expanding bandwidth. And don't forget about the studies out there that show a relationship between load time and revenue in e-commerce: "A 100ms decrease in checkout page load speed amounted to a 1.55% increase in session-based conversion."
- simplyinfinity 6y agoDoes your use case need low latency? No? Don't worry about it then. yes? Does it have clear measurable impact on the biz? No? probably wont' be able to convince anyone to do anything about it. yes? Show them the difference and clear benefits of how this improves customer/business value.
- axismundi 6y agoIt's a real time trading website, so every millisecond counts.
- WJW 6y agoIf it needs to be real time enough for trading then surely even the 15 ms for a non-TLS connection is too much? HFT system latency is measured in nanoseconds these days and I'm pretty sure the HTTP stack has too much overhead to be used in that domain.
- spdionis 6y agoDon't think a real time trading website has anything to do with HFT. Still, some of these trading websites that target day traders for customers need to be pretty fast.
- PaywallBuster 6y agoAs mentioned, the TLS session is probably not cached yet, following requests should be faster. However it's possible Citrix is just not as optimized as other providers/services or not support newer protocols like TLS 1.3 However, 100ms doesn't seem critical
- axismundi 6y agoThat's a valuable tip, thanks. I will look into the LB config if there is anything cache-related.
- jlgaddis 6y agoTBQH, the "acceptable latency" is whatever is defined in your contract with them. If the response to that is "there isn't one", well, there's your real problem.
- jamesmishra 6y agoI don't think every performance metric has to be tied to an explicit contract. There are thousands of such metrics in a large organization's tech stack. What OP is asking is whether the performance they are seeing is comparable to the industry-leading load balancers that HN readers are familiar with. The answer to that question is independent of any contract.
- cflewis 6y agoI once asked an engineer I respect deeply “what’s an acceptable timeout for this integration test?” His answer has stuck with me for a long time: “whatever your users will tolerate when they attempt that workflow.” Software can always be made better and can always be made faster. But you do have to know when to stop, and for me it’s always at “what my users will tolerate.”
- sgc 6y agoDon't push them to the edge every time. Try to have a conformable margin so they can use your services without it grating on them.
- contravariant 6y agoThe reason they're asking about acceptable latency is that unless the contract specifies differently both parties are bound by what the counterparty can reasonably expect. Each country has contract law like this, with lots of legal precedents on what can and cannot reasonably be expected in certain situations. Part of the reason there are all these terms and conditions documents that nobody reads is to prevent this ambiguity (to a point). Although legally it's a bit weak because you cannot really expect people to have agreed to all terms (but it does reverse the burden of proof somewhat, so it's still useful).
- erichurkman 6y agoCheck the endpoint with one of the SSL tools out there, like SSL Labs: https://www.ssllabs.com/ssltest/analyze.html?d=news.ycombinator.com https://www.ssllabs.com/ssltest/analyze.html?d=news.ycombina... Some of the settings there link to tips for more investigation. Look at the session resumption settings in particular.
- axismundi 6y agoThanks, I did. Here are relevant results: https://ybin.me/p/c73726c16d669a5d#WGH7arFJpP8aitaNcBCGqRP3t4e/JSSvQfm2wf2Lvyg= https://ybin.me/p/c73726c16d669a5d#WGH7arFJpP8aitaNcBCGqRP3t...
- c0l0 6y ago100ms TLS session setup time with RTT <15ms seems rather unacceptable to me. Can you provide the timings this command produces for you? curl -s -w 'TCP=%{time_connect} TLS=%{time_appconnect} ALL=%{time_total}\n' https://YOUR_REMOTE_HOST_GOES_HERE -o /dev/null That should provide an easy to use ruler to compare measurements.
- zokier 6y agoI was bored so I tested this on AWS from EC2 instance to ELB HTTPS load balancer (in the same VPC), and I got about 5-10ms time_total as reported by curl. So that sort of gives a baseline on what TLS latency can be when network is fast.
- deleted 6y ago[deleted]
- peter_l_downs 6y agowhat are the output units? seconds, or ms? my results make most sense if it's seconds: $ curl -s -w 'TCP=%{time_connect} TLS=%{time_appconnect} ALL=%{time_total}\n' https://<my-domain>/ -o /dev/null TCP=0.021106 TLS=0.052928 ALL=0.143908
- eindiran 6y agoAll of curl's --write-out values that begin with `time_` are given in seconds.
- peter_l_downs 6y agothank you
- axismundi 6y agoI've written a simple bash script based on your one liner: https://ybin.me/p/53f0aa4e1204b470#c561Frz6wpKkfvsTklDdZFOeriVqr1kJtw6GCFo7VUs= https://ybin.me/p/53f0aa4e1204b470#c561Frz6wpKkfvsTklDdZFOer... Here is the output for 100 reqs: https://ybin.me/p/1e68f80f6910ba0d#USqFo3Loksx5rHQIe16NHp299WNIk8Ea91fyliOcZ0Y= https://ybin.me/p/1e68f80f6910ba0d#USqFo3Loksx5rHQIe16NHp299...
- fxtentacle 6y agoMake sure you use HTTP 1.1 or newer and keep-alive. That way, you only have the SSL negotiation overhead on the first HTTPS request, as for all following requests, the connection will be re-used. And then, 100ms for the first request and 15ms for each following request sounds reasonable to me.
- Terretta 6y agoYes, jumping from 15ms to 100ms is rubbish. To compare network latency in terms everyday techies can follow, use video gaming: 15ms latency is 60 fps, and 100ms latency is 10 fps. Which game would they want to play? That said, it's not clear if you mean "time to first byte" or "time to transfer 1000 bytes" or both combined. The first is less of an issue; figure out which part of the request and data transfer timing is changing. (Disclosure: In a past life I built a white label global content delivery network.)
- user5994461 6y agoI down voted against the example. It's really bad because latency has no relevance to FPS and games can actually approach 100 ms and work just fine (consider rendering time and TV delay as well). It's fine because the latency is constant. Your brain internalizes that there is a delay between moving buttons and things happening. Your brain internalizes that you need to shoot where things are going not where they are now. If anything, latency is an example that consistency is more important than speed.
- Terretta 6y agoLatency has no relevance to FPS (frames per second, not first person shooter)? Why do gamers care about not just TV frame rates, but how many frames delay (latency) come from the TV's video scalar / processor? When dealing with round trip times (RTT) for TCP or video gaming, latency limits the number of round trips or control loops per second and the gap between input and action. To your consistency point, agree -- jitter matters a lot to human perception and response too, why so many games "lock" to 30 or 60 fps rather than vary at rates above 30 or 60. Also, it was not an example. It was an analogy to put the differences in milliseconds into amounts of time that matter to a broader set of people than networking times matter to.
- user5994461 6y agoLatency is not related to frame per seconds, yes. You could feed a thousand frames per second to a TV and the TV can have 100 ms latency (delay to display a frame after it received it). You could feed 30 frames per second to the same TV and it'd still have a 100 ms latency. The latency is rarely advertised. It's highly variable per TV and per image. There's the time it takes to start displaying an image and the time until the image is fully set (it is much longer to go white to black than white to grey).
- NovemberWhiskey 6y agoIt depends on exactly what and how you're measuring and why. TLS has a non-trivial bootstrap cost, due to the need for extra network round-trips during establishment (mitigated with 0-RTT, if available), the additional byte overhead of certificate exchange (which can be substantial relative to small payloads if you have a long chain or are sending unnecessary certs, and more so if you're using RSA keys), and the cost of performing the crypto operations for the public key part and key exchange. So if your use-case is "client arrives from the blue with a one-off API request and latency is important" then you are going to suffer with TLS. If this is a web application (on the other hand), then what's actually going to happen is that the browser is going to establish HTTP/1.1 connections over TLS to the load-balancer, keep them alive, and re-use them. Assuming a well-configured load-balancer, it's also going to have the same from it's back-end to your actual service implementation. However, once that's done, the only necessary overhead is the symmetric key stuff (microscopic) and maybe rekeying in long sessions. Using curl to do a one-and-done connection to your service (as suggested by another poster) will give you an estimate which is only relevant to the first use case I described.
- vld 6y agoHow much network latency is between you and the load balancer, and between the load balancer and the web servers? Normally, the initial TLS connection should be around 4 * the network latency, and subsequent requests should be very close to the network latency. Give httping [0] a try for benchmarks. [0]: https://www.vanheusden.com/httping/ https://www.vanheusden.com/httping/
- toast0 6y agoHave you (or they) made sure the load balancer is using the best options for TLS handshake performance for client facing and server facing connections? If possible, Eliptic Key certificates and ECDHE can be a lot faster than RSA certificates and RSA based DHE. Especially if the load balancer is CPU constrained. Make sure TLS sessions or tickets are honored (tickets prefered over sessions so servers don't need to maintain a session database) In my personal opinion, I'd rather not have the load balancer terminate TLS, but you lose a lot of features that way, and that might not be an option.
- archi42 6y agoFetching a one-TCP-package file via HTTP should take about 2 RTT_1 from a webserver (and ignoring connection tear down). The LB needs another RTT_2 to fetch data over a (hopefully) pre-established TCP connection from your webserver. Assuming a 2ms RTT_2 at the data center, your RTT_1 seems to be about 5.5ms. A quick google search indicates with TLS the whole think takes at least 4 RTT_1, plus the RTT_2 because the LB still has to fetch your data, so your hard limit is 24ms, plus how long it takes to actually transfer the additional data (especially certificates). This leaves about 86ms. Now, does your client perform any checks for a revoked certificate? Assuming these checks are also done via TLS, and a "far away" server with a RTT of 15ms, then that's a whooping 60ms your client is taking to ensure that the certificates are valid. In that case your LB would merely be 26ms slower than the theoretical optimum.
- bogomipz 6y agoA TLS 1.3 handshake adds 1 extra RTT to the normal TCP 3-way handshake(assuming 0-RTT is not in use)
- shyn3 6y agoI haven't worked with Citrix ADC but if it's like any other load balancer you want to look at the SNI configurations settings and the TCP profile/buffer size. Increasing the buffer size may help but from my experience if you are offloading SSL then the buffer window shouldn't matter since the HTTPS connection is not being decrypted at the destination but who knows maybe ADC requires a larger buffer for HTTPS versus HTTP.
- deleted 6y ago[deleted]
- alyandon 6y agoCitrix LBs come in many flavors. At a guess, since you talk about it being externally managed - I'd ask your vendor if you are on dedicated hardware/instance vs being yet another tenant on an (and I might be forgetting my terminology) already oversubscribed VPX instance. I can tell you from my personal experience that +85ms overhead is definitely not what I'd expect for TLS termination.
- axismundi 6y agoI should clarify that 110ms is happening for all requests. Tested with this: https://ybin.me/p/53f0aa4e1204b470#c561Frz6wpKkfvsTklDdZFOeriVqr1kJtw6GCFo7VUs= https://ybin.me/p/53f0aa4e1204b470#c561Frz6wpKkfvsTklDdZFOer... and got this: https://ybin.me/p/1e68f80f6910ba0d#USqFo3Loksx5rHQIe16NHp299WNIk8Ea91fyliOcZ0Y= https://ybin.me/p/1e68f80f6910ba0d#USqFo3Loksx5rHQIe16NHp299...
- viraptor 6y agoYou're still testing only the initial connection timing this way. Curl does not preserve the TLS sessions between executions by default. To test session resumption you can do: "curl https://your-url https://your-url https://your-url https://your-url https://your-url https://your-url https://your-url https://your-url" instead.
- axismundi 6y agoHere are some details from SSL Labs checker: https://ybin.me/p/c73726c16d669a5d#WGH7arFJpP8aitaNcBCGqRP3t4e/JSSvQfm2wf2Lvyg= https://ybin.me/p/c73726c16d669a5d#WGH7arFJpP8aitaNcBCGqRP3t...