10 ms·
The Case of the 50ms request
- YarickR2 5y agoError: <<you-said>>: error within widget contents (Error: cannot find a closing tag for HTML <pid>) This hints to a possible XSS and/or code injection (not completely quoted input). Input was "strace -s128 -f -p <pid>" , as an answer to "how do you strace server process"
- noodlesUK 5y agoAs requested by the commenter below (now I want spoiler tags): SPOILER FOR THE GAME I’ve seen TCP_NODELAY all over the place before, but never known why. This was a fun way to find out.
- dbt00 5y agohackernews feature request: spoiler tags. :)
- joshspankit 5y agoWith auto rot-13
- lloeki 5y agoRot13 the spoiling text!
- rozab 5y agoecho "rnfl fcbvyre gntf!" | tr a-z n-za-m :)
- tommek4077 5y agoJavaScript is required. Please enable it to continue. No
- noahtallen 5y agoIt’s an interactive page, not a blog post
- neolog 5y agoIt's a webpage with buttons that take you to another webpage.
- slx26 5y agoGiven the technical nature of the puzzles/adventures, I guess that if there's something we could all agree on is that the message about why javascript is required could be a bit more nuanced. Beyond that, sure, it could be done without javascript, but it's easy to see how it helps using javascript. It's a pretty fair use-case.
- klodolph 5y agoIt’s stateful. You can visit the same location with different states.
- keinplan 5y agoThats what cookies are for.
- krapht 5y agoWhat's with all the griping going on regarding javascript? Twine is made for these "choose your own adventure" type games, and it's faster to use it than handcode everything while serializing all your app state to a cookie. What's next, a bunch of complaining that electron apps are slow? That people are writing video games in managed languages?
- keinplan 5y agoI also think that Twine/Client-Side-JS isn't the worst choice here. It takes unnecessary load from the server and is easy to use, at least for the one creating the game. It's just so frustrating to watch how having loads of Javascript on websites that absolutely don't need it has been normalized. I don't think people should be blamed for using Javascript (or Electron, .NET, whatever), especially for hobby projects. But I also think users shouldn't be silent about it, especially since there are young developers who actually don't know software could be better. It's just such a massive waste of resources and breaks accesibility way to often.
- CaliforniaKarl 5y agoA comment from the inventor of Nagle’s algorithm: https://news.ycombinator.com/item?id=9050645 https://news.ycombinator.com/item?id=9050645 (tl;dr Try turning off delayed ACK first, especially if you can’t update the code.)
- eddyg 5y agoThanks for pointing out this comment! Setting TCP_NODELAY seems to be the go-to / default solution for some reason (cargo-culting? not understanding the interaction with Delayed ACKs?) when there are times when Nagle’s algorithm could probably help.
- lmilcin 5y agoAlso fun: try to tunnel TCP over TCP.
- wonnage 5y agoI ran into an interesting variation of this where we shouldn't have had any problems with small packets, but it turned out we had having jumbo frames enabled in AWS (which seems to be a default now). Together with gzip, you can actually have a bit of trouble filling up a packet, which will then be delayed by the commonly mentioned interaction with delayed ACKs.
- deleted 5y ago[deleted]
- tpetry 5y agoA few months ago there had been multiple articles about this behaviour but i really don‘t remember the details anymore. Does anyone know a writeup with a detailled explanation to understand how it is happening and tests to see whether your systems are affected?
- captn3m0 5y agoThere’s 5 links once you finish the puzzle
- mansoor_ 5y agoA couple bugs in this game, I wanted to SSH onto the client, took me to the server instead.
- self 5y agoIt does ssh you to the client -- the text is wrong.
- gizmo 5y agoPretty fun puzzle, but this kind of debugging is alien to me. Without going into too much detail, it's pretty easy to see something suspect is going on, and this hunch can be easily tested. This debugging tour takes you to strace, wireshark, all sorts of other low level debugging techniques, when really all you had to do was simulate the client with curl -d and the problem would have been pretty obvious. And in this case the more complicated debugging tools didn't even explain anything. As the last page says, the answer is kind of a leap. If you already knew about this problem curl would have solved it immediately, and if you didn't you'd still be baffled even after knowing exactly why the stack traces are the way they are.
- spuz 5y agoHow would using curl -d make the problem obvious?
- barrkel 5y agoIt wouldn't have split the request over separate sends. That's what flushHeaders() is doing; it does a separate send for headers vs the body of the request. https://nodejs.org/api/http.html#http_request_flushheaders https://nodejs.org/api/http.html#http_request_flushheaders
- spuz 5y agoOk, I think that would have helped me learn that the client can some how trigger the bug but I don't think it would have allowed me to narrow it down to flushHeaders.
- gizmo 5y agoThere is no such thing as flushing a TCP socket which is why that function jumped out at me. curl would have confirmed the server isn't misbehaving. Googling flushHeaders shows it's a node implementation specific thing and the docs explain that it has to do with the way node buffers http headers. Which makes sense because you don't want to call socket.send() for every http header you add to your request. But you also don't want to send() in between your headers and the body, because then you violate TCP package size expectations.
- elric 5y agoThis is great. Julia Evans has some great content in general, she's really good at explaining complex topics in an engaging and accessible way.
- 1vuio0pswjnm7 5y agoAll due respect, this is a neat advertisment for the "storytelling" Javascript library she is using, but I learn much more by reading W R Stevens' books. There is more to TCP/IP than what one can do through Berkeley sockets. Plus reading Stevens' books does not require Javascript.
- makach 5y agoThat was exceptionally fun. I thought I had the answer but I was completely wrong. I shouldn't have stopped the debugging and rush to the solution. Unfortunately it is a game, and it allowed me to do it. To me this seems pretty obscure and you debug pretty deep into and outside of your application. One part of me thinks of this as Somebody Else's Problem, but definitively makes me rethink it as a SEP and something devs should know about. Specially in time critical/real time systems.
- nickkell 5y agoReally nice game/tutorial. The best job interview I ever had was framed like this. The interviewer told me there was a bug in the system and had a stack of pages he'd printed out that would provide successive clues as to what caused it. I could ask them questions, in effect using the interviewer as a search engine/debugger. It was the closest an interview has ever come to simulating the day-to-day of a web developer.
- amalcon 5y agoI've said it before and will say it again: One code review or debugging session gives an interviewer more actionable information than an effectively unbounded number of logic puzzles.
- ExtraE 5y agoJanuary 3, 2032. For several years now, programming has been a required course for all grade-school students. The tech field is flooded with talent, and it is virtually impossible to get a job. The technical interview has grown at an unbounded rate. An aspiring engineer walks into the interview room, freshly shaven. He sits down, cracks his knuckles, takes a long sip from his branded thermos. Waits. And then it begins. The interview. 10, 100, 1000 logic puzzles. Faster than seems possible, he recites answers memorized from algoexpert.io. Cut to a wide shot. Speed up time. Stubble, then a beard, appear on his face. The sun rises and sets, and yet he dare not sleep. Eventually, he forgets language, reason, civilization, coffee. The touch of his baby son's skin, although he's not such a baby anymore. He grunts, diagrams, and whiteboards. That's all his life is now. "Ok, well thank you very much! You'll hear from us soon," Says the interviewer. The sounds are foreign to the engineer, but he is lead out of the office and onto the bright street. His car sits there, rusted. The interviewer motions the next candidate inside. He doesn't hear from them.
- galimaufry 5y agoI think you have rediscovered Forlesen! https://www.wolfewiki.com/pmwiki/pmwiki.php?n=Stories.Forlesen https://www.wolfewiki.com/pmwiki/pmwiki.php?n=Stories.Forles...
- deleted 5y ago[deleted]
- fmajid 5y agoI got it straight away, but I have a telecoms engineering and networking background. This just goes to show how poorly networking is taught in most courses (as are databases), and specially in boot camps. Self-taught programmers are also very unlikely to be exposed to this kind of topic. One thing that was not offered as an option was to use a packet analyzer like tcpdump or Wireshark, even though that is the most reliable and systematic way to get to the bottom of many performance problems. You'd think the popularity of the network tab in Chrome's dev tools would make this less scary.
- b0afc375b5 5y agoI was afraid of using tcpdump until I learned that the ip and tcp packet has a very specific structure. I had to use tcpdump+wireshark for debugging recently and it felt like having a mini-superpower.
- rkachowski 5y agoIf you're talking about the game, then there is the option to use tcpdump directly on the client.
- fmajid 5y agoOh, OK, I didn't see it and I went straight to "GUess the problem" or whatever it was called.
- deleted 5y ago[deleted]
- lbriner 5y agoThe interesting thing for me is that I would suspect most devs including myself would assume that if the request takes 50ms, that's how long it takes (because networks!). I wonder how many of us are able to judge how long something should take? Not me, except anecdotally.
- amcvitty 5y agoI used to work in an overseas office and quickly learned the ping times to the other branches worldwide so got a “feel” for how long things should take. 10/10 would recommend
- pgt 5y agoSPOILER ALERT I answered "req.flushHeaders()" but surprisingly it doesn't accept that as a cause, even though the headers would be sent with the initial packet and should improve the latency.
- sdflhasjd 5y agoIn the solution section, one of the answers is about preventing the message being split into two packets.
- adriancr 5y agoThis was fun and a lesson to look at the code first before jumping into tcpdump/tracing
- jvns 5y agoAuthor here. I wrote a post about the design of this game on my blog: https://jvns.ca/blog/2021/04/16/notes-on-debugging-puzzles/ https://jvns.ca/blog/2021/04/16/notes-on-debugging-puzzles/
- deleted 5y ago[deleted]
- brna 5y agoNice, but not perfect: ``` You said: "strace -p $(ps aux| grep server.py| grep -v "grep"| awk -F ' ' '{print $2}')". To strace the server, first you need to find its PID. You know that the program is called server.py. ```
- jedimastert 5y agoIt doesn't actually check your answer. You're supposed to check your answer against the game yourself
- impoppy 5y agoLoved it! Although it didn’t give a comprehensive answer on how to preventively solve the problem on any platform. Is there one? Or should software developers just stick to one way or another like some kind of an unspoken rule?
- rhacker 5y agogrr.. I felt stupid going through the puzzle.. but I was looking at that flushHeaders call and thought that might be a problem - simply because I never call that.
- emeraldd 5y agoTwine and SugarCube! Interesting to see that pop up here: https://www.motoslave.net/sugarcube/2/ https://www.motoslave.net/sugarcube/2/
- avidiax 5y agoKnowledge of delayed ack and nagle's algorithm wasn't necessary to solve. The explanation didn't mention the flushHeaders call, which is apparently the fix. I didn't run any tests, just looked at the JS and figured that sending 2 packets is worse than sending 1 w.r.t. latency. It's also a pretty strong intuition that the client side tends to have issues, since the server is usually well-tested and standardized. Also, very often people are measuring wrong, so checking the JS to be sure the time recorded is accurate is also important. https://nodejs.org/api/http.html#http_request_flushheaders https://nodejs.org/api/http.html#http_request_flushheaders
- Seb-C 5y agoAgree, this knowledge is unnecessary. The first thing I did was use curl to check if it's a client or server problem. Then I looked at the code and immediately noticed the unnecessary flush call, the unnecessary loading of the whole file in memory and the unnecessary fs.stats.
- bajsejohannes 5y agoThere's a huge difference between solving an instance of a problem and understanding the problem. Enough commits of "this seems to fix the problem" and the codebase becomes really hard to work with.
- avidiax 5y agoIt's not reasonable to expect that developers understand how everything works. There are billions of lines of code deployed that work for reasons that the developers either don't know or only think they know. In this case, in my judgement, you only need to know about the delayed-ack/Nagling concepts if there's a engineering requirement to keep the flushHeaders call, and the 50ms latency is a dealbreaker (i.e. not merely a curiosity). Even then, it's not clear how to fix unless you can disable one of those features.
- axaxs 5y agoBased on the description, guessed 'nagle' without any debugging. Not sure if it implies I was right or wrong, but it explained Nagle's algorithm. Asked what to do about it and typed 'tcpnodelay'. It replied it wasn't smart and asked me to click a button. Feels pretty basic to anyone who's ever really touched TCP in code.
- 1vuio0pswjnm7 5y ago"Also, the Linux kernel doesn't always enable delayed ACKs -- to reproduce this I actually had to write a Python program that explicitly turns them on. I haven't been able to find a clear explanation of exactly when delayed ACKs are used." Delayed ACKs can be enabled on Linux kernels 3.11+ with ip(8). ip route change ROUTE quickack 1 On MacOS and Windows, delayed ACKs can be configured through sysctl and the registry, respectively. Delayed ACKs may be used in response to congestion. For example, in bulk, i.e., non-interactive, transfers with large packets, delayed ACKs can be useful. This is covered in Chapters 15 (15.3) and 16 of Stevens' TCP/IP Illustrated Vol. 1. This draft suggests delayed ACKs are useful during TLS handshake. https://tools.ietf.org/id/draft-stenberg-httpbis-tcp-03.html https://tools.ietf.org/id/draft-stenberg-httpbis-tcp-03.html Also, socat allows for setting TCP options via setsockopt. No need to write a new program.
- 1vuio0pswjnm7 5y agoip route change ROUTE quickack 1 dev STRING