17 ms·
Why does SSH send 100 packets per keystroke?
- snowmobile 8mo ago> That 20ms is a smoking gun - it lines up perfectly with the mysterious pattern we saw earlier! Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"
- Hikikomori 8mo agoIt's a smoking gun of Claude usage.
- lloydatkinson 8mo agosmoking gun, you're absolutely right, good question, em dash, "it isn't just foo, it's also bar", real honest truth, brutal truth, underscores the issue, delves into, more em dashes, <20 different hr/corporate/cringe phrases>. It's nauseating.
- cubano 8mo agoCome on...haven't we all had to deal with the crazy smart lead who was loaded with those same types of annoying tics? Considering what these LLMs bring to the table, I think a little tolerance for their cringe phrases is in order.
- jcynix 8mo agoIt's what they read on The Internets when training, so don't expect them to generate new phrases, other than what they learned from it?
- Terretta 8mo ago### The answer that fits everything (and what to do about it)
- calvinmorrison 8mo agocant wait for chatgpt to make me read about grandmas secret recipe and scroll through 6 ads to see the ingredients for my chicken teriyaki dinner
- jcynix 8mo agoMaybe we need a real AI which creates new phrases and teaches the poor LLMs? Looking back we already had similar problems, when we had to ask our colleagues, students, whomever "Did you get your proposed solution from the answers part or the questions part of a stackoverflow article?" :-0
- MaxBarraclough 8mo agoThat's the point though, it doesn't reflect human usage of the word. If delve were so commonly used by humans too, we wouldn't be discussing how it's overused by LLMs.
- hamdingers 8mo agoYou might find this a fun read: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
- Telemakhos 8mo agoI've love to delve into that. https://pshapira.net/2024/03/31/delving-into-delve/ https://pshapira.net/2024/03/31/delving-into-delve/
- bdamm 8mo agoWithout knowing how LLM's personality tuning works, I'd just hazard a guess that the excitability (tendency to use excided phrases) is turned up. "smoking gun" must be highly rated as a term of excitability. This should apply to other phrases like "outstanding!" or "good find!" "You're right!" etc.
- yread 8mo agoChatGPT too. And "lines up perfectly" when it doesnt actually line up with anything
- dave78 8mo agoSame with Gemini.
- MonkeyClub 8mo agoYou can absolutely see this pattern in Gemini in 2026. Btw, is the injection of "absolutely" and "in $YEAR" prevalent in other LLMs as well, or is it just in Gemini's dialect?
- cristoperb 8mo agoIt's just Gemini. I'm guessing they changes the system prompt for the new year or something, but it's pretty annoying.
- nurettin 8mo agoI've had gemini tell me "We are debugging this problem here in İstanbul" and talking about an istanbul evening, trying to give uplifting or familiar vibes while being creepy. I think there was a setting about time and location which finally got rid of that behavior.
- locallost 8mo agoI chuckled out loud. It's funny cause it's true.
- redwall_hp 8mo ago"You're so right, that nice catch lines up perfectly!"
- smallmancontrov 8mo agoIt's not just a coincidence, it's the emergence of spurious statistical correlations when observations happen across sessions rather than within sessions.
- cipehr 8mo agoI don't think claude has even once used this in my conversations (Claude Desktop, Claude Code, Voice conversations...) Sycophancy, yes absolutely! Maybe it has something to do with your profile/memories?
- eieio 8mo agoYes! While this post was written entirely by me, I wouldn't be surprised if I had "smoking gun" ready to go because I spent so much time debugging with Claude last night.
- gf000 8mo agoReminds me of ethimology nerd's videos. He has some content about how LLMs will influence human language.
- grim_io 8mo agoThe "maybe" of yesterday is the "you're absolutely right!" of tomorrow.
- hinkley 8mo agoSome day in the future we will complain about AIs with a 2015 accent because that’s the last training data that wasn’t recursive.
- ranger_danger 8mo agoshouldn't it be "human language influences human language"?
- rubslopes 8mo agoIt's interesting how LLMs influence us, right? The opposite happened to me: I loved using em dashes, but AI ruined it for me.
- andai 8mo agoI still love using emdashes, and people already thought I was a robot! https://xkcd.com/3126/ https://xkcd.com/3126/ Soon the Andy 3000 will finally be a reality...
- pcthrowaway 8mo agoThat's a sweet ass—reference
- jcynix 8mo agoYou might see certain phrases and mdashes ;-) rather often, because … these programs are trained on data written by people (or Microsoft's spelling correction) which overused them in the last n years? So what should these poor LLMs generate instead?
- simonjgreen 8mo agoI see it from GPT5 too a lot
- Fnoord 8mo ago> Speaking of smoking guns Oh shoot! A shooting. So the TL;DR of this post is: don't change this setting unless you know what you're doing.
- kevin_thibedeau 8mo agoChastise it with a reminder that you're using smokeless powder.
- observationist 8mo agoOr the "Eureka! That's not just a smoking gun, it's a classic case of LLMspeak." Grok, ChatGPT, and Claude all have these tics, and even the pro versions will use their signature phrases multiple times in an answer. I have to wonder if it's deliberate, to make detecting AI easier?
- WesolyKubeczek 8mo agoA computational necromancer has likely figured out a way to power a data center by making Archimedes spin in his grave very fast.
- nurettin 8mo agoAt this I'm just so glad that "you're absolutely right!" phase is over.
- layer8 8mo agoYes, it’s kind of a corpus delicti. ;)
- jcims 8mo agoI'm working on a little SRE agent to pre-load tickets with information to help our on-call and I'm already tired of Claude finding 'smoking guns'.
- HPsquared 8mo agoThey love clichés, and hate repeating the same words for something (repetition penalty) so they'll say something like "cause" then it's a "smoking gun" then it's something else
- ycombinatrix 8mo agoYou can also use TCP_CORK to reduce the number of packets without any increased latency. Disabling TCP_NODELAY would also reduce number of packets + be portable & simpler to implement - but would incur a latency penalty.
- eieio 8mo agoOh wow - I've never heard of TCP_CORK before. Without disabling pings I'd still pay the cost of receiving way more packets, but maybe that'd be tolerable if I didn't have to send so many pongs. This is super handy; excited to play around with it. I am aware of TCP_NODELAY (funny enough I recently posted about TCP_NODELAY to HN[1] when I was thinking about it for the same game that I wrote about here). But I think the latency hit from disabling it just doesn't work for me. [1] https://news.ycombinator.com/item?id=46359120 https://news.ycombinator.com/item?id=46359120
- joshstrange 8mo agoI missed that thread originally, the post and the comments where a good read, thank you for sharing. I got a kick out of this comment [0]. "BenjiWiebe" made a comment about the SSH packets you stumbled across in that thread. Obviously making the connection between what you were seeing in your game and this random off-hand comment would be insane (if you had seen the comment at all), but I got a smile out of it. [0] https://news.ycombinator.com/item?id=46366291 https://news.ycombinator.com/item?id=46366291
- eieio 8mo agowow, I missed that comment, that's an incredible connection. Thank you!
- BenjiWiebe 8mo agoFirst time I've been reading on HN and come across my name randomly.
- 8mo ago
- cheschire 8mo agoI enjoyed this write up as it touched on several topics I enjoy reading about. Also I was unfamiliar with SSH being vulnerable in the past to keystroke timing!
- pixl97 8mo agohttps://news.ycombinator.com/item?id=37307708 https://news.ycombinator.com/item?id=37307708 2023 discussion about it here.
- dathinab 8mo ago> Keystroke obfuscation can be disabled client-side. please never do that (in production) if anyone half way serious tries they _will_ be able to break you encryption end find what you typed this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.
- lazypenguin 8mo agoThey literally explain the mechanism in the post and then explain why the security tradeoff made sense for their ssh game………
- shadowgovt 8mo agoBut they'd have to be on the same network as me to do that attack, right?
- benlivengood 8mo agoYep, like ECHELON and friends are. The metadata recorded about your (all of our) traffic is probably enough to perform the timing attack.
- shadowgovt 8mo agoHey, if ECHELON snuck a listener into my house, where six devices hang out on a local router... Good for them, they're welcome to my TODO lists and vast collection of public-domain 1950s informational videos. (I wouldn't recommend switching the option off for anything that could transit the Internet or be on a LAN with untrusted devices. I am one of those old sods who doesn't believe in the max-paranoia setting for things like "my own house," especially since if I dial that knob all the way up the point is moot; they've already compromised every individual device at the max-knob setting, so a timing attack on my SSH packet speed is a waste of effort).
- advisedwang 8mo agoThat doesn't sound right to me. This obfuscation isn't about a side-channel on a crypto implementation, this is about literally when your keystrokes happen. In the right circumstances, keystroke timing can reduce the search space for bruteforcing a password [1] but it's overstating to describe that as broken encryption. [1] https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf
- swiftcoder 8mo ago> Obviously forking go’s crypto library is a little scary, and I’m gonna have to do some thinking about how to maintain my little patch in a safe way This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth
- eikenberry 8mo ago+1... Given how much SSH is used for computer-to-computer communication it seems like there really should be a way to disable this when it isn't necessary.
- jacquesm 8mo agoIn practice I've never felt this was an issue. But I can see how with extremely low bandwidth devices it might be, for instance LoRa over a 40 km link into some embedded device.
- geocar 8mo agoHah no. Nobody is running TCP on that link, let alone SSH.
- jacquesm 8mo agohttps://github.com/markqvist/Reticulum https://github.com/markqvist/Reticulum and RNode would be a better match.
- dsrtslnd23 8mo agoIn aerial robotics, 900MHz telemetry links (like Microhard) are standard. And running SSH over them is common practice I guess.
- BenjiWiebe 8mo agoWhy do you guess? I wouldn't expect SSH to be used on a telemetry link. Nor TCP, and probably not IP either.
- pixl97 8mo ago>very confidently told me that my tcpdump output was normal ssh behavior: I mean, for modern version of Openssh it's not exactly wrong. The failure was to tell you why that is the normal behavior.
- raggi 8mo ago> I am working on a high-performance game that runs over ssh. WAT. Please no.
- shitter 8mo agoWhy not? If it's high-performance, it's fine.
- pseidemann 8mo agoPerforming with highly elevated privileges? (Joke)
- qudat 8mo agoSSH suffers from tcp-in-tcp issues which means it’ll always take a performance hit over other protocols
- PunchyHamster 8mo agoIf you spend entire CPU to process few megabits of SSH traffic, it isn't high performance
- zamadatix 8mo agoVery interesting, I hadn't heard of this obfuscation before so it was well worth clicking. Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet. For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.
- charcircuit 8mo agoIt made the front page when it was added. https://news.ycombinator.com/item?id=37307708 https://news.ycombinator.com/item?id=37307708
- jachee 8mo agoNot everyone sees the HN frontpage every day, and sometimes especially-esoteric things spend a fairly short timespan on there.
- sam_lowry_ 8mo agoSadly, much fewer computer systems have telnet nowadays. Also, port 21 is often blocked.
- tracker1 8mo ago23 (21 is ftp)
- sam_lowry_ 8mo agoYup
- moffkalast 8mo agoTelnet, I want her to know it was me.
- Veserv 8mo agoThe really mysterious part is how ~10,000 packets per second costs ~20% of a core. That would mean SSH is bottlenecking in its code at ~50,000 packets per second per core which would be ~500 Mbps per core (assuming full packets) which is ludicrously slow. It is trivial to do 10x that packet per second rate. Is SSH really that poorly designed?
- diath 8mo ago> It is trivial to do 10x that packet per second rate. When making this statement, are you taking into account that SSH encrypts the traffic by default?
- Veserv 8mo agoI do not know where people get the idea that encryption is that slow. Standard AES hardware acceleration instructions do ~25 Gbps per core (on a 2023 CPU) which is ~50x that rate [1]. I have heard modern cores can do ~40-50 Gbps, but I have not been able to find any independent benchmarks of that. Even the Intel i5-2500, a CPU from 2011, averages ~10 Gbps which is ~20x that rate. Even unaccelerated encryption can do ~2-5 Gbps in pure software which is 4-10x the SSH rate. And in this situation, the amount of encrypted payload in each packet is 36 bytes which is ~40x less than a full packet of ~1500 bytes. You would almost surely hit packet per second limits before you hit payload throughput limits at these small sizes. Encryption is slow when compared to data throughput you can get with a properly designed transport stack, but that is because it is in comparison to 100 Gbps per core even with no hardware offload. Anything less than ~10 Gbps/1 million packets per second (ignoring other bottlenecks, so only the software transport is the limit) is not merely unoptimized, it is pessimized. [1] https://calomel.org/aesni_ssl_performance.html https://calomel.org/aesni_ssl_performance.html
- PunchyHamster 8mo agodoing a gigabit takes ~35% of single core to saturate my 1Gbit ethernet. On i3-3250 which is 12 years old CPU Your assumptions are way off
- svnt 8mo ago> I am working on a high-performance game that runs over ssh. Found your problem. But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.
- ycombinatrix 8mo agoYea UDP is technically more performant, but then you need a crypto layer + reliable message delivery layer + bespoke client. Using a plain old SSH client is cool. However, there are existing libraries for exactly this use case - see https://github.com/ValveSoftware/GameNetworkingSockets https://github.com/ValveSoftware/GameNetworkingSockets I guess QUIC libraries would also work.
- convolvatron 8mo agoits not really a question of 'udp performs better'. in tcp we have to live to head-of-line blocking on losses and congestion control. if you don't care about receiving every packet, but only the most recent, then udp is a good choice. running without congestion control means that you avoid slowstart. but at a certain rate you run into poorly defined 'fairness' issues where you can easily negatively impact other flows. past that point, you can actually self-interfere and cause excessive losses for yourself. quic uses congestion control, but uses latency estimates and variance as a signal to back off. it still imposes an ordering on a per-stream basis. so it might not be ideal either. sctp has a mode which supports reliable and unordered, which might be something to consider so really - if you care about latency and have a different reliability model, its worth unpacking all these considerations and using them to select your transport layer or even consider writing a minimal one yourself
- ycombinatrix 8mo ago>in tcp we have to live to head-of-line blocking on losses and congestion control. Is this not a performance consideration? Either way, using plain old SSH means a metric bajillion computers have a client for your game built in.
- PaulHoule 8mo agoI find it disturbing. One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users. Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself. It's OK if you work for Google where there is limitless dark fiber but what about people in African countries? It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.
- starttoaster 8mo agoThere's a good chance you have other options. Regardless of how you feel about the company's head, Starlink would probably be one of them, with likely better performance than you're dealing with on ADSL. But you cannot just sue a company because their network connected software doesn't work well on slow networks. Let alone a project like OpenSSH. It would be like me suing a game studio because my PC doesn't meet their listed minimum requirements to play the game.
- PaulHoule 8mo agoHey, it is one thing to buy a new computer, it is another thing to ask people to move. A better analogy is a bank redlining neighborhoods. The cost to run fiber to difficult rural locations pays itself easily if you look at a 25-year time span and is an order of magnitude less than building a new housing unit on the West Coast.
- SAI_Peregrinus 8mo agoIt's OSS with no warranty. You can compile it yourself with the option disabled. It's only ever on for pty connections (physical user with a keyboard), there's no added traffic for ttys.
- idontwantthis 8mo agoIf security doesn’t matter then why not use telnet or something else besides ssh instead of forking a security library?
- layer8 8mo agoTelnet nowadays typically isn’t available by default for security reasons, and OP wants people to be able to play the game just by typing “ssh thegamehost”.
- AceJohnny2 8mo ago> Telnet nowadays typically isn’t available by default for security reasons And with good reason. This CVE is from yesterday: https://nvd.nist.gov/vuln/detail/CVE-2026-24061 https://nvd.nist.gov/vuln/detail/CVE-2026-24061 > telnetd in GNU Inetutils through 2.7 allows remote authentication bypass via a "-f root" value for the USER environment variable.
- layer8 8mo agoTelnetd is the server though, and OP wouldn’t be using that.
- AceJohnny2 8mo agoah, good point.
- kenmacd 8mo ago@eieio: whatever email protection you're running is triggering on the extension info. For example I see: > And they’re sent to servers that advertise the availability of the [email protected] extension. What if we just…don’t advertise [email protected]?
- eieio 8mo agoIs it possible that this is on your end? The extension is "ping@openssh.com." It shows up in the blog reliably for me across several browsers and devices.
- wizzwizz4 8mo agoNo, it's Cloudflare munging the HTML. Cloudflare then provides JavaScript to un-munge it, but that's not reliable.
- eieio 8mo agoTIL! I'll see if I can change that.
- qingcharles 8mo agoAnd of course it totally doesn't work if the client doesn't have JavaScript at all. I read the HN front-page through an AI summary and it also got censored when it scraped the article.
- Animats 8mo agoIn 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys. Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.
- adgjlsfhk1 8mo agoTyping with an extra 50ms latency will be fairly unpleasant.
- omoikane 8mo ago> send all typed characters at 50ms intervals Wouldn't this just change the packet interval from 20ms to 50ms? Or did you mean a constant stream of packets at 50ms intervals, nonstop? I think the idea behind the current implementation is that the keystrokes are batched in 20ms intervals, with the optimization that a sufficiently long silence stops the chaff stream, so the keystroke timing is obfucated with an increased error bar of 20ms multiplied by number of chaff packets.
- mystraline 8mo ago[flagged]
- frotaur 8mo agoThe problem is not knowing whether someone is typing, as far as I understand. But that you may extract some information about what keys are being typed, based on the small differences in timings between them.
- davidhyde 8mo agoI wonder if this is the same reason why Microsoft's Remote SSH plugin on VS Code is so flaky even with a decent internet connection. Every couple of months I try to give it another go and give up due to the poor keyboard latency I inevitably experience. And the slow reconnects whenever I glance away from my computer monitor briefly. This is on a fiber connection with a 20ms ping to the remote machine.
- WesolyKubeczek 8mo agoYou surely mean the latency in its embedded terminal and not the code editor, right? I use VSCode’s remote SSH specifically so that code editing doesn’t suck. It really does not.
- davidhyde 8mo agoYou're right, the latency is in the embedded terminal. Perhaps it is trying to run SSH inside SSH. Still, the disconnects are a pain too.
- JohnLeitch 8mo agoThe reliance on LLMs is unfortunate. I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark. The Wireshark dissectors are quite mature, SSH is covered fairly well.
- MrDarcy 8mo agoHow much are you staking on that bet?
- mystraline 8mo agoSigh. I'm still waiting for a systems engineering tool that can log every layer, and handle SSL the whole pipe wide. Im covering everything from strafe and ltrace on the machine, file reads, IO profiling, bandwidth profiling. Like, the whole thing, from beginning to end. Theres no tool that does that. Hell, I can't even see good network traces within a single Linux app. The closest you'll find is https://github.com/mozillazg/ptcpdump https://github.com/mozillazg/ptcpdump But especially with Firefox, good luck.
- fragmede 8mo agoReal talk though, how much would such a tool be worth to you? Would you pay, say, $3,000/license/year for it? Or, after someone puts in the work to develop it, would you wait for someone else to duct tape something together approximately similar enough using regexps that open source but 10% as good, and then not pay for the good proprietary tool because we're all a bunch of cheap bastards? We have only ourselves to blame that there aren't better tools (publicly) available. If I hypothetically (really!) had such a tool, it would be an advantage over every other SRE out there that could use it. Trying to sell it directly comes with more headaches than money, selling it to corporations has different headaches, open-sourcing it don't pay the bills, nevermind the burnout (people don't donate for shit). So the way to do it is make a pitch deck, get VC funding so you're able to pay rent until it gets acquired by Oracle/RedHat/IBM (aka the greatest hits for Linux tool acquisition), or try and charge money for it when you run out of VC funding, leading to accusations of "rug pull" and development of alternatives (see also: docker) just to spite you. In the base case you sell Hashimoto and your bank account has two (three!) commas, but worst case you don't make rent and go homeless when instead you could've gone to a FAANG and made $250k/yr instead of getting paid $50k/yr as the founder and burning VC cash and eating ramen that you have to make yourself. I agree, that would be an awesome tool! Best case scenario, a company pays for that tool to be developed internally, the company goes under, it gets sold as an asset and whomever buys it forms a compnay and tries to sell it directly and then that company goes under but that whomever finally open sources it because they don't want it to slip into obscurity but if falls into obscurity anyway because it only works on Linux 5.x kernels and can't be ported to the 6.x series that we're on now easily.
- flumpcakes 8mo agoI don't see how Claude helped the debugging at all. It seemed like the author knew what to do and it was more telling Claude to think about that. I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..
- AceJohnny2 8mo agoEven if the chatbot served only as a Rubber Ducky [1], that's already valuable. I've used Claude for debugging system behavior, and I kind of agree with the author. While Claude isn't always directly helpful (hallucinations remain, or at least outdated information), it helps me 1) spell out my understanding of the system (see [1]) and 2) help me keep momentum by supplying tasks. [1] https://en.wikipedia.org/wiki/Rubber_duck_debugging https://en.wikipedia.org/wiki/Rubber_duck_debugging
- NewJazz 8mo agoA rubber ducky demands that you think about your own questions, rather than taking a mental back seat as you get pummeled with information that may or may not be relevant.
- supern0va 8mo agoI assure you that if you rubber duck at another engineer that doesn't understand what you're doing, you will also be pummeled with information that may or may not be relevant. ;)
- stephenr 8mo agoThat isn't rubber duck debugging. It's just talking to someone about the problem. The entire point of rubber duck debugging is that the other side literally cannot respond - it's an inanimate object, or even a literal duck/animal.
- fragmede 8mo ago> I am working on a high-performance game that runs over ssh. Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ https://www.psc.edu/hpn-ssh-home/introduction/ instead Step two, tune TCP/IP stack Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)
- whiterook6 8mo agoTell me more about this game!
- jaimex2 8mo agoIt's vibe coded
- markhahn 8mo agofwiw, I tcdumped between two systems running fedora43 and saw no chaff. (one packet out, one reply, one tcp ack.)
- PunchyHamster 8mo agoOn Debian 13 I get a bunch when just typing interactively on shell instance
- gogasca 8mo ago[dead]
- jaimex2 8mo agoI loled and closed the article after 'I am working on a high-performance game that runs over ssh.' Vibe coders man...
- gafferongames 8mo agoAmen brother
- deepsun 8mo agoWell, security is the #1 consideration for SSH, but if the author doesn't need security, why use ssh? For example, "nc" (netcat) is pre-installed on all platforms where ssh is.
- zinekeller 8mo ago> For example, "nc" (netcat) is pre-installed on all platforms where ssh is. This is technically incorrect, because Windows now includes SSH too!
- perching_aix 8mo agoI seem to hit this logic often recently for some reason. There are two issues with it: - a primary is not a totality: if "security is the #1 consideration for SSH", that implies there's a #2, maybe even a #3 and so on consideration. So the question that follows becomes tautological: "but if the author doesn't need security, why use ssh?" -> surely for one or more of the #2, #3, etc. considerations, right? - overabstraction (*): you ended up strawmanning the author. What they had issue with was keystroke timing obfuscation, which is a privacy feature. Timing attacks are (in part) a privacy concern, and privacy is a security concern, yes, but security is not just privacy concern, and privacy concerns are not just about timing attacks; these groups are not equal. For example, they might very well want the transmitted keypresses themselves to remain confidential, or they might very well want to retain cryptographic assurance of their integrity. These are security features they can continue to utilize by sticking with SSH. All of this is to say, it's not even necessarily them using SSH for a hypothetical #2 or #3 (...etc...) reason, but likely because they still very much want to make use of large chunks of #1, which disabling keypress obfuscation does not actually rid SSH of, only at most weakens it in ways they clearly seem to be okay with. (*) although if I zoom out enough, this is once again just "a primary is not a totality", just implicitly
- breakingcups 8mo agoDepends on what kind of security. They might care about connection integrity. If a faulty (or malicious) router in-between client and server starts malforming packets, `nc` will display those malformed packets. SSH will only show you what the server intended, or nothing.
- xer0x 8mo agoNot related to SSH, but does the eieio.games website make anyone else's monitor flicker? When the website is fullscreen it overwhelms something. I thought my monitor's backlight was going.
- jachee 8mo ago>>> That makes a lot of sense for regular ssh sessions, where privacy is critical. But it’s a lot of overhead for an open-to-the-whole-internet game where latency is critical. Switching to telnet instead of SSH might be an option.
- almosthere 8mo agoBecause we stopped coding for performance years ago.
- OhMeadhbh 8mo agoOr you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency. The big guys already do something similar with HTTP over TLS for DDoS protection and to limit end to end latency on TLS. Granted... it would increase the cost (since you're adding reverse proxies) but it would be a quick way to get acceptable latency, rudimentary DDoS protection, and you could try different connection options independent of the main app's logic. It would be hard to estimate how much latency you're adding with a SSH2 reverse proxy in this case, but it's probably lower than one might think. The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.] But I loved this write-up. It was readable and explained the problem the OP was encountering and proposed solutions well.
- eieio 8mo ago> Or you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency. I've been thinking about some stuff like this! Not being able to put my game behind Cloudflare[1] is a bummer. Substantial architectural overhead though. > The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.] I hear you, but FWIW the patch I was reverting was trivial (and it's also in the go crypto library, which is pretty easy to read). It's a couple-of-line change[2], and Claude did almost exactly what I would have done (I was tired and would have forgotten to shrink the handshake payload). [1] This isn't strictly true, Cloudflare spectrum exists, but its pricing is an insane $1/GB last I checked. [2] https://cs.opensource.google/go/x/crypto/+/833695f0a57b3037385dc9c0073bc88773cae6f3 https://cs.opensource.google/go/x/crypto/+/833695f0a57b30373...
- OhMeadhbh 8mo agoNice, but shouldn't the behaviour change be behind a config setting? And it's not clear what the intent of the change is. Implementing PING/PONG seems different from what you said you were trying to do. And it's section 1.8 of the OpenSSH [PROTOCOL] reference, not section 1.9. But... before you think I'm trying to be negative... good on you. I wish you well. Getting crypto/security code into open source projects can be a slog as people frequently come out of the woodwork, so don't get discouraged. And the more I think about this... there's plenty of examples out there about doing HTTP based reverse proxying, but essentially zero for SSH proxying, so if you do that, it would make a great blog post.
- brendangregg 8mo agoFunny to see this fixed in 2023 and the side effects. Back in 2004, before I focused on performance, I did some security work including inter-keystroke latency analysis of captured SSH sessions to estimate the commands typed: https://www.brendangregg.com/sshanalysis.html https://www.brendangregg.com/sshanalysis.html The 2023 patch should finally fix that 2004 issue.
- jonaslejon 8mo agoMemories! I was at the hacking conference HAL2001 and listening to Dug Song and Solar Designer, who were talking about their SSH timing analysis: https://download.openwall.net/pub/advisories/OW-003-ssh-traffic-analysis/OW-003-ssh-traffic-analysis.txt https://download.openwall.net/pub/advisories/OW-003-ssh-traf... Time flies
- rmunn 8mo agoWow, I did not realize that SSH did that. Good to know, and it makes sense as a default, because the people who need it need to have it on by default. But I think I'm going to be turning that off, because it's a security measure that doesn't make sense for my particular environment: 1) I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing. (Or rsync'ing a file, which ditto). 2) I'm not in a high-security environment where nation-states have an interest in sniffing my keystrokes. 3) I often open SSH connections to servers in other continents. Those underwater cables have massive bandwidth, but they're also in constant use by thousands upon thousands of people. So anything I can do to reduce my bandwidth by 100x is probably worth doing. Any reason you can think of why I should not be setting ObscureKeystrokeTiming=no in my ~/.ssh/config?
- fulafel 8mo agoI think those all have reasonable counterarguments: (1) This sounds brittle. Are you really going to have a good mental model about what's secret when using ssh and reliably refrain from typing those things? Seems to kinda defeat the idea of securing the channel. Also, as a collection your activities might be more confidential to you than single inputs, or correlated with your other activities outside ssh, etc - it's hard to keep a mental model of this as well. Aka optimism is not a form of security. (2) There isn't a reason to think this is a difficult attack that only a powerful adversary could mount. Seems like a college lab level thing to me. And very amenable to AI help as well. Also here optimism is not a form of security. It's a 25 year old attack[1] so there's a lot of existing research[2] around. (3) Saving 100x bandwidth on single keystrokes on an internet dominated by video traffic just because it's 100x doesn't make sense. Also it's good to cultivate a mindset that steers away from trading off security in favour of trivial resource savings. [1] https://www.usenix.org/conference/10th-usenix-security-symposium/timing-analysis-keystrokes-and-timing-attacks-ssh https://www.usenix.org/conference/10th-usenix-security-sympo... (probably older stuff exists outside open literature) [2] eg https://crzphil.github.io/posts/ssh-obfuscation-bypass/ https://crzphil.github.io/posts/ssh-obfuscation-bypass/
- 8mo ago
- qudat 8mo agoNice job! I need to learn how to use tcpdump apparently
- varun_ch 8mo agoFunny that this comes up today! I was just looking into adding a keyboard monitor to my website (I have a goal of making my 'contact me' page have oddly specific information). I wouldn't show the actual keys, just show a blinking light when there's activity, but I guess the timing really could expose quite a lot of information. I did add a trackpad monitor though. It shows my raw MacBook trackpad data. https://varun.ch/contact/ https://varun.ch/contact/
- bibimsz 8mo agoI got 99 packets but an SSH aint one
- eru 8mo agoHmm, if the author is doing something high performance, they should probably use whatever mosh is doing to update the screen, not ssh.
- theblazehen 8mo agoThat would require end users to install additional software though, which they do not want
- eru 8mo agoOh, true, ssh is not just the protocol, but also the name of the client software. Though I would suggest to make mosh available, too. Many nethack servers are available via mosh and ssh. (And in an earlier age, telnet.)
- dgan 8mo ago"The smoking gun!" got me laughing, i am not a native english speaker and only ever seen that expression from Claude, and who knew? Its gaining popularity!
- zoobab 8mo agoJust replace it with zeromq with curvemq. I could vibecode an SSH zmq daemon in an afternoon.
- fuxirheu 8mo agoHow do the HPN patches compare?
- taegee 8mo agoNo TLDR. -.-
- rurban 8mo agoWait, go back to the first sentence: > I am working on a high-performance game that runs over ssh. The TUI for the game is created in bubbletea 1 and sent over ssh via wish. > The game is played in an 80x60 window that I update 10 times a second. I’m targeting at least 2,000 concurrent players, which means updating ~100 million cells a second. I care about performance. High performance with ssh and wish? For sure not. Rather use UDP over secure sockets. Or just normal sockets. Even Claude would come up with much faster code than the ssh/wish nonsense. Or mosh, but this also too complicated.
- puilp0502 8mo agoThe author wanted people to be able to just "ssh mygame", no? In that sense, ssh was a design requirement.
- rurban 8mo agoI didn't think about such throwback to the 80ies. Could be, yes. But then he cannot control the ssh option, and with 2000 users, maybe 10 would set it. I don't think so.
- lighthouse1212 8mo agoThe 2023 timing obfuscation is a nice case study in security defaults vs edge cases. Most SSH users won't notice 100 packets per keystroke - it's noise in the bandwidth budget. But for high-frequency terminal apps, it becomes the dominant cost. At 2000 concurrent players updating 80x60 chars at 10fps, a custom protocol might be the right answer regardless of obfuscation settings.
- sam_lowry_ 8mo agoJust think of the trees burnt in the name of security!
- deleted 8mo ago[deleted]
- Sebb767 8mo agoEach of our devices spents a lot of energy dedicated to encryption. By now, all disks you did not set up manually are most likely encrypted and hardly any unencrypted package will travel out of your network. That's not to mention the tons of load and dedicated hardware we have just to terminate https and scan traffic for suspicious activity or the hardware being replaced because it's internal security triggered/broke. In a perfect world, we could send all traffic completely unencrypted and never scan for a malicious payload, saving all that energy and hardware. But we do not live in that world and drawing the line with this minor, mostly unintrusive security feature seems strange.
- sam_lowry_ 8mo agoShouldn't we sacrifice some security for convenience? And shouldn't we at least have a public discussion where to draw the line? I already don't encrypt my Pinebook storage, because the device is low-powered. I now disabled ObscureKeystrokeTiming on the ssh clients where it does not matter. And it should not matter in 99.9999% of cases. P.S. There's a good reason airline frequencies are unencrypted AM and I hope IT "security" mindset does not reach its dirty hands up the air.
- hgo 8mo agoIt's wonderful to see LLM's being used to increase the programming community's general quality level of work, as more things become worth doing.
- coldtea 8mo ago>In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys. Why not just add random "jitter" to the keystroke packets, but keeping just the 1 actual packet?
- varispeed 8mo agoJitter could be filtered out, I presume.
- fc417fc802 8mo agoHow? You can't average out the noise here because the attack involves discriminating the different types of events from one another based on the thing you'd be averaging.
- varispeed 8mo agoOne clue is that you cannot predict what key user is going to press next reliably, so the jitter would always be added to actual key press. You can minimise that by adding constant latency, so that you could simulate pulling events back in time, but still this is going to get complex quick and still could be filtered out. As for methods, it depends on the jitter. Think of things like noise removal in audio and adaptive filtering. Adding extra packets is much easier and more secure.
- fc417fc802 8mo agoOkay I think I see the issue (and slight misunderstanding). I believe the problem is actually latency. I was assuming the jitter interval would be noticeably larger than the gap between typical (say 95%) of key presses. Any smaller than that and you start to need cover traffic. Such an interval would still face correlation issues due to the varying nature of the overlap between the jitter intervals, however it seems like that should be trivial to address. That said, just throwing in some cover traffic is bound to be simpler. But a jitter interval long enough that keystroke packets can change order is going to be noticeable to a human typing quickly on what should be a solid connection - my WiFi is only at 3 to 6 ms RTT and I already notice that versus a wired connection. That doesn't sound so trivial to fix, and once again just throwing in some cover traffic completely solves the issue. So just do what's simple. My next question was going to be, why on the order of 100 extra packets instead of just 1 or 2? But of course an attacker could attempt to search some set of permutations for recognizable words. So either you drown everything out (simple) or you hook a multilingual dictionary up to a key stroke delay model for your cover traffic generator (complex). But really shouldn't this feature be implemented as some constant (low) background level of cover traffic that scales up as your typing frequency increases but caps out at some (still fairly low) rate? That seems both less likely to suffer from inadvertent leaks as well as not running afoul of the issue in the article.
- m000 8mo agoGenuinely asking: Wasn't the reason this happens kind of obvious in the first place?
- gafferongames 8mo agoJust wait until they discover head of line blocking
- canibal 8mo agoAm I missing something? This isn't what ssh's purpose is. Why should anyone care? We're talking about a game built to run over an encryption protocol? What are we even doing anymore? Also, correct me if I'm wrong, but client-side option existing is secure design, really feels like it shouldn't be circumvented server-side without giving the client the choice to do so or not by default. Don't lobby for watering down security for convenience, especially for trivially important objectives, please?
- teaearlgraycold 8mo agoYour security is safe. This game isn’t going to cause SSH to degrade. I assume this is done for novelty. There is also Terminal coffee which is a coffee company that takes orders for delivery over SSH.
- hackrmn 8mo agoWhy not "amortise" the period of sending keystrokes -- buffer them in a queue, and process the queue for sending these on a regular (and short enough for the human at the client end feeling the interactivity) interval, so there's no latency difference between sending an 'a' vs a 'q' and so on. If we assume some average typing speed on the bell curve, say, around 250 keystrokes per minute, the queue can be picked for sending every 250 milliseconds or so. That solution wouldn't require injecting extra packets on the network. What am I missing?
- esafak 8mo agoProduct behavior should be explainable without sleuthing. The well-named variables in the logs are serving this purpose.
- blabla_bla 8mo agoSSH now deliberately sends many dummy (“chaff”) packets per keystroke to obscure typing patterns and defeat keystroke-timing attacks. This privacy feature makes a single keypress look like heavy network activity, explaining why it can generate around 100 packets.
- GuB-42 8mo ago1980s: 1 packet per keystroke is too much, we must find a solution to bundle them together, for efficiency (see Nagle's algorithm, delayed ACK), also let's send everything in plaintext, including passwords 2020s: ha! with some advanced probabilistic models, we may be able to deduce something about what is being typed behind one of our layers of encryption, let's sent 100 packets per keystroke to mitigate that
- fsniper 8mo agoUnfortunate result of the security theater.. "Someone who has access to run privileged application can run side channel attacks! Let's drop cpu performance 20 percent over the world"
- ruszki 8mo agoAs I understood it’s enough to have “access to run privileged application” anywhere where the packet goes through. So, not necessarily at client or server sides. Or did I misunderstand?
- costco 8mo agoI think he's referring to CPU mitigations: https://en.wikipedia.org/wiki/Transient_execution_CPU_vulnerability https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner...
- peter_d_sherman 8mo ago>I am working on a high-performance game that runs over ssh. By 'ssh', you mean 'ssh' (library/program + protocol + encryption + decryption) on top of TCP/IP, on top of the Internet, right? OK, I'm not against it... but you do understand that there are all kinds of ways for that to slow things down, right? Your issues may (or may not!) include such things as: o Nagle's algorithm AKA buffering AKA packets not being sent until N bytes (where N > 1) ready to send, as other posters have suggested; o Slower encryption/decryption on older hardware (if users with older hardware is a target market, and if the added loss in speed makes an impact in gameplay, depending on the game, this may or may not be the case...) o The fact that TCP/IP (as opposed to UDP / Datagrams / "Raw" sockets) imposes a connection-oriented abstraction, requiring additional round trips of ACK ("I got the packet") RESEND ("I didn't get the packet") on top of the connectionless architecture that is the Internet (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...), which adds additonal latency, so, for example if a rural user in Australia experiences a 350ms delay for a raw packet to get to a U.S. server (or vice versa), then TCP/IP might make this 700ms or more, depending on the quality of the connection! o The speed of the game limited to both the bandwidth and latency of the slowest user (if a multi-player game, and if the game must not update until that user "moves"... again, game architecture will determine this, and it wouldn't be applicable to all games...) Now, you could use UDP, as other posters have suggested, but then you must manually manage connections and encryption... That may be the right choice for some types programmers, some types of games/applications -- but equal-and-oppositely it may be the wrong choice for others... Anyway, wishing you well with your game development! I haven't used SSH (at least, not in a debug capacity), so I'm not sure what SSH debugging options exist -- but it would be nice if SSH had a full logging debug mode, which would explain exactly WHY it chose to send any given packet that it did along with related helpful information, such as latency/time/other metrics, etc., if it doesn't have this/these feature(s) already...
- TruePath 8mo agoWhat is the usecase for using ssh at all where you don't need to be resistant against timing analysis? Either it's not sensitive and you can use telnet (if necessary after using ssh to authenticate) or the game (or other stuff on the connection) might be sensitive and you need traffic analysis resistance. If you get clever and write a client to ensure sensitive data like passwords or email are sent in a burst you could just use an encryption library just for that data instead.
- halJordan 8mo agoDont let this article be blinders to you. Ssh does much more than obfuscate keypress timings. Not needing the chaff means turn it off and keep all the other benefits. It doesn't mean "revert to telnet"
- dent9 8mo ago> When you say “LLMs did not fully solve this problem” some people tend to respond with “you’re holding it wrong!” > > I think they’re sometimes right! Interacting with LLMs is a new skill, and it feels pretty weird if you’re used to writing software like it’s 2020. A more talented user of LLMs may have trivially solved this problem. So one thing I only recently figured out is that using ChatGPT via the web browser chat is massively different from using OpenAI's code-focused Codex model / interface. Once I switched to using Codex (via the VS Code extension + my own ChatGPT subscription) the quality of answers I got improved massively. So if you're trying to use LLM to help with debug, make sure you're using the right model!! There are apparently massive differences between models of the same generation from the same company
- Aachen 8mo agoTL;DR it is keystroke timing obfuscation (of course)
- deleted 8mo ago[deleted]