25 ms·

I think Catalina 10.15.4 broke SSH

733 points by chmaynard 7y ago

supernintendo 7y ago
I've moved to Linux for 99% of my computing but still use macOS for some audio production work. Catalina is unusable for me personally (most of the software I need just silently crashes) so I disabled the upgrade prompt: sudo softwareupdate --ignore "macOS Catalina" defaults write com.apple.systempreferences AttentionPrefBundleIDs 0 killall Dock Apple should really slow down on major releases of macOS or stop altogether in my opinion. macOS Mojave is a great OS and it's basically feature complete. Just stick with that, introduce bug fixes and security patches as needed and I think people will be happy.
_-___________-_ 7y ago
If you have Homebrew or something similar, I recommend installing openssh through there -- you get a newer version to boot.
- rovr138 7y ago
  I always do this. I'm not sure what's the current state, but there are features on SSH I wasn't able to use due to the version provided being old. I know that `Include` on `config` is/was one. Include "some/path" This is something I use frequently that wasn't available on previous built in versions.
  - oefrha 7y ago
    > I'm not sure what's the current state, but there are features on SSH I wasn't able to use due to the version provided being old. > I know that `Include` on `config` is/was one. That's both terribly out of date info and hardly ever true as far as I can tell. The Include directive was a new feature of OpenSSH 7.3, released on 2016-08-01.[1] Apple shipped OpenSSH 7.3 in macOS 10.12.2[2][3], released on 2016-12-13. That's a very reasonable four months gap. I only use the system ssh because stock OpenSSH didn't integrate well with system keychain many years ago (not sure about the current state). But I've been using the Include directive for a long time. [1] https://www.openssh.com/txt/release-7.3 https://www.openssh.com/txt/release-7.3 [2] https://opensource.apple.com/release/macos-10122.html https://opensource.apple.com/release/macos-10122.html [3] https://opensource.apple.com/source/OpenSSH/OpenSSH-209.30.4/openssh/version.h.auto.html https://opensource.apple.com/source/OpenSSH/OpenSSH-209.30.4...
- 2ion 7y ago
  It's usually not a full replacement. SSH for macOS has some integration built in that current OpenSSH does not have, like Keychain integration.
  - skoskie 7y ago
    Really? I have my ~/.ssh/config file set to “AddToKeychain” on all entries and it doesn’t seem to be a problem.
  - lilyball 7y ago
    I thought Homebrew patched OpensSSH using Apple's keychain patch, but looking at the formula right now I see # Please don't resubmit the keychain patch option. It will never be accepted. # https://github.com/Homebrew/homebrew-dupes/pull/482#issuecomment-118994372 Sadly the homebrew-dupes repo seems to have been deleted so this comment can't be read anymore.
    - saagarjha 7y ago
      > Sadly the homebrew-dupes repo seems to have been deleted so this comment can't be read anymore. They're kinda bad at that in general :/
    - CKN23-ARIN 7y ago
      Archive of the discussion: https://archive.is/hSB6d https://archive.is/hSB6d > We are uncomfortable continually supporting a 1900+ line patch which upstream hasn't signed off on that has the potential to both compromise OpenSSH security and Keychain security. From 10.11 it will also be impossible to edit plists in /System/* without disabling rootless, which isn't a configuration we'll be intentionally supporting.
  - _-___________-_ 7y ago
    I prefer ssh-agent anyway, but yeah, I think they did remove the keychain integration patch.
- Doctor_Fegg 7y ago
  Oh god no. Homebrew managing openssh has been the cause of more command-line instability and forced reinstalls than anything else I’ve encountered in the last few years of OS X (sorry, macOS). I’ve started installing stuff from source again just to prevent a cascade of Homebrew upgrades breaking everything.
  - _-___________-_ 7y ago
    Never experienced this in a decade or so of using Homebrew's OpenSSH, but you can absolutely use something other than Homebrew to get a more up-to-date and standard OpenSSH install if you prefer.
  - lloeki 7y ago
    > I’ve started installing stuff from source again just to prevent a cascade of Homebrew upgrades breaking everything. Since you crossed that line, do yourself a favour and check out nixpkg.
  - theonemind 7y ago
    I sometimes use NetBSD's pkgsrc on macOS because it installs super cleanly in any prefix you like and never, ever breaks the system. It doesn't have everything, and you will occasionally encounter a package that won't build, but it doesn't even dream of taking over /usr/local or disrupting your system. You could install it into your home directory if you wanted to (which I have done, on systems where I don't have root or enough ownership to just throw things anywhere)
  - rswail 7y ago
    Why don't people use MacPorts instead? I've never had any problems with it. Homebrew wants to screw around in /usr, Macports installs itself in /opt and doesn't interfere with things in the MacOS world. Set your PATH to have /opt/local/{bin,sbin} and everything Just Works.
    - saila 7y ago
      What is the practical difference between /usr/local and /opt or /opt/local? I don't think macOS puts anything in /usr/local.
deleted 7y ago
[deleted]
awinder 7y ago
“I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina.” Welp.
- draw_down 7y ago
  Which is a weird sentiment. HN is, let's say, not exactly shy about criticizing Apple hardware/software, especially software.
- hoistbypetard 7y ago
  Which is more common? Someone says "I don't want to end up on ___ news again" and they really mean they don't? Or they say that and they really want to? It feels like the old "Please, Br'er Fox, don't upvote this post. I don't want anyone to see it." (I'm about 80% kidding but amused. Lockdown.)
  - tylerhall 7y ago
    Sadly, I mean it. Don't want to go through this again. Taking down the post for now.
    - azinman2 7y ago
      Go through what? Why would you take down the post as it’s probably useful to others?
      - untog 7y ago
        Hacker News can be a cesspool at times. One important thing to always remember is that unless someone posted their article to Hacker News themselves they might have had absolutely no expectation that a huge audience was about to descend and dissect everything they wrote. They might have just been talking off the cuff, mentally noodling around or even just using the process of writing stuff down as a means to sort their thoughts. Far too often HN commenters work from the assumption that an author is intending to make A Big Point and very uncharitably deconstruct every sentence the author wrote. It's only a matter of time before we see a reply along the lines of "OBVIOUSLY 10.15.4 did NOT break SSH, the author just didn't do X Y and Z to fix a very OBVIOUS mistake in their SSH config". > Why would you take down the post as it’s probably useful to others? More broadly, Tyler doesn't owe the world anything in this regard. If he wants to post it, cool. If he wants to remove it, cool.
floatingatoll 7y ago
To summarize OP: Their 10.15.4 macOS built-in ssh terminal command is unable to reach hostnames when a port number higher than 8192 is used. EDIT: Comments differ; one indicates issues SSH'ing to lower than 8192 ports, another indicates no issues SSH'ing to higher than 8192 ports.
- weej 7y ago
  And Tyler "end(ed) up on Hacker News again bitching about Catalina"
  - saagarjha 7y ago
    That’s not completely his fault, is it?
- Apofis 7y ago
  chaos:1111
derefr 7y ago
Is macOS /user/bin/ssh just upstream OpenSSH, or does Apple maintain a fork? If no fork, this would be an upstream OpenSSH problem, no?
- saagarjha 7y ago
  https://opensource.apple.com/source/OpenSSH/OpenSSH-235/ https://opensource.apple.com/source/OpenSSH/OpenSSH-235/
  - paxswill 7y ago
    Looks like that's for macOS 10.15.0 as version.h for that OpenSSH release is for 7.9p1 (macOS 10.15.4 has OpenSSH 8.1p1).
    - saagarjha 7y ago
      It is, I pulled it from https://opensource.apple.com/release/macos-1015.html https://opensource.apple.com/release/macos-1015.html.
- paxswill 7y ago
  Apple includes a customized version of OpenSSH. From what I recall from the last time I looked at it, the changes were mostly integrating the key retrieval mechanisms with the rest of macOS. For example, Apple's ssh-add can store key passphrase in Keychain with the -K option, and then later access those passphrase with the -A flag.
  - floatingatoll 7y ago
    Those stored key passphrases are visible with the Keychain Access application, Kind: "application password", name: "SSH: /full/path/to/key", in the login & iCloud keychains.
  - skoskie 7y ago
    If using the upstream version there is one line to add to a startup script or to your zshrc (et. al.) file ... ssh-add -A > /dev/null ... and one default value to place in your ssh config file... AddToKeychain Yes ... to get around this issue. It works fine after that. (On mobile. Sorry for formatting)
dimtion 7y ago
I'm surprised nobody noticed this bug at Apple before the release. Is there nobody there that connects by hostname to a ssh server with a port > 8192?
- deleted 7y ago
  [deleted]
- saagarjha 7y ago
  Probably not. I doubt I’ve used a port that high.
  - Yetanfou 7y ago
    Why not? I regularly use 5-digit ports - 3-digit prefix plus 2-digit 'official' port - for various systems which reside behind a NATting router.
    - the_mitsuhiko 7y ago
      I'm pretty sure it's generally not advisable to use ports this high since they are used for other purposes.
      - Avamander 7y ago
        All purposes are equally as valid on unallocated port ranges. Inconfigurable port ranges with no fallback when those ports are in use is bad design.
        user5994461 7y ago
        High port numbers, above 30000 usually, are ephemeral ports and get pre-empted by the system. They're not safe to listen to for server applications.
    - syncsynchalt 7y ago
      Ports <1024 require root access to bind, so on a multi-user system it would be insecure to run ssh on such a high port. (Granted, multi-user hosts are very rare nowadays).
AdmiralAsshat 7y ago
Mental note for future blog post: the secret to getting onto the front page of HN is to publicly state that I do not want this blog post getting onto the front page of HN.
- throwanem 7y ago
  I feel like the subject matter also signifies. I mean, if I post a 5000-word paean to the Polistinae - they're good wasps, Brent - and say I don't want it to end up on the front page of HN, I feel like I'm not going to get what I don't want, you know?
hazebooth 7y ago
I'm on mac OS 10.15.4. SSH has worked on all betas and the official release from Apple.
- beervirus 7y ago
  Using a port above 8192 and connecting by hostname?
  - hazebooth 7y ago
    Yes, but maybe I've mucked around with my ssh too much.
- deleted 7y ago
  [deleted]
sgehly 7y ago
>>I don’t want to end up on Hacker News again bitching about Catalina. Hahaha
ProAm 7y ago
"It just works" -- Is Apple too large now? Is this a QA problem, product team problem? Management? Catalina is still stumbling and Im surprised to be honest after the past 4 years.
- cflewis 7y ago
  My feeling is that Apple beancounters have decided macOS is mostly a gateway to Xcode for iOS development, anything else is just to help sell laptops. The stuff in "anything else" doesn't need to actually work well, just exist so it can be something on the features list.
  - api 7y ago
    It does help that there is no overall competitor to MacBooks in terms of ease of use or (now that the butterfly keyboard is dead) build quality. There are decent build PC laptops but you have to run Windows or Linux on them. Windows is a dumpster fire these days with ads in the start menu, the use of "dark patterns" to herd people into MS cloud, and out of control unnecessary telemetry. Linux is fine only if you have a lot of time on your hands to troubleshoot edge case issues and hunt for drivers. Linux also still (through no fault of its own) can't run a lot of apps that many people need.
    - aftbit 7y ago
      Linux is the only option IMO, but I have a very high yakshaving tolerance. That said, if you run a recent Ubuntu, most stuff "just works" as long as you don't need Photoshop or the Office suite.
      - deleted 7y ago
        [deleted]
      - jacobsenscott 7y ago
        Or 4k monitors, or screen sharing when running more than one monitor. That's the reason I haven't switched from macos back to linux (I was all in on linux until about 5 years ago when I started to care about display quality and working remotely).
AsyncAwait 7y ago
Another post to my collection of how macOS works flawlessly and Linux breaks stuff when it comes around again.
- rimliu 7y ago
  I think the technical term for this is "cherry-picking".
  - fanatic2pope 7y ago
    Exactly. Just like people do with Linux issues.
    - chipotle_coyote 7y ago
      As a long-time Mac user who gets irritated about something trivial within minutes of using any Linux desktop environment, I'd have to say this is pretty spot on. :)
  - Yetanfou 7y ago
    If this tree only carried a few cherries you'd be right. By now it is such an abundant source of fruit that they'd do well by changing their logo to just this, a Catalina Cherry [1,2]. [1] https://en.wikipedia.org/wiki/Prunus_ilicifolia https://en.wikipedia.org/wiki/Prunus_ilicifolia [2] https://calscape.org/Prunus-ilicifolia-ssp.-lyonii-(Catalina-Cherry)?srchcr=sc5849e2614df95 https://calscape.org/Prunus-ilicifolia-ssp.-lyonii-(Catalina...
germs12 7y ago
Original Post: I was completely at my wit’s end and feeling like I had lost my mind until about a half hour ago. Let me start from the beginning… I don’t have an exact date, but within the last week I realized that I was unable to ssh into my primary web server – the one that runs my business website, activation server, etc. It’s sort of the linchpin for my tiny software company. When it goes down, I get worried. At first I thought maybe the server was down? I hadn’t received any alerts, so I did a quick check. And, yes, it was still up and running and serving web traffic. Ok, did sshd somehow become unresponsive? I login through the Linode control panel and restart the service. Still can’t login. It’s odd. I don’t get a connection refused. Not even a timeout. It just…hangs. That’s the ssh output with the verbose flag. Nothing. I waited 10+ minutes and it never timed out or produced any other output. I reboot the server itself and the problem persists. Then, I notice some more oddities. I’m able to connect using ForkLift – my FTP client, which connects via SFTP. Also, SequelPro is able to connect to MySQL via ssh as well. And then things get even stranger. This is all happening on my iMac. I try connecting from my laptop, and it works. My MacBook Pro is at home right next to my iMac, which is refusing to login. They’re both on the same wifi and thus the same IP. So, it can’t be that my home IP address got mistakenly banned somehow. Next, I ssh into a different server and then hop to the problematic one. It connects without any trouble. At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup. Still can’t login. Ok. I think about it for a few minutes and then – aha! – I have an Ubuntu virtual machine running on this iMac inside Parallels. I’ll ssh into it and then try and connect. That will rule out if there’s just something odd about my iMac’s LAN IP. (To be clear, my home network is perfectly ordinary. Just a cable modem and a router.) So, I login to the VM, try and connect, and it works fine. At this point here’s what I’ve found: My iMac is the only machine that cannot login. I’ve connected successfully from behind the same public IP using a laptop, a virtual machine, and my iPhone and iPad. I’ve verified my ssh keys are correct and have the appropriate permissions. I can connect to other servers from the problematic machine – both at the same hosting provider (Linode) and others (AWS and DigitalOcean). I can connect from my iMac if I jump through any other server, first. I start trying to think what could possibly be different about this one machine. And then it dawns on me. This all started around the time I updated my iMac to 10.15.4. My laptop is still on 10.15.3 – and, of course, the virtual machine isn’t macOS at all. Totally grasping for straws I google for “10.15.4 ssh” and find this top result on the Apple discussion forums: Catalina 10.15.4 SSH port > 8192 does not work when using server name instead of IP This issue started just after upgrading to macOS Catalina 10.15.4. After that update I am no longer able to open a SSH connection to a port greater than 8192 using server name (instead of IP). Yes, I do change the port on the server side prior to every test. That can’t possibly be real? Up until this point, I was connecting via a saved hostname defined in my ~/.ssh/config, which let me login simply by tying ssh some-server. So, I tried ssh ip-address -p9944 and it worked! (That server runs on an alternate ssh port.) Ok. Time to narrow this down a bit further. I changed the server to listen on standard port 22 and tried connecting via the hostname once again. Holy crap, it worked. The user in the Apple forums was right. At least in my case, my one server that happened to be running on a non-standard ssh port above 8192 will not connect from Catalina 10.15.4 when using the hostname instead of the IP address. Just to verify, I boot up a Mojave and Catalina (10.15.3) VM on the same iMac. They both connect fine, while the host machine continues to fail. The internals of this is all so incredibly above my head I have no idea what the underlying problem might be. Am I and this one other forum poster just doing something totally bizarre yet the same? This ssh setup has been working for years for me until just the last week. I would love to be proven wrong and told I’m an idiot. But I don’t know what difference connecting via the hostname versus the IP address would make when specifically using a non-standard port above a certain threshold. It just….sigh. I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer.
slovette 7y ago
Catalina is broken in many ways. This complain and Remote Access in (so I can SSH to my $4k MacBook) disables itself anytime the computer is restarted. But more importantly, I’ve still not found a Thunderbolt Display that doesn’t routinely crash screen manager services upon idle user activity. 3 x $300 thunderbolt3 dock solutions later and not a one hasn’t crashed this computer. All main brands, two of which sell accessories in the Apple store. Problem also existed with a top of the line 13” MacBook Pro. I’ve just gotten used to the shoddy-ness that is Catalina. Figure if I go to the bathroom, upon return I have a fresh, new clean desktop environment. Feature not a bug. Yay!
- unicornmama 7y ago
  I completely disabled everything to do with sleep. “Solved” the problem for me.
  - anonexpat 7y ago
    The first thing I install on a fresh Mac is amphetamine.
    - SamuelAdams 7y ago
      Just curious, how is this different from Caffeine? http://lightheadsw.com/caffeine/ http://lightheadsw.com/caffeine/
      - blacksmith_tb 7y ago
        Or just running caffeinate -t 10000?
      - wyclif 7y ago
        One difference is that Amphetamine isn't updated as well, also seems more hackish. I switched to Caffeine a while ago.
      - oefrha 7y ago
        It has more features. Check https://apps.apple.com/us/app/amphetamine/id937984704?mt=12 https://apps.apple.com/us/app/amphetamine/id937984704?mt=12 for details. The one you linked to looks like a simple wrapper of caffeinate(8).
nathell 7y ago
I've been a Linux user for the last ~15 years, and now I need to do some iOS development so just a few days ago I've ordered a Mac Mini. I guess I'm in for a bumpy ride. Oh well.
- redsymbol 7y ago
  I used linux on all my laptops/work machines for about 7 years, then switched to macbook pros 5 years ago. Definitely some things you have to adapt to (I still miss focus-follows-mouse). But for the most part, you'll find it's a smooth ride. When bumps like this happen, they tend to push out a fix quickly - especially when it gets traction on HN like this.
  - john_alan 7y ago
    macOS is POSIX UNIX. it definitely has focus follows mouse. You just mean that hover over a lower window allows scrolling right? MacOS has that.
    - redsymbol 7y ago
      Not just scrolling. https://en.wikipedia.org/wiki/Focus_(computing)#Focus_follows_pointer https://en.wikipedia.org/wiki/Focus_(computing)#Focus_follow...
  - quesera 7y ago
    For me, focus-follows-mouse is most useful for terminal windows. It is a feature you can enable in iTerm2. This doesn't help across applications of course, and there's a reasonable argument that the inconsistency is worse than the absence -- but for me, iTerm2's FFM feature helps.
    - redsymbol 7y ago
      Didn't know iTerm2 did that. Thanks, I'll check it out - might help a lot.
  - 7y ago
0x0 7y ago
I can't reproduce this. macOS 10.15.4, ssh'ing to a very high (5digit) port with a hostname no problems.
- jpwgarrison 7y ago
  Same here, 5 digits and a hostname = no problem. There must be some other factor(s) in play.
- 0xff00ffee 7y ago
  Ditto. There's a post a few levels above where they are digging into the source of ssh, I'm following that!
- ajphdiv 7y ago
  I can't either, in fact all 10+ of the hosts that I routinely access have ports higher than OPs issue.
  - andai 7y ago
    Offtopic but why are people using high port numbers? Additional security due to a nonstandard port? If so, does that go together with anything additional like port knocking? Or is it multiple hosts on the same IP, but different ports?
    - cozzyd 7y ago
      A relatively common use case is multiple devices behind a NAT, where each port goes to a different device.
    - deleted 7y ago
      [deleted]
    - vbezhenar 7y ago
      Some people think that it adds to security. Some people want to reduce noise in logs.
    - 7y ago
judge2020 7y ago
In case it's slow for you too: <removed since they wanted it taken down>
- saagarjha 7y ago
  It’s been taken down on purpose: https://news.ycombinator.com/item?id=22738841 https://news.ycombinator.com/item?id=22738841
Zelphyr 7y ago
As an aside; is there a reason to host SSH on a non-standard port? I recently came across a system that had it listening to a really high port number. I dismissed it as security through (bad) obscurity but is there a valid security reason to do this? EDIT: Thanks to everyone who answered my question! It makes sense to me now why one might do this.
- 0x0 7y ago
  It reduces the amount of endless bruteforce attempts somewhat, so log files are slightly more readable. Although recently this seems to be significantly less effective compared to the previous 20 years...
- joombaga 7y ago
  If it's exposed to the internet, definitely. Port 22 gets hammered.
  - wazoox 7y ago
    Indeed. In fact it's hammered so bad that it made my home machine crawl at times. By changing ports it went from several thousands attempts per hour to a few per day.
- _-___________-_ 7y ago
  The author didn't say it had anything to do with security. I have one or two SSH daemons out there listening on non-standard ports because of stupid limitations of middleboxes I'm forced to use, or because they're port-forwarded through something that already listens on 22 itself. Security-wise, it seems pointless; my daemons on random non-standard ports still get hammered, and fail2ban takes care of keeping the log spam down just as easily as it does the ones on 22.
- deleted 7y ago
  [deleted]
- spzb 7y ago
  It massively reduces the number of script kiddie attacks. It's not hard to find SSH on a non-standard port but most SKs don't know or don't bother. Other possible reason is NAT. If you've got several machines or VMs but only one public IP you can port forward different public ports to port 22 on different machines. Not the only solution by a long way but a relatively straightforward one.
vgene 7y ago
I had the same problem on a MacBook after upgrading to 10.15.4. However, I wasn't using a port number higher than 8192, the socket was 75 with a hostname. The problem was solved when I replaced the hostname with its IP or plugged in an Ethernet Cable. I tried to restart mDNSResponder and flush the dns cache and switch to a different DNS server. Nothing works so far.
deeblering4 7y ago
> I changed the server to listen on standard port 22 and tried connecting via the hostname once again. Holy crap, it worked. Shocking! Granted high ports shouldn't be broken, but running SSH on a non-standard port is security (read obscurity) theater at best. There's really not much benefit, unless you need multiple sshds on the same IP, but at that point I'd question the sanity of the approach.
- notyourday 7y ago
  For me SSH on a standard port get connection attempts and every few minutes. SSH on a non-standard port get connections attempts every few weeks.
- lukevp 7y ago
  Security through obscurity shouldn’t be used as an edict that something is not effective. You are talking about the fact that it doesn’t increase the security of the protocol itself or the passphrases/keys used. This is true. However, there are tons of bots out there that scan 22 and try to exploit common logins. There are presumably quite a few less that are port scanning every machine for every possible high port and attempting to handshake ssh and then try logins. Do you disagree with that? If not, this is not security through obscurity, it has a very real impact on the volume of bots that have knowledge that this service is running and are actively exploiting it. It’s just a different type of security, it’s discoverability of the service. Here’s another example. Say you have a web server running that is only for internal employee use. But you want to expose it externally so that they can reach it without a VPN. Even if you follow proper security protocols, why would you not turn off search engine indexing on this page, and limit the pages that link to it? It will not increase the inherent security of the protocol or the user accounts, but it will drastically lower the # of bots using up CPU and iptables entries trying to fail2ban or blacklist them. Security is a spectrum and you want to have defense in depth. Moving ssh to a nonstandard port is a security best practice and you shouldn’t be advising people not to use it. But should they also have good key setting, fail2ban, ip whitelisting/blacklisting, etc? Of course they should.
  - deeblering4 7y ago
makecheck 7y ago
So, some quick debugging here... In his screenshot the bad login hangs at "Connecting to clickontyler.com port" (noting that no port number appears and no period at the end). While I can’t be sure exactly which "ssh" patch Apple may have, this seems to be the relevant file and logging code (starting at line 448): https://github.com/openssh/openssh-portable/blob/master/sshconnect.c#L488 https://github.com/openssh/openssh-portable/blob/master/sshc... In that code, the only thing that can set the "strport" value that is used in the log is a call to getnameinfo(). If that string is corrupted in any way, e.g. not terminated or perhaps has invisible characters that trigger bad terminal behavior (such as invisibility), the act of logging it might produce the apparent hang seen here. Again, a guess but it is possible that getnameinfo() is not necessarily processing the record correctly (for whatever reason). One such example is in the "getnameinfo" man page at the end, under CAVEATS, where they show an example of not simply trusting the result of the first call.
- 0x0 7y ago
  Maybe there is something funny in /etc/services on this machine that throws the call into an infinite loop? Perhaps near the bottom beyond port 8192?
- tylerhall 7y ago
  Good sleuthing, but the missing port number is simpler than that. I just blacked it out of the screenshot. I know very well that running sshd on a non-standard port has no benefits security-wise, but it does lessen the length of my log files from dumb script kiddies. I redacted the port in the screenshot for that reason.
  - jonny_eh 7y ago
    You should mention that in the caption, or use a non-black colour as a mask.
  - 0x0 7y ago
    Have you tried running ssh in lldb/gdb and dumping a stacktrace when it hangs? Might have to copy the ssh binary to a temp dir to avoid SIP denying ptrace.
StreamBright 7y ago
Thank god I did not upgrade. Software upgrades are the best way to waste time.
- rovr138 7y ago
  They're the best way to stay secured and receive new features. The issue with them is lack of testing before deploying them.
  - StreamBright 7y ago
    You can just get the security patches for Mojave. You are trying to make it sound like of you are on previous version there are no security patches. Factually untrue.
  - anonymou2 7y ago
    I think proprietary software and staying secure are contradictory ideas.
tambourine_man 7y ago
>I don’t want to end up on Hacker News again bitching about Catalina.
- davidkuhta 7y ago
  To be fair, full context tempers that sentence: > I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer.
deleted 7y ago
[deleted]
chmaynard 7y ago
Apple's stance is that it didn't happen unless someone reports it using Radar (internal) or bugreport.apple.com (external). Unfortunately, they don't believe in Linus's Law, which states that "given enough eyeballs, all bugs are shallow."
- saagarjha 7y ago
  Things that end up on the front page of Hacker News get fixed.
oefrha 7y ago
One more data point: just tried to repro with hostname and port 8193 and failed, so the issue is probably more intricate than described. (Guest in my test: OpenSSH 7.6p1 on ubuntu bionic, stock config other than sshd port.)
- zimpenfish 7y ago
  Tried with dropbear and sshd on Arch, port 9022 from 10.15.4 Beta (19E258a) - no issues with a variety of hostnames that end up at the same host.
  - zimpenfish 7y ago
    Just updated to 10.15.5 Beta (19F53f) and still no issues with dropbear or sshd.
colechristensen 7y ago
I experienced a similar issue with a git repository hosted on a high port, `brew install openssh` fixed it even though the homebrew `ssh` was not first on my $PATH. Didn't bother to investigate further.
- moonchild 7y ago
  Possibly that overwrote the config file for the system ssh?
  - colechristensen 7y ago
    I was thinking homebrew's git perhaps had a different $PATH (or was using shared objects?) that used the different openssh. Just guessing, didn't seem worth my effort at the time.
- saagarjha 7y ago
  Isn't /usr/local/bin first on $PATH?
  - KarlKemp 7y ago
    Homebrew tends not to install to bust/local/bin when that would replace a native version.
jki275 7y ago
I'm using ssh to a named server online using a port >8192 from a 10.15.4 machine right now. Have had no issues with it at all.
m_a_g 7y ago
>I don’t want to end up on Hacker News again bitching about Catalina. Ends up on Hacker News again bitching about Catalina.
- jbverschoor 7y ago
  I don't wanna pay taxes. Paying taxes anyway
kulix425 7y ago
"I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina." lol
krzysztofeng123 7y ago
laughs in Linux
- dang 7y ago
  Please don't do this here.
zeveb 7y ago
> Am I and this one other forum poster just doing something totally bizarre yet the same? One might uncharitably suggest that using macOS and expecting standard decades-old Unix behaviour is itself bizarre … but that's also true of using Linux with systemd (viz., nohup no longer nohups, or systemd-resolved, or innumerable other broken bits). It's almost as though no-one cares about quality anymore.
acdha 7y ago
I suspect this is due to a feature being enabled for canonicalization and that the key part is the presence of the colon rather than the port number. On a 10.15.4 system, I see a line in the debug output which is not present in the screenshot: > debug1: resolve_canonicalize: hostname example.org:7999 is an unrecognised address If instead I use `-p` or a config-file option, everything works as expected.
- lilyball 7y ago
  hostname:port is not a valid destination according to ssh syntax. A destination may either be [user@]hostname or a URI of the form ssh://[user@]hostname[:port].
- richardkmichael 7y ago
  The Apple forum article cites by the OP indicates it's a problem with `-p <PORT>` as well. https://discussions.apple.com/thread/251226509 https://discussions.apple.com/thread/251226509
- jki275 7y ago
  That's a fair point. I use either a ssh config file with all the correct options in it, or the -p option if I'm doing it without a config file, perhaps that's why I've never had a problem.
yasp 7y ago
Catalina also broke `apropos`. https://apple.stackexchange.com/questions/374025/errors-from-whatis-command-unable-to-rebuild-database-with-makewhatis https://apple.stackexchange.com/questions/374025/errors-from...
geuis 7y ago
Looks like the post was just deleted.
deleted 7y ago
[deleted]
viburnum 7y ago
If I'm still on Sierra, which version should I upgrade to?
- v64 7y ago
  I have a MacBook Pro (Retina, 13-inch, Mid 2014) and Mojave runs very well on it. I have no plans to upgrade to Catalina.
di3goleite 7y ago
I faced the same problem two weeks ago with the previous version of Catalina (I don't remember the correct number but was a previous on 10.15.4) and git (I use SSH to authenticate with the server). So I did a report to Bitbucket with a solution that worked for me after investigate more about the problem: https://twitter.com/di3goleite/status/1239596891471581189?s=20 https://twitter.com/di3goleite/status/1239596891471581189?s=... Thank you about that clarification. Also your website seems to be down actually.
JdeBP 7y ago
Here's an actual bug report: * https://openradar.appspot.com/radar?id=4931259776106496 https://openradar.appspot.com/radar?id=4931259776106496 From that and the discussions.apple.com. post, hyperlinked elsewhere in this discussion, it appears that the >8192 condition varies according to what the hostname actually is. The bug report is datelined 2020-04-26, interestingly. There might be a bug in the bug reporting system. (-:
- saagarjha 7y ago
  OpenRadar is community-maintained, and rather poorly at that these days.
- oefrha 7y ago
  > The bug report is datelined 2020-04-26, interestingly. There might be a bug in the bug reporting system. No, you can type whatever date you want. The "add a new radar" screen is just a bunch of text input boxes: https://i.imgur.com/nNf457J.png https://i.imgur.com/nNf457J.png
  - JdeBP 7y ago
    The ability to type whatever date one wants is often considered to be a bug. The ability to post-date reports a month into the future sometimes is, too. (-:
    - oefrha 7y ago
      You can not only type whatever date, but also whatever non-date. The site just assumes good intentions and is working as designed.
neonate 7y ago
https://archive.md/lvCKA https://archive.md/lvCKA
- deleted 7y ago
  [deleted]
- ncmncm 7y ago
  Anyway the parent item seems to be something about a specific, named release of Apple's proprietary MacOS operating system. It seems we are being warned that if we upgrade to it, something will not work anymore. Those of us not inclined to install Catalina may sigh in relief; our instances of SSH have not been broken.
  - new_realist 7y ago
    99.999% of all of us got that just from the title, thanks. Congrats on puzzling it out.
    - sudoaza 7y ago
      I didn't and funny you'd think Mac userbase is that big
      - albedoa 7y ago
        Mac users are not the only group who know what those words mean.
        segfaultbuserr 7y ago
        Yeah, I don't use macOS and I don't own any Apple computer. But a weird California location name + version number 10 is a dead giveaway.
        sudoaza 7y ago
        didn't know Catalina was a California location either, to me is just a female name. I actually though she might be a contributor to SSH project.
buildbot 7y ago
Wow, just the other day this started happening to me as well with one of my serversfrom my MacBook. It used to work fine, but now only that laptop can’t connect to it. iIt’s on a high port too.
stock_toaster 7y ago
Maybe a weird ControlMaster/ControlPath config? I have had issues with the ControlPath result being too long with certain hostname/port combinations in the past -- which resulted in ssh to ip working but ssh to hostname not working. As a result, I haven since started using %C instead of %l%h%p%r in my ControlPath config.
anongraddebt 7y ago
"I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina." +1
tkubacki 7y ago
It is time to move to Linux or Windows desktop. Really if you are not hostage of Apple ecosystem then decent desktop is much more reliable in my experience (got old MacBook which is ok too but eg can't connect to old vpn on it)
stinos 7y ago
At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup Is that a common thing to do, or any reason why the OP would do that? Doesn't ssh reject your key, saying it does that if there's such a problem? And even if not wouldn't it be advisable to at least look at the permissions; I mean suppose they're not -rw------- or so, wouldn't you want to know that, and also why they are not ok?
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
KiDD 7y ago
"I don’t want to end up on Hacker News again bitching about Catalina." Pretty sure that guarantees getting to the front page :D
- phlakaton 7y ago
  This is truly the darkest timeline for that poor blogger. :-P
- spaniard_dev 7y ago
  Those are the magic words. F*ck SEO, add that and you'll get a couple million page views.
liquidify 7y ago
Are you talking about the new apple feature?... non working ssh? It's all the rage.
LilBytes 7y ago
This is an anecdote but the latest update forced me to rebuild my Mac from a hard/factory reset. My Dell D6000 on my 2019 MacBook Pro no longer charges the laptop. I've tried. My Mac's resources were getting gobbled up by an internal process I coudln't terminate and my keychain was borked and I couldn't log in after a reset (to try and get around the resource hogging). Recovery didn't get me any where so I used Recovery over the Internet to do a clean install. I'm running 10.15.4, no issues as of yet. And this all occurred after the security update. I'm running on the version prior for now but will make sure I've got a good backup and give it another go.
- gangstead 7y ago
  I had a d6000 and had to update the firmware (with a borrowed windows machine) before it would charge my mbp.
  - LilBytes 7y ago
    Hadn't considered that, I'll give it a go. Thanks!
    - LilBytes 7y ago
      If anyone comes across this comment, this 100% worked.
ThePowerOfFuet 7y ago
> Next, I ssh into a different server and then hop to the problematic one. It connects without any trouble. At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup. Still can’t login. Someone should have paid more attention to that verbose SSH output first.
teilo 7y ago
Well, the description of this bug is not generally reproducible, so whatever is causing it, it's not as simple as using a high port with a server name. I tested this specifically on a number of servers that I run with port numbers > 10000, using /usr/bin/ssh on macOS 10.15.4, with and without IP addresses. Nothing broke for me.
ulkesh 7y ago
I agree, that’s an annoying bug/feature. However, there is an amazingly easy workaround, assuming the IP and port don’t change often: create a ~/bin shell script that connects via IP and port, make it executable, and add ~/bin to PATH. This workaround doesn’t excuse Apple of doing something so egregiously stupid, but it’s so easy that you may as well do it and move on.
ipv6ipv4 7y ago
In my experience the breakage is with IPv6. Try forcing IPv4. 'ssh -4 <hostname>'
epiphanitus 7y ago
What’s the backstory to why apples bash has to be different than gnu bash? I love having the Linux kernel with a nice UI but there are some useful commands that are missing. There are ways to get them set up but in any case it’s kind of a pain
Elrac 7y ago
Am I the only one who thought this article was about Apache Tomcat?
steve1977 7y ago
"I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer." Ok. And did you actually report it as a bug to Apple?
- wyattpeak 7y ago
  Whether or not he did, there's absolutely nothing wrong with posting about a problem in case others come across the same. In fact it's damned helpful.
neilwilson 7y ago
Catalina seems to have bust Wifi monitor mode on tcpdump on my MacMini 2018, yet it works fine on my Mac Air. Still not sure if that is my machine, or a general fault - but the lack of monitor and promiscuous mode is playing havoc with IPv6 multicast packets from VMware Fusion VMs.
vgaldikas 7y ago
>I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. Whoops

733 points by chmaynard 7y ago

I've moved to Linux for 99% of my computing but still use macOS for some audio production work. Catalina is unusable for me personally (most of the software I need just silently crashes) so I disabled the upgrade prompt: sudo softwareupdate --ignore "macOS Catalina" defaults write com.apple.systempreferences AttentionPrefBundleIDs 0 killall Dock Apple should really slow down on major releases of macOS or stop altogether in my opinion. macOS Mojave is a great OS and it's basically feature complete. Just stick with that, introduce bug fixes and security patches as needed and I think people will be happy.

_-___________-_ 7y ago

If you have Homebrew or something similar, I recommend installing openssh through there -- you get a newer version to boot.

rovr138 7y ago
I always do this. I'm not sure what's the current state, but there are features on SSH I wasn't able to use due to the version provided being old. I know that `Include` on `config` is/was one. Include "some/path" This is something I use frequently that wasn't available on previous built in versions.
- oefrha 7y ago
  > I'm not sure what's the current state, but there are features on SSH I wasn't able to use due to the version provided being old. > I know that `Include` on `config` is/was one. That's both terribly out of date info and hardly ever true as far as I can tell. The Include directive was a new feature of OpenSSH 7.3, released on 2016-08-01.[1] Apple shipped OpenSSH 7.3 in macOS 10.12.2[2][3], released on 2016-12-13. That's a very reasonable four months gap. I only use the system ssh because stock OpenSSH didn't integrate well with system keychain many years ago (not sure about the current state). But I've been using the Include directive for a long time. [1] https://www.openssh.com/txt/release-7.3 https://www.openssh.com/txt/release-7.3 [2] https://opensource.apple.com/release/macos-10122.html https://opensource.apple.com/release/macos-10122.html [3] https://opensource.apple.com/source/OpenSSH/OpenSSH-209.30.4/openssh/version.h.auto.html https://opensource.apple.com/source/OpenSSH/OpenSSH-209.30.4...
2ion 7y ago
It's usually not a full replacement. SSH for macOS has some integration built in that current OpenSSH does not have, like Keychain integration.
- skoskie 7y ago
  Really? I have my ~/.ssh/config file set to “AddToKeychain” on all entries and it doesn’t seem to be a problem.
- lilyball 7y ago
  I thought Homebrew patched OpensSSH using Apple's keychain patch, but looking at the formula right now I see # Please don't resubmit the keychain patch option. It will never be accepted. # https://github.com/Homebrew/homebrew-dupes/pull/482#issuecomment-118994372 Sadly the homebrew-dupes repo seems to have been deleted so this comment can't be read anymore.
  - saagarjha 7y ago
    > Sadly the homebrew-dupes repo seems to have been deleted so this comment can't be read anymore. They're kinda bad at that in general :/
  - CKN23-ARIN 7y ago
    Archive of the discussion: https://archive.is/hSB6d https://archive.is/hSB6d > We are uncomfortable continually supporting a 1900+ line patch which upstream hasn't signed off on that has the potential to both compromise OpenSSH security and Keychain security. From 10.11 it will also be impossible to edit plists in /System/* without disabling rootless, which isn't a configuration we'll be intentionally supporting.
- _-___________-_ 7y ago
  I prefer ssh-agent anyway, but yeah, I think they did remove the keychain integration patch.
Doctor_Fegg 7y ago
Oh god no. Homebrew managing openssh has been the cause of more command-line instability and forced reinstalls than anything else I’ve encountered in the last few years of OS X (sorry, macOS). I’ve started installing stuff from source again just to prevent a cascade of Homebrew upgrades breaking everything.
- _-___________-_ 7y ago
  Never experienced this in a decade or so of using Homebrew's OpenSSH, but you can absolutely use something other than Homebrew to get a more up-to-date and standard OpenSSH install if you prefer.
- lloeki 7y ago
  > I’ve started installing stuff from source again just to prevent a cascade of Homebrew upgrades breaking everything. Since you crossed that line, do yourself a favour and check out nixpkg.
- theonemind 7y ago
  I sometimes use NetBSD's pkgsrc on macOS because it installs super cleanly in any prefix you like and never, ever breaks the system. It doesn't have everything, and you will occasionally encounter a package that won't build, but it doesn't even dream of taking over /usr/local or disrupting your system. You could install it into your home directory if you wanted to (which I have done, on systems where I don't have root or enough ownership to just throw things anywhere)
- rswail 7y ago
  Why don't people use MacPorts instead? I've never had any problems with it. Homebrew wants to screw around in /usr, Macports installs itself in /opt and doesn't interfere with things in the MacOS world. Set your PATH to have /opt/local/{bin,sbin} and everything Just Works.
  - saila 7y ago
    What is the practical difference between /usr/local and /opt or /opt/local? I don't think macOS puts anything in /usr/local.

deleted 7y ago

[deleted]

awinder 7y ago

“I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina.” Welp.

draw_down 7y ago
Which is a weird sentiment. HN is, let's say, not exactly shy about criticizing Apple hardware/software, especially software.
hoistbypetard 7y ago
Which is more common? Someone says "I don't want to end up on ___ news again" and they really mean they don't? Or they say that and they really want to? It feels like the old "Please, Br'er Fox, don't upvote this post. I don't want anyone to see it." (I'm about 80% kidding but amused. Lockdown.)
- tylerhall 7y ago
  Sadly, I mean it. Don't want to go through this again. Taking down the post for now.
  - azinman2 7y ago
    Go through what? Why would you take down the post as it’s probably useful to others?
    - untog 7y ago
      Hacker News can be a cesspool at times. One important thing to always remember is that unless someone posted their article to Hacker News themselves they might have had absolutely no expectation that a huge audience was about to descend and dissect everything they wrote. They might have just been talking off the cuff, mentally noodling around or even just using the process of writing stuff down as a means to sort their thoughts. Far too often HN commenters work from the assumption that an author is intending to make A Big Point and very uncharitably deconstruct every sentence the author wrote. It's only a matter of time before we see a reply along the lines of "OBVIOUSLY 10.15.4 did NOT break SSH, the author just didn't do X Y and Z to fix a very OBVIOUS mistake in their SSH config". > Why would you take down the post as it’s probably useful to others? More broadly, Tyler doesn't owe the world anything in this regard. If he wants to post it, cool. If he wants to remove it, cool.

floatingatoll 7y ago

To summarize OP: Their 10.15.4 macOS built-in ssh terminal command is unable to reach hostnames when a port number higher than 8192 is used. EDIT: Comments differ; one indicates issues SSH'ing to lower than 8192 ports, another indicates no issues SSH'ing to higher than 8192 ports.

weej 7y ago
And Tyler "end(ed) up on Hacker News again bitching about Catalina"
- saagarjha 7y ago
  That’s not completely his fault, is it?
Apofis 7y ago
chaos:1111

derefr 7y ago

Is macOS /user/bin/ssh just upstream OpenSSH, or does Apple maintain a fork? If no fork, this would be an upstream OpenSSH problem, no?

saagarjha 7y ago
https://opensource.apple.com/source/OpenSSH/OpenSSH-235/ https://opensource.apple.com/source/OpenSSH/OpenSSH-235/
- paxswill 7y ago
  Looks like that's for macOS 10.15.0 as version.h for that OpenSSH release is for 7.9p1 (macOS 10.15.4 has OpenSSH 8.1p1).
  - saagarjha 7y ago
    It is, I pulled it from https://opensource.apple.com/release/macos-1015.html https://opensource.apple.com/release/macos-1015.html.
paxswill 7y ago
Apple includes a customized version of OpenSSH. From what I recall from the last time I looked at it, the changes were mostly integrating the key retrieval mechanisms with the rest of macOS. For example, Apple's ssh-add can store key passphrase in Keychain with the -K option, and then later access those passphrase with the -A flag.
- floatingatoll 7y ago
  Those stored key passphrases are visible with the Keychain Access application, Kind: "application password", name: "SSH: /full/path/to/key", in the login & iCloud keychains.
- skoskie 7y ago
  If using the upstream version there is one line to add to a startup script or to your zshrc (et. al.) file ... ssh-add -A > /dev/null ... and one default value to place in your ssh config file... AddToKeychain Yes ... to get around this issue. It works fine after that. (On mobile. Sorry for formatting)

dimtion 7y ago

I'm surprised nobody noticed this bug at Apple before the release. Is there nobody there that connects by hostname to a ssh server with a port > 8192?

deleted 7y ago
[deleted]
saagarjha 7y ago
Probably not. I doubt I’ve used a port that high.
- Yetanfou 7y ago
  Why not? I regularly use 5-digit ports - 3-digit prefix plus 2-digit 'official' port - for various systems which reside behind a NATting router.
  - the_mitsuhiko 7y ago
    I'm pretty sure it's generally not advisable to use ports this high since they are used for other purposes.
    - Avamander 7y ago
      All purposes are equally as valid on unallocated port ranges. Inconfigurable port ranges with no fallback when those ports are in use is bad design.
      - user5994461 7y ago
        High port numbers, above 30000 usually, are ephemeral ports and get pre-empted by the system. They're not safe to listen to for server applications.
  - syncsynchalt 7y ago
    Ports <1024 require root access to bind, so on a multi-user system it would be insecure to run ssh on such a high port. (Granted, multi-user hosts are very rare nowadays).

AdmiralAsshat 7y ago

Mental note for future blog post: the secret to getting onto the front page of HN is to publicly state that I do not want this blog post getting onto the front page of HN.

throwanem 7y ago
I feel like the subject matter also signifies. I mean, if I post a 5000-word paean to the Polistinae - they're good wasps, Brent - and say I don't want it to end up on the front page of HN, I feel like I'm not going to get what I don't want, you know?

hazebooth 7y ago

I'm on mac OS 10.15.4. SSH has worked on all betas and the official release from Apple.

beervirus 7y ago
Using a port above 8192 and connecting by hostname?
- hazebooth 7y ago
  Yes, but maybe I've mucked around with my ssh too much.
deleted 7y ago
[deleted]

sgehly 7y ago

>>I don’t want to end up on Hacker News again bitching about Catalina. Hahaha

ProAm 7y ago

"It just works" -- Is Apple too large now? Is this a QA problem, product team problem? Management? Catalina is still stumbling and Im surprised to be honest after the past 4 years.

cflewis 7y ago
My feeling is that Apple beancounters have decided macOS is mostly a gateway to Xcode for iOS development, anything else is just to help sell laptops. The stuff in "anything else" doesn't need to actually work well, just exist so it can be something on the features list.
- api 7y ago
  It does help that there is no overall competitor to MacBooks in terms of ease of use or (now that the butterfly keyboard is dead) build quality. There are decent build PC laptops but you have to run Windows or Linux on them. Windows is a dumpster fire these days with ads in the start menu, the use of "dark patterns" to herd people into MS cloud, and out of control unnecessary telemetry. Linux is fine only if you have a lot of time on your hands to troubleshoot edge case issues and hunt for drivers. Linux also still (through no fault of its own) can't run a lot of apps that many people need.
  - aftbit 7y ago
    Linux is the only option IMO, but I have a very high yakshaving tolerance. That said, if you run a recent Ubuntu, most stuff "just works" as long as you don't need Photoshop or the Office suite.
    - deleted 7y ago
      [deleted]
    - jacobsenscott 7y ago
      Or 4k monitors, or screen sharing when running more than one monitor. That's the reason I haven't switched from macos back to linux (I was all in on linux until about 5 years ago when I started to care about display quality and working remotely).

AsyncAwait 7y ago

Another post to my collection of how macOS works flawlessly and Linux breaks stuff when it comes around again.

rimliu 7y ago
I think the technical term for this is "cherry-picking".
- fanatic2pope 7y ago
  Exactly. Just like people do with Linux issues.
  - chipotle_coyote 7y ago
    As a long-time Mac user who gets irritated about something trivial within minutes of using any Linux desktop environment, I'd have to say this is pretty spot on. :)
- Yetanfou 7y ago
  If this tree only carried a few cherries you'd be right. By now it is such an abundant source of fruit that they'd do well by changing their logo to just this, a Catalina Cherry [1,2]. [1] https://en.wikipedia.org/wiki/Prunus_ilicifolia https://en.wikipedia.org/wiki/Prunus_ilicifolia [2] https://calscape.org/Prunus-ilicifolia-ssp.-lyonii-(Catalina-Cherry)?srchcr=sc5849e2614df95 https://calscape.org/Prunus-ilicifolia-ssp.-lyonii-(Catalina...

germs12 7y ago

Original Post: I was completely at my wit’s end and feeling like I had lost my mind until about a half hour ago. Let me start from the beginning… I don’t have an exact date, but within the last week I realized that I was unable to ssh into my primary web server – the one that runs my business website, activation server, etc. It’s sort of the linchpin for my tiny software company. When it goes down, I get worried. At first I thought maybe the server was down? I hadn’t received any alerts, so I did a quick check. And, yes, it was still up and running and serving web traffic. Ok, did sshd somehow become unresponsive? I login through the Linode control panel and restart the service. Still can’t login. It’s odd. I don’t get a connection refused. Not even a timeout. It just…hangs. That’s the ssh output with the verbose flag. Nothing. I waited 10+ minutes and it never timed out or produced any other output. I reboot the server itself and the problem persists. Then, I notice some more oddities. I’m able to connect using ForkLift – my FTP client, which connects via SFTP. Also, SequelPro is able to connect to MySQL via ssh as well. And then things get even stranger. This is all happening on my iMac. I try connecting from my laptop, and it works. My MacBook Pro is at home right next to my iMac, which is refusing to login. They’re both on the same wifi and thus the same IP. So, it can’t be that my home IP address got mistakenly banned somehow. Next, I ssh into a different server and then hop to the problematic one. It connects without any trouble. At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup. Still can’t login. Ok. I think about it for a few minutes and then – aha! – I have an Ubuntu virtual machine running on this iMac inside Parallels. I’ll ssh into it and then try and connect. That will rule out if there’s just something odd about my iMac’s LAN IP. (To be clear, my home network is perfectly ordinary. Just a cable modem and a router.) So, I login to the VM, try and connect, and it works fine. At this point here’s what I’ve found: My iMac is the only machine that cannot login. I’ve connected successfully from behind the same public IP using a laptop, a virtual machine, and my iPhone and iPad. I’ve verified my ssh keys are correct and have the appropriate permissions. I can connect to other servers from the problematic machine – both at the same hosting provider (Linode) and others (AWS and DigitalOcean). I can connect from my iMac if I jump through any other server, first. I start trying to think what could possibly be different about this one machine. And then it dawns on me. This all started around the time I updated my iMac to 10.15.4. My laptop is still on 10.15.3 – and, of course, the virtual machine isn’t macOS at all. Totally grasping for straws I google for “10.15.4 ssh” and find this top result on the Apple discussion forums: Catalina 10.15.4 SSH port > 8192 does not work when using server name instead of IP This issue started just after upgrading to macOS Catalina 10.15.4. After that update I am no longer able to open a SSH connection to a port greater than 8192 using server name (instead of IP). Yes, I do change the port on the server side prior to every test. That can’t possibly be real? Up until this point, I was connecting via a saved hostname defined in my ~/.ssh/config, which let me login simply by tying ssh some-server. So, I tried ssh ip-address -p9944 and it worked! (That server runs on an alternate ssh port.) Ok. Time to narrow this down a bit further. I changed the server to listen on standard port 22 and tried connecting via the hostname once again. Holy crap, it worked. The user in the Apple forums was right. At least in my case, my one server that happened to be running on a non-standard ssh port above 8192 will not connect from Catalina 10.15.4 when using the hostname instead of the IP address. Just to verify, I boot up a Mojave and Catalina (10.15.3) VM on the same iMac. They both connect fine, while the host machine continues to fail. The internals of this is all so incredibly above my head I have no idea what the underlying problem might be. Am I and this one other forum poster just doing something totally bizarre yet the same? This ssh setup has been working for years for me until just the last week. I would love to be proven wrong and told I’m an idiot. But I don’t know what difference connecting via the hostname versus the IP address would make when specifically using a non-standard port above a certain threshold. It just….sigh. I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer.

slovette 7y ago

Catalina is broken in many ways. This complain and Remote Access in (so I can SSH to my $4k MacBook) disables itself anytime the computer is restarted. But more importantly, I’ve still not found a Thunderbolt Display that doesn’t routinely crash screen manager services upon idle user activity. 3 x $300 thunderbolt3 dock solutions later and not a one hasn’t crashed this computer. All main brands, two of which sell accessories in the Apple store. Problem also existed with a top of the line 13” MacBook Pro. I’ve just gotten used to the shoddy-ness that is Catalina. Figure if I go to the bathroom, upon return I have a fresh, new clean desktop environment. Feature not a bug. Yay!

unicornmama 7y ago
I completely disabled everything to do with sleep. “Solved” the problem for me.
- anonexpat 7y ago
  The first thing I install on a fresh Mac is amphetamine.
  - SamuelAdams 7y ago
    Just curious, how is this different from Caffeine? http://lightheadsw.com/caffeine/ http://lightheadsw.com/caffeine/
    - blacksmith_tb 7y ago
      Or just running caffeinate -t 10000?
    - wyclif 7y ago
      One difference is that Amphetamine isn't updated as well, also seems more hackish. I switched to Caffeine a while ago.
    - oefrha 7y ago
      It has more features. Check https://apps.apple.com/us/app/amphetamine/id937984704?mt=12 https://apps.apple.com/us/app/amphetamine/id937984704?mt=12 for details. The one you linked to looks like a simple wrapper of caffeinate(8).

nathell 7y ago

I've been a Linux user for the last ~15 years, and now I need to do some iOS development so just a few days ago I've ordered a Mac Mini. I guess I'm in for a bumpy ride. Oh well.

redsymbol 7y ago
I used linux on all my laptops/work machines for about 7 years, then switched to macbook pros 5 years ago. Definitely some things you have to adapt to (I still miss focus-follows-mouse). But for the most part, you'll find it's a smooth ride. When bumps like this happen, they tend to push out a fix quickly - especially when it gets traction on HN like this.
- john_alan 7y ago
  macOS is POSIX UNIX. it definitely has focus follows mouse. You just mean that hover over a lower window allows scrolling right? MacOS has that.
  - redsymbol 7y ago
    Not just scrolling. https://en.wikipedia.org/wiki/Focus_(computing)#Focus_follows_pointer https://en.wikipedia.org/wiki/Focus_(computing)#Focus_follow...
- quesera 7y ago
  For me, focus-follows-mouse is most useful for terminal windows. It is a feature you can enable in iTerm2. This doesn't help across applications of course, and there's a reasonable argument that the inconsistency is worse than the absence -- but for me, iTerm2's FFM feature helps.
  - redsymbol 7y ago
    Didn't know iTerm2 did that. Thanks, I'll check it out - might help a lot.
- 7y ago

0x0 7y ago

I can't reproduce this. macOS 10.15.4, ssh'ing to a very high (5digit) port with a hostname no problems.

jpwgarrison 7y ago
Same here, 5 digits and a hostname = no problem. There must be some other factor(s) in play.
0xff00ffee 7y ago
Ditto. There's a post a few levels above where they are digging into the source of ssh, I'm following that!
ajphdiv 7y ago
I can't either, in fact all 10+ of the hosts that I routinely access have ports higher than OPs issue.
- andai 7y ago
  Offtopic but why are people using high port numbers? Additional security due to a nonstandard port? If so, does that go together with anything additional like port knocking? Or is it multiple hosts on the same IP, but different ports?
  - cozzyd 7y ago
    A relatively common use case is multiple devices behind a NAT, where each port goes to a different device.
  - deleted 7y ago
    [deleted]
  - vbezhenar 7y ago
    Some people think that it adds to security. Some people want to reduce noise in logs.
  - 7y ago

judge2020 7y ago

In case it's slow for you too: <removed since they wanted it taken down>

saagarjha 7y ago
It’s been taken down on purpose: https://news.ycombinator.com/item?id=22738841 https://news.ycombinator.com/item?id=22738841

Zelphyr 7y ago

As an aside; is there a reason to host SSH on a non-standard port? I recently came across a system that had it listening to a really high port number. I dismissed it as security through (bad) obscurity but is there a valid security reason to do this? EDIT: Thanks to everyone who answered my question! It makes sense to me now why one might do this.

0x0 7y ago
It reduces the amount of endless bruteforce attempts somewhat, so log files are slightly more readable. Although recently this seems to be significantly less effective compared to the previous 20 years...
joombaga 7y ago
If it's exposed to the internet, definitely. Port 22 gets hammered.
- wazoox 7y ago
  Indeed. In fact it's hammered so bad that it made my home machine crawl at times. By changing ports it went from several thousands attempts per hour to a few per day.
_-___________-_ 7y ago
The author didn't say it had anything to do with security. I have one or two SSH daemons out there listening on non-standard ports because of stupid limitations of middleboxes I'm forced to use, or because they're port-forwarded through something that already listens on 22 itself. Security-wise, it seems pointless; my daemons on random non-standard ports still get hammered, and fail2ban takes care of keeping the log spam down just as easily as it does the ones on 22.
deleted 7y ago
[deleted]
spzb 7y ago
It massively reduces the number of script kiddie attacks. It's not hard to find SSH on a non-standard port but most SKs don't know or don't bother. Other possible reason is NAT. If you've got several machines or VMs but only one public IP you can port forward different public ports to port 22 on different machines. Not the only solution by a long way but a relatively straightforward one.

vgene 7y ago

I had the same problem on a MacBook after upgrading to 10.15.4. However, I wasn't using a port number higher than 8192, the socket was 75 with a hostname. The problem was solved when I replaced the hostname with its IP or plugged in an Ethernet Cable. I tried to restart mDNSResponder and flush the dns cache and switch to a different DNS server. Nothing works so far.

deeblering4 7y ago

> I changed the server to listen on standard port 22 and tried connecting via the hostname once again. Holy crap, it worked. Shocking! Granted high ports shouldn't be broken, but running SSH on a non-standard port is security (read obscurity) theater at best. There's really not much benefit, unless you need multiple sshds on the same IP, but at that point I'd question the sanity of the approach.

notyourday 7y ago
For me SSH on a standard port get connection attempts and every few minutes. SSH on a non-standard port get connections attempts every few weeks.
lukevp 7y ago
Security through obscurity shouldn’t be used as an edict that something is not effective. You are talking about the fact that it doesn’t increase the security of the protocol itself or the passphrases/keys used. This is true. However, there are tons of bots out there that scan 22 and try to exploit common logins. There are presumably quite a few less that are port scanning every machine for every possible high port and attempting to handshake ssh and then try logins. Do you disagree with that? If not, this is not security through obscurity, it has a very real impact on the volume of bots that have knowledge that this service is running and are actively exploiting it. It’s just a different type of security, it’s discoverability of the service. Here’s another example. Say you have a web server running that is only for internal employee use. But you want to expose it externally so that they can reach it without a VPN. Even if you follow proper security protocols, why would you not turn off search engine indexing on this page, and limit the pages that link to it? It will not increase the inherent security of the protocol or the user accounts, but it will drastically lower the # of bots using up CPU and iptables entries trying to fail2ban or blacklist them. Security is a spectrum and you want to have defense in depth. Moving ssh to a nonstandard port is a security best practice and you shouldn’t be advising people not to use it. But should they also have good key setting, fail2ban, ip whitelisting/blacklisting, etc? Of course they should.
- deeblering4 7y ago

makecheck 7y ago

So, some quick debugging here... In his screenshot the bad login hangs at "Connecting to clickontyler.com port" (noting that no port number appears and no period at the end). While I can’t be sure exactly which "ssh" patch Apple may have, this seems to be the relevant file and logging code (starting at line 448): https://github.com/openssh/openssh-portable/blob/master/sshconnect.c#L488 https://github.com/openssh/openssh-portable/blob/master/sshc... In that code, the only thing that can set the "strport" value that is used in the log is a call to getnameinfo(). If that string is corrupted in any way, e.g. not terminated or perhaps has invisible characters that trigger bad terminal behavior (such as invisibility), the act of logging it might produce the apparent hang seen here. Again, a guess but it is possible that getnameinfo() is not necessarily processing the record correctly (for whatever reason). One such example is in the "getnameinfo" man page at the end, under CAVEATS, where they show an example of not simply trusting the result of the first call.

0x0 7y ago
Maybe there is something funny in /etc/services on this machine that throws the call into an infinite loop? Perhaps near the bottom beyond port 8192?
tylerhall 7y ago
Good sleuthing, but the missing port number is simpler than that. I just blacked it out of the screenshot. I know very well that running sshd on a non-standard port has no benefits security-wise, but it does lessen the length of my log files from dumb script kiddies. I redacted the port in the screenshot for that reason.
- jonny_eh 7y ago
  You should mention that in the caption, or use a non-black colour as a mask.
- 0x0 7y ago
  Have you tried running ssh in lldb/gdb and dumping a stacktrace when it hangs? Might have to copy the ssh binary to a temp dir to avoid SIP denying ptrace.

StreamBright 7y ago

Thank god I did not upgrade. Software upgrades are the best way to waste time.

rovr138 7y ago
They're the best way to stay secured and receive new features. The issue with them is lack of testing before deploying them.
- StreamBright 7y ago
  You can just get the security patches for Mojave. You are trying to make it sound like of you are on previous version there are no security patches. Factually untrue.
- anonymou2 7y ago
  I think proprietary software and staying secure are contradictory ideas.

tambourine_man 7y ago

>I don’t want to end up on Hacker News again bitching about Catalina.

davidkuhta 7y ago
To be fair, full context tempers that sentence: > I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer.

deleted 7y ago

[deleted]

chmaynard 7y ago

Apple's stance is that it didn't happen unless someone reports it using Radar (internal) or bugreport.apple.com (external). Unfortunately, they don't believe in Linus's Law, which states that "given enough eyeballs, all bugs are shallow."

saagarjha 7y ago
Things that end up on the front page of Hacker News get fixed.

oefrha 7y ago

One more data point: just tried to repro with hostname and port 8193 and failed, so the issue is probably more intricate than described. (Guest in my test: OpenSSH 7.6p1 on ubuntu bionic, stock config other than sshd port.)

zimpenfish 7y ago
Tried with dropbear and sshd on Arch, port 9022 from 10.15.4 Beta (19E258a) - no issues with a variety of hostnames that end up at the same host.
- zimpenfish 7y ago
  Just updated to 10.15.5 Beta (19F53f) and still no issues with dropbear or sshd.

colechristensen 7y ago

I experienced a similar issue with a git repository hosted on a high port, `brew install openssh` fixed it even though the homebrew `ssh` was not first on my $PATH. Didn't bother to investigate further.

moonchild 7y ago
Possibly that overwrote the config file for the system ssh?
- colechristensen 7y ago
  I was thinking homebrew's git perhaps had a different $PATH (or was using shared objects?) that used the different openssh. Just guessing, didn't seem worth my effort at the time.
saagarjha 7y ago
Isn't /usr/local/bin first on $PATH?
- KarlKemp 7y ago
  Homebrew tends not to install to bust/local/bin when that would replace a native version.

jki275 7y ago

I'm using ssh to a named server online using a port >8192 from a 10.15.4 machine right now. Have had no issues with it at all.

m_a_g 7y ago

>I don’t want to end up on Hacker News again bitching about Catalina. Ends up on Hacker News again bitching about Catalina.

jbverschoor 7y ago
I don't wanna pay taxes. Paying taxes anyway

kulix425 7y ago

"I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina." lol

krzysztofeng123 7y ago

laughs in Linux

dang 7y ago
Please don't do this here.

zeveb 7y ago

> Am I and this one other forum poster just doing something totally bizarre yet the same? One might uncharitably suggest that using macOS and expecting standard decades-old Unix behaviour is itself bizarre … but that's also true of using Linux with systemd (viz., nohup no longer nohups, or systemd-resolved, or innumerable other broken bits). It's almost as though no-one cares about quality anymore.

acdha 7y ago

I suspect this is due to a feature being enabled for canonicalization and that the key part is the presence of the colon rather than the port number. On a 10.15.4 system, I see a line in the debug output which is not present in the screenshot: > debug1: resolve_canonicalize: hostname example.org:7999 is an unrecognised address If instead I use `-p` or a config-file option, everything works as expected.

lilyball 7y ago
hostname:port is not a valid destination according to ssh syntax. A destination may either be [user@]hostname or a URI of the form ssh://[user@]hostname[:port].
richardkmichael 7y ago
The Apple forum article cites by the OP indicates it's a problem with `-p <PORT>` as well. https://discussions.apple.com/thread/251226509 https://discussions.apple.com/thread/251226509
jki275 7y ago
That's a fair point. I use either a ssh config file with all the correct options in it, or the -p option if I'm doing it without a config file, perhaps that's why I've never had a problem.

yasp 7y ago

Catalina also broke `apropos`. https://apple.stackexchange.com/questions/374025/errors-from-whatis-command-unable-to-rebuild-database-with-makewhatis https://apple.stackexchange.com/questions/374025/errors-from...

geuis 7y ago

Looks like the post was just deleted.

deleted 7y ago

[deleted]

viburnum 7y ago

If I'm still on Sierra, which version should I upgrade to?

v64 7y ago
I have a MacBook Pro (Retina, 13-inch, Mid 2014) and Mojave runs very well on it. I have no plans to upgrade to Catalina.

di3goleite 7y ago

I faced the same problem two weeks ago with the previous version of Catalina (I don't remember the correct number but was a previous on 10.15.4) and git (I use SSH to authenticate with the server). So I did a report to Bitbucket with a solution that worked for me after investigate more about the problem: https://twitter.com/di3goleite/status/1239596891471581189?s=20 https://twitter.com/di3goleite/status/1239596891471581189?s=... Thank you about that clarification. Also your website seems to be down actually.

JdeBP 7y ago

Here's an actual bug report: * https://openradar.appspot.com/radar?id=4931259776106496 https://openradar.appspot.com/radar?id=4931259776106496 From that and the discussions.apple.com. post, hyperlinked elsewhere in this discussion, it appears that the >8192 condition varies according to what the hostname actually is. The bug report is datelined 2020-04-26, interestingly. There might be a bug in the bug reporting system. (-:

saagarjha 7y ago
OpenRadar is community-maintained, and rather poorly at that these days.
oefrha 7y ago
> The bug report is datelined 2020-04-26, interestingly. There might be a bug in the bug reporting system. No, you can type whatever date you want. The "add a new radar" screen is just a bunch of text input boxes: https://i.imgur.com/nNf457J.png https://i.imgur.com/nNf457J.png
- JdeBP 7y ago
  The ability to type whatever date one wants is often considered to be a bug. The ability to post-date reports a month into the future sometimes is, too. (-:
  - oefrha 7y ago
    You can not only type whatever date, but also whatever non-date. The site just assumes good intentions and is working as designed.

neonate 7y ago

https://archive.md/lvCKA https://archive.md/lvCKA

deleted 7y ago
[deleted]
ncmncm 7y ago
Anyway the parent item seems to be something about a specific, named release of Apple's proprietary MacOS operating system. It seems we are being warned that if we upgrade to it, something will not work anymore. Those of us not inclined to install Catalina may sigh in relief; our instances of SSH have not been broken.
- new_realist 7y ago
  99.999% of all of us got that just from the title, thanks. Congrats on puzzling it out.
  - sudoaza 7y ago
    I didn't and funny you'd think Mac userbase is that big
    - albedoa 7y ago
      Mac users are not the only group who know what those words mean.
      - segfaultbuserr 7y ago
        Yeah, I don't use macOS and I don't own any Apple computer. But a weird California location name + version number 10 is a dead giveaway.
        sudoaza 7y ago
        didn't know Catalina was a California location either, to me is just a female name. I actually though she might be a contributor to SSH project.

buildbot 7y ago

Wow, just the other day this started happening to me as well with one of my serversfrom my MacBook. It used to work fine, but now only that laptop can’t connect to it. iIt’s on a high port too.

stock_toaster 7y ago

Maybe a weird ControlMaster/ControlPath config? I have had issues with the ControlPath result being too long with certain hostname/port combinations in the past -- which resulted in ssh to ip working but ssh to hostname not working. As a result, I haven since started using %C instead of %l%h%p%r in my ControlPath config.

anongraddebt 7y ago

"I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina." +1

tkubacki 7y ago

It is time to move to Linux or Windows desktop. Really if you are not hostage of Apple ecosystem then decent desktop is much more reliable in my experience (got old MacBook which is ok too but eg can't connect to old vpn on it)

stinos 7y ago

At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup Is that a common thing to do, or any reason why the OP would do that? Doesn't ssh reject your key, saying it does that if there's such a problem? And even if not wouldn't it be advisable to at least look at the permissions; I mean suppose they're not -rw------- or so, wouldn't you want to know that, and also why they are not ok?

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

KiDD 7y ago

"I don’t want to end up on Hacker News again bitching about Catalina." Pretty sure that guarantees getting to the front page :D

phlakaton 7y ago
This is truly the darkest timeline for that poor blogger. :-P
spaniard_dev 7y ago
Those are the magic words. F*ck SEO, add that and you'll get a couple million page views.

liquidify 7y ago

Are you talking about the new apple feature?... non working ssh? It's all the rage.

LilBytes 7y ago

This is an anecdote but the latest update forced me to rebuild my Mac from a hard/factory reset. My Dell D6000 on my 2019 MacBook Pro no longer charges the laptop. I've tried. My Mac's resources were getting gobbled up by an internal process I coudln't terminate and my keychain was borked and I couldn't log in after a reset (to try and get around the resource hogging). Recovery didn't get me any where so I used Recovery over the Internet to do a clean install. I'm running 10.15.4, no issues as of yet. And this all occurred after the security update. I'm running on the version prior for now but will make sure I've got a good backup and give it another go.

gangstead 7y ago
I had a d6000 and had to update the firmware (with a borrowed windows machine) before it would charge my mbp.
- LilBytes 7y ago
  Hadn't considered that, I'll give it a go. Thanks!
  - LilBytes 7y ago
    If anyone comes across this comment, this 100% worked.

ThePowerOfFuet 7y ago

> Next, I ssh into a different server and then hop to the problematic one. It connects without any trouble. At this point I’m thinking maybe the permissions on my local private key got screwed up. So, I blow away ~/.ssh and recreate all of my keys from a backup. Still can’t login. Someone should have paid more attention to that verbose SSH output first.

teilo 7y ago

Well, the description of this bug is not generally reproducible, so whatever is causing it, it's not as simple as using a high port with a server name. I tested this specifically on a number of servers that I run with port numbers > 10000, using /usr/bin/ssh on macOS 10.15.4, with and without IP addresses. Nothing broke for me.

ulkesh 7y ago

I agree, that’s an annoying bug/feature. However, there is an amazingly easy workaround, assuming the IP and port don’t change often: create a ~/bin shell script that connects via IP and port, make it executable, and add ~/bin to PATH. This workaround doesn’t excuse Apple of doing something so egregiously stupid, but it’s so easy that you may as well do it and move on.

ipv6ipv4 7y ago

In my experience the breakage is with IPv6. Try forcing IPv4. 'ssh -4 <hostname>'

epiphanitus 7y ago

What’s the backstory to why apples bash has to be different than gnu bash? I love having the Linux kernel with a nice UI but there are some useful commands that are missing. There are ways to get them set up but in any case it’s kind of a pain

Elrac 7y ago

Am I the only one who thought this article was about Apache Tomcat?

steve1977 7y ago

"I don’t want to end up on Hacker News again bitching about Catalina. I just hope I’ve stuffed this post with enough keywords so that anyone else searching on Google might come across the answer." Ok. And did you actually report it as a bug to Apple?

wyattpeak 7y ago
Whether or not he did, there's absolutely nothing wrong with posting about a problem in case others come across the same. In fact it's damned helpful.

neilwilson 7y ago

Catalina seems to have bust Wifi monitor mode on tcpdump on my MacMini 2018, yet it works fine on my Mac Air. Still not sure if that is my machine, or a general fault - but the lack of monitor and promiscuous mode is playing havoc with IPv6 multicast packets from VMware Fusion VMs.

vgaldikas 7y ago

>I’m not even going to go into it. I don’t want to end up on Hacker News again bitching about Catalina. Whoops