5 ms·

Manjaro website off-line again due to lapsed certificate

66 points by hexagonsuns 7mo ago

vpShane 7mo ago
not the first time, I stopped using manjaro when I noticed ping.manjaro.org was being pinged every 30 seconds on a new router I setup. nothanks on that. but seriously, sudo crontab -e, @monthly cerbot renew No excuses.
- fishgoesblub 7mo ago
  It's not uncommon for a Distro to point NetworkManager or whoever to check for connectivity using their own servers, Arch does it themselves[0]. [0] ping.archlinux.org
- altairprime 7mo ago
  Note that the certbot instructions are to renew 2x a day with up to one hour of randomized delay; using @monthly as suggested here will result in occasional outages if the "once a month" renewal attempt fails in two consecutive months due to transient peak service blips (such as those caused by '@monthly' hardcoding for month X day 1 time 00:00 often UTC without randomization), especially as Let's Encrypt drops their lifetimes to 45 days over the next 2 years, which would result in certificates avoidably expiring in production. Please instead use certbot's recommended 2x/day renew with a random sleep of up to an hour before initiating each attempt; at least one of cronie, at, bash, python, perl random sleep methods are available on most* platforms, and are offered up by the crontab-command generator at https://certbot.eff.org/instructions https://certbot.eff.org/instructions . * There is a stack overflow page from 2016 filled with solutions for Busybox, so I'd say 'all' rather than 'some' but someone out there is hosting a webserver on a potato, so better safe than sorry.
- marginalia_nu 7mo ago
  Certbot would be like the supply chain attack holy grail. Not sure I'd want software like that running unmonitored automatically with root privileges.
joecool1029 7mo ago
Oops, it's back now though...
Mond_ 7mo ago
I used Manjaro for a few years. That's how I learned a pretty important lesson about software engineering that still informs how I work to this day. "A layer of abstraction on top of a stateful legacy system often doesn't result in a simpler system, it just introduces exciting new failure possibilities. This especially applies when the owners of the legacy system have no responsibility over the abstraction layer."
- peeters 7mo ago
  This comment made a lot more sense to me once I realized we weren't talking about an aggressively marketed weight loss drug.
  - ChadNauseam 7mo ago
    It's still true. Your metabolic system is probably not simpler after taking terzepatide. Although, just because it's not simpler doesn't mean it can't be better. I'm very glad for the C++ abstraction layer over assembly, even if the stack is more complicated than if it were just assembly
- perching_aix 7mo ago
  The word "legacy" doesn't seem needed there.
- HendrikHensen 7mo ago
  Can you explain it for those out of the loop?
  - Mond_ 7mo ago
    Manjaro sells itself as "Arch, but more approachable". In reality, you'll often end up with "Arch, but with additional weird package management upgrade issues that are a byproduct of Manjaro's own repositories interacting with the arch on your system." Instead of just having to track the arch repos, you suddenly have those and Manjaro's own stuff (and own package manager tool etc.), which is another point of failure. Every new bit of technology is another part that can fail.
ddtaylor 7mo ago
A lot of repositories and similar go offline randomly. It hasn't happened in a few months but usually the Microsoft package mirrors go past their Azure limits and I get reminders.
- arcanemachiner 7mo ago
  This is like the third or fourth time this has happened to them. The Manjaro team has also caught flak for a bunch of other stuff. There's a page or two our there that detail the issues, which I'm too lazy to link here. But let's just say this isn't their first rodeo.
  - xethos 7mo ago
    Agreed. This is not the first time Manjaro has made a boneheaded mistake, nor will it be the last. This is just the most recent.
  - AuthAuth 7mo ago
    The page is pretty nitpicky with its issues. There is only 1 that was actually something to concern over iirc.
  - alexpotato 7mo ago
    Going to play devil's advocate and say that they make minimal to no revenue off of their website so it being down is not a huge deal. It's exactly the opposite to what happens if the the main ad server for a company in the ad serving business looks at things. Or another example: From an inventory management perspective, it's ok to be out of stock for low margin items b/c the opportunity cost is low.
    - deleted 7mo ago
      [deleted]
    - ddtaylor 7mo ago
      > Going to play devil's advocate and say that they make minimal to no revenue off of their website so it being down is not a huge deal. How much revenue they make doesn't matter if it impacts users. Prospective users need to be able to see what they are getting and download and verify ISOs from the team. I despise ZorinOS for what they are doing in many ways that violate software licenses, but one thing they got right is charging users and being somewhat accountable to their userbase.
allddd 7mo ago
At this point we have to assume they're doing it for attention. I refuse to believe a team of people that can ship an OS, even if it's just a riced Arch, cannot figure out acme.sh. Come on...
aslihana 7mo ago
I love Manjaro too much, use it as daily distro but their certificate issues and its recursive behaviour threaten me a little bit.
- hexagonsuns 7mo ago
  Try CachyOS
  - brendyn 7mo ago
    Same issue as Monds comments. When arch pushes an update the package database becomes inconsistent until cachyos syncs it, which can take 30 minutes. Cachy packages just increment a nonce on the package version numbers so when arch pushes updates they get considered to be more recent than the Cachy versions, cause .so dependency errors. It's all just one fragile stateful system
    - hexagonsuns 7mo ago
      yeah its fast doe
- retrochameleon 7mo ago
  I went from Manjaro to EndeavourOS. Great experience and I don't see it changing anytime soon. The community is healthy and the project well managed.
9cb14c1ec0 7mo ago
Just use Caddy. It's that simple.
- dijit 7mo ago
  "I use arbitrarily complex software that has a rapid SDLC to obfuscate the issue with the fact that we have to have military grade encryption for displaying the equivalent of a poster over the internet". The state of our industry is such that there will be a lot of people arguing for this absurdity in the replies to me. (or I'll be flagged to death). Package integrity makes sense, and someone will make the complicated argument that "well ackshually someone can change the download links" completely ignoring the fact that a person doing that would be quickly found out, and if it's up the chain enough then they can get a valid LE cert anyway, it's trivially easy if you are motivated enough and have access to an ASN.
  - harmonics 7mo ago
    Nah, you've simply never lived in a country which is afraid of its own population and does (or tried to) MITM internet traffic. Mine does both, there was a scandal several years ago: https://news.ycombinator.com/item?id=20472179 https://news.ycombinator.com/item?id=20472179 I'll take enforced HTTPS for absolutely everything, thank you very much. Preferably with certificate pinning and similar aggressive measures to thwart any attempts to repeat this.
    - hulitu 7mo ago
      > MITM internet traffic. Are there countries that don't do this ?
  - ratorx 7mo ago
    Changing the links and doing nothing else would be a pretty dumb MITM. You could do a more complex variant which is not so easy to spot (targeting specific networks, injecting malware whilst modifying the checksum) The key property of SSL that is useful for tamper resistance is that it’s hard to do silently. A random ASN doing a hijack will cause an observable BGP event and theoretically preventable via RPKI. If your ISP or similar does it, you can still detect it with CT logs. Even the issuance is a little better, because LE will test from multiple vantage points. This doesn’t protect against an ISP interception, but it’s better than no protection.
KronisLV 7mo ago
Uptime Kuma supports certificate expiry notifications and will send you messages in whatever channel (e.g. e-mail, Slack, ...) you configure ahead of time: https://uptimekuma.org/ https://uptimekuma.org/ That way, even if some of your automation is borked (or if you don't have any), you'll at least be reminded. Though with this being pushed, feels like nobody will have much choice, but automate: https://www.digicert.com/blog/tls-certificate-lifetimes-will-officially-reduce-to-47-days https://www.digicert.com/blog/tls-certificate-lifetimes-will...
nottorp 7mo ago
Technically it wasn't offline, was it? You could even browse it if you used a browser who still treats you like an adult and allows you to ignore certificate warnings.
- UqWBcuFx6NV4r 7mo ago
  [flagged]
toddgardner 7mo ago
If you never want this to happen again to your systems, we’re building a tool that bakes monitoring and validation into automatic cert renewals. <https://www.certkit.io/ https://www.certkit.io/>
- exac 7mo ago
  Respectfully we have had Certbot for 11 years now.
- LorenDB 7mo ago
  Meanwhile Caddy exists
- 9dev 7mo ago
  Paying for certificates..? Manually copying cert files? Man, this reads like it was 2010 or something. Best of luck, but I don’t know why I wouldn’t just use acme.sh and systemd timers instead of this.
- lachiflippi 7mo ago
  You're developing "certbot, but it's paid and sends private keys around the network instead of generating the csr locally"? Why? Who's the target audience? Platforms that can't run certbot, or any of the infinite amount of other acme clients, most likely won't be able to run your agent as well, so what's the value add vs just running a regular, well-defined (and free!) acme client and just moving the cert over manually?
PaulCarrack 7mo ago
I'm often stuck at my desk for long hours, which made it difficult to maintain a healthy weight. I started using Manjaro a few months ago, and it's had a significant impact. It helped control my appetite and reduced the constant cravings I'd usually get, making it easier to make healthier choices. I've lost around 15-20 lbs so far, and I feel much more in control of my eating habits. I know it's a bit anecdotal but it's been an important part of my weight loss journey
johnbarron 7mo ago
[dead]

66 points by hexagonsuns 7mo ago

not the first time, I stopped using manjaro when I noticed ping.manjaro.org was being pinged every 30 seconds on a new router I setup. nothanks on that. but seriously, sudo crontab -e, @monthly cerbot renew No excuses.

fishgoesblub 7mo ago
It's not uncommon for a Distro to point NetworkManager or whoever to check for connectivity using their own servers, Arch does it themselves[0]. [0] ping.archlinux.org
altairprime 7mo ago
Note that the certbot instructions are to renew 2x a day with up to one hour of randomized delay; using @monthly as suggested here will result in occasional outages if the "once a month" renewal attempt fails in two consecutive months due to transient peak service blips (such as those caused by '@monthly' hardcoding for month X day 1 time 00:00 often UTC without randomization), especially as Let's Encrypt drops their lifetimes to 45 days over the next 2 years, which would result in certificates avoidably expiring in production. Please instead use certbot's recommended 2x/day renew with a random sleep of up to an hour before initiating each attempt; at least one of cronie, at, bash, python, perl random sleep methods are available on most* platforms, and are offered up by the crontab-command generator at https://certbot.eff.org/instructions https://certbot.eff.org/instructions . * There is a stack overflow page from 2016 filled with solutions for Busybox, so I'd say 'all' rather than 'some' but someone out there is hosting a webserver on a potato, so better safe than sorry.
marginalia_nu 7mo ago
Certbot would be like the supply chain attack holy grail. Not sure I'd want software like that running unmonitored automatically with root privileges.

joecool1029 7mo ago

Oops, it's back now though...

Mond_ 7mo ago

I used Manjaro for a few years. That's how I learned a pretty important lesson about software engineering that still informs how I work to this day. "A layer of abstraction on top of a stateful legacy system often doesn't result in a simpler system, it just introduces exciting new failure possibilities. This especially applies when the owners of the legacy system have no responsibility over the abstraction layer."

peeters 7mo ago
This comment made a lot more sense to me once I realized we weren't talking about an aggressively marketed weight loss drug.
- ChadNauseam 7mo ago
  It's still true. Your metabolic system is probably not simpler after taking terzepatide. Although, just because it's not simpler doesn't mean it can't be better. I'm very glad for the C++ abstraction layer over assembly, even if the stack is more complicated than if it were just assembly
perching_aix 7mo ago
The word "legacy" doesn't seem needed there.
HendrikHensen 7mo ago
Can you explain it for those out of the loop?
- Mond_ 7mo ago
  Manjaro sells itself as "Arch, but more approachable". In reality, you'll often end up with "Arch, but with additional weird package management upgrade issues that are a byproduct of Manjaro's own repositories interacting with the arch on your system." Instead of just having to track the arch repos, you suddenly have those and Manjaro's own stuff (and own package manager tool etc.), which is another point of failure. Every new bit of technology is another part that can fail.

ddtaylor 7mo ago

A lot of repositories and similar go offline randomly. It hasn't happened in a few months but usually the Microsoft package mirrors go past their Azure limits and I get reminders.

arcanemachiner 7mo ago
This is like the third or fourth time this has happened to them. The Manjaro team has also caught flak for a bunch of other stuff. There's a page or two our there that detail the issues, which I'm too lazy to link here. But let's just say this isn't their first rodeo.
- xethos 7mo ago
  Agreed. This is not the first time Manjaro has made a boneheaded mistake, nor will it be the last. This is just the most recent.
- AuthAuth 7mo ago
  The page is pretty nitpicky with its issues. There is only 1 that was actually something to concern over iirc.
- alexpotato 7mo ago
  Going to play devil's advocate and say that they make minimal to no revenue off of their website so it being down is not a huge deal. It's exactly the opposite to what happens if the the main ad server for a company in the ad serving business looks at things. Or another example: From an inventory management perspective, it's ok to be out of stock for low margin items b/c the opportunity cost is low.
  - deleted 7mo ago
    [deleted]
  - ddtaylor 7mo ago
    > Going to play devil's advocate and say that they make minimal to no revenue off of their website so it being down is not a huge deal. How much revenue they make doesn't matter if it impacts users. Prospective users need to be able to see what they are getting and download and verify ISOs from the team. I despise ZorinOS for what they are doing in many ways that violate software licenses, but one thing they got right is charging users and being somewhat accountable to their userbase.

allddd 7mo ago

At this point we have to assume they're doing it for attention. I refuse to believe a team of people that can ship an OS, even if it's just a riced Arch, cannot figure out acme.sh. Come on...

aslihana 7mo ago

I love Manjaro too much, use it as daily distro but their certificate issues and its recursive behaviour threaten me a little bit.

hexagonsuns 7mo ago
Try CachyOS
- brendyn 7mo ago
  Same issue as Monds comments. When arch pushes an update the package database becomes inconsistent until cachyos syncs it, which can take 30 minutes. Cachy packages just increment a nonce on the package version numbers so when arch pushes updates they get considered to be more recent than the Cachy versions, cause .so dependency errors. It's all just one fragile stateful system
  - hexagonsuns 7mo ago
    yeah its fast doe
retrochameleon 7mo ago
I went from Manjaro to EndeavourOS. Great experience and I don't see it changing anytime soon. The community is healthy and the project well managed.

9cb14c1ec0 7mo ago

Just use Caddy. It's that simple.

dijit 7mo ago
"I use arbitrarily complex software that has a rapid SDLC to obfuscate the issue with the fact that we have to have military grade encryption for displaying the equivalent of a poster over the internet". The state of our industry is such that there will be a lot of people arguing for this absurdity in the replies to me. (or I'll be flagged to death). Package integrity makes sense, and someone will make the complicated argument that "well ackshually someone can change the download links" completely ignoring the fact that a person doing that would be quickly found out, and if it's up the chain enough then they can get a valid LE cert anyway, it's trivially easy if you are motivated enough and have access to an ASN.
- harmonics 7mo ago
  Nah, you've simply never lived in a country which is afraid of its own population and does (or tried to) MITM internet traffic. Mine does both, there was a scandal several years ago: https://news.ycombinator.com/item?id=20472179 https://news.ycombinator.com/item?id=20472179 I'll take enforced HTTPS for absolutely everything, thank you very much. Preferably with certificate pinning and similar aggressive measures to thwart any attempts to repeat this.
  - hulitu 7mo ago
    > MITM internet traffic. Are there countries that don't do this ?
- ratorx 7mo ago
  Changing the links and doing nothing else would be a pretty dumb MITM. You could do a more complex variant which is not so easy to spot (targeting specific networks, injecting malware whilst modifying the checksum) The key property of SSL that is useful for tamper resistance is that it’s hard to do silently. A random ASN doing a hijack will cause an observable BGP event and theoretically preventable via RPKI. If your ISP or similar does it, you can still detect it with CT logs. Even the issuance is a little better, because LE will test from multiple vantage points. This doesn’t protect against an ISP interception, but it’s better than no protection.

KronisLV 7mo ago

Uptime Kuma supports certificate expiry notifications and will send you messages in whatever channel (e.g. e-mail, Slack, ...) you configure ahead of time: https://uptimekuma.org/ https://uptimekuma.org/ That way, even if some of your automation is borked (or if you don't have any), you'll at least be reminded. Though with this being pushed, feels like nobody will have much choice, but automate: https://www.digicert.com/blog/tls-certificate-lifetimes-will-officially-reduce-to-47-days https://www.digicert.com/blog/tls-certificate-lifetimes-will...

nottorp 7mo ago

Technically it wasn't offline, was it? You could even browse it if you used a browser who still treats you like an adult and allows you to ignore certificate warnings.

UqWBcuFx6NV4r 7mo ago
[flagged]

toddgardner 7mo ago

If you never want this to happen again to your systems, we’re building a tool that bakes monitoring and validation into automatic cert renewals. <https://www.certkit.io/ https://www.certkit.io/>

exac 7mo ago
Respectfully we have had Certbot for 11 years now.
LorenDB 7mo ago
Meanwhile Caddy exists
9dev 7mo ago
Paying for certificates..? Manually copying cert files? Man, this reads like it was 2010 or something. Best of luck, but I don’t know why I wouldn’t just use acme.sh and systemd timers instead of this.
lachiflippi 7mo ago
You're developing "certbot, but it's paid and sends private keys around the network instead of generating the csr locally"? Why? Who's the target audience? Platforms that can't run certbot, or any of the infinite amount of other acme clients, most likely won't be able to run your agent as well, so what's the value add vs just running a regular, well-defined (and free!) acme client and just moving the cert over manually?

PaulCarrack 7mo ago

I'm often stuck at my desk for long hours, which made it difficult to maintain a healthy weight. I started using Manjaro a few months ago, and it's had a significant impact. It helped control my appetite and reduced the constant cravings I'd usually get, making it easier to make healthier choices. I've lost around 15-20 lbs so far, and I feel much more in control of my eating habits. I know it's a bit anecdotal but it's been an important part of my weight loss journey

johnbarron 7mo ago

[dead]