13 ms·

OVH is down along with its status page

234 points by rgun 5y ago

codedeadlock 5y ago
What could be the best way for status page apart from third party services? I remember one project that used IPFS as distributed status page. https://news.ycombinator.com/item?id=16273609 https://news.ycombinator.com/item?id=16273609
- fragmede 5y ago
  That third party service has still got to be hosted somewhere though. What happens when the cloud they're running on goes down? The rule should be host your status page on your competitor's cloud. If you're AWS, host it in Azure, if you're Azure, host it in GCP, if you're GCP, host it in AWS. (Linode, Digital Ocean, OVH, etc can do their own dance.)
lerela 5y ago
Tough timing, considering their IPO is tomorrow [1]. [1]: https://ipo.ovhcloud.com/sites/default/files/2021-10/OVHcloud%20-%20Press%20Release%20-%20Oct%205%202021%20-%20OVHcloud%20launches%20its%20initial%20public%20offering%20on%20Euronext%20Paris%20and%20announces%20its%20indicative%20price%20range.pdf https://ipo.ovhcloud.com/sites/default/files/2021-10/OVHclou...
- SiempreViernes 5y ago
  So ovhcloud seems up, where's that hosted?
  - throwaway81523 5y ago
    Stuff is apparently starting to come back up. Mostly still inaccessible though.
- speedgoose 5y ago
  Wasn't the fire in one of the Strasbourg's datacenter the night after the announcement of the IPO ? They have bad luck.
  - cyanydeez 5y ago
    they also have garbage it support
    - malikNF 5y ago
      This is one of those things I keep hearing but in my case I experienced the opposite. I have been using OVH for a long time, and their paid support is actually really good. One of our projects used to get ddosed and as a small company the only reason we managed to survive was thanks to OVH. When the attackers kept changing their attacks, we were direct to talk to their engineers and they helped fix the issue, all we had to do was send a bunch of packet captures to them. Over the years since then we had a few issues and every single time their customer support was really helpful. I have used paid support from google, azure and amazon, but I have had a better customer service experience with OVH overall.
- oars 5y ago
  Pure coincidence.
  - jackhalford 5y ago
    not really, from Octave’s twitter it seems that DDoS attempts have increased which is usual right before an IPO. They were installing new gear because of the higher DDoS.
    - MrWiffles 5y ago
      Could you please elaborate on this? I’ve never heard of that being related to an IPO before, so I’m wondering if this is a standard practice for competitors to get known groups to do this sort of thing, or if this is an economy destabilization measure that’s in the playbook of some nation state actors. Any insight there?
      - tdeck 5y ago
        It's more likely to be a "DDoS until you pay a ransom" sort of deal where the attacker thinks the company will value uptime more highly during that period.
      - MrWiffles 5y ago
        *NATION state actors. Damn mobile keyboards. Stupid app wouldn’t let me edit the reply either, hence this.
        Symbiote 5y ago
        The term is state actor, anyway. A state is a sovereign political division, or a subdivision within a federation (France, Arizona). A nation is a community of people with a common culture, language, ethnicity etc (the Navaho, the Welsh, the Japanese). When these are the same entity, it's a nation state (Japan), but there are many cases where they aren't the same (Belgium, China, USA, Russia).
        5y ago
- jacquesm 5y ago
  If this is an unforced error that would be monumentally stupid.
  - mhaberl 5y ago
    > Those last few days, DDoS attacks intensity has grown a lot. not unforced, prolly ransom requests before IPO
    - jacquesm 5y ago
      Yes, that would be the expected thing. Around an IPO you should have your house in order otherwise you will get burned.
- HelloNurse 5y ago
  Maybe competitors doing their worst to cripple the IPO? Are hosting/CDN firms evil enough?
Shamanoid 5y ago
Our whole OVH infra is down, composed of OVH baremetal + SoYouStart an OVH sister company. http://travaux.ovh.net/ http://travaux.ovh.net/ is also down which is their WIP tasks dashboard, having worked there I can tell that something definitely went very very wrong
- Shamanoid 5y ago
  Update: We recovered our OVH sandbox env (yay) but prod is still very down, guess they are rolling back DC by DC
- Shamanoid 5y ago
  Update 2: Our call center provider has VoIP provided by OVH as well, they are down as well... It's going to be a very long day...
didymospl 5y ago
Our servers in Strasburg were down for around 10 minutes, now they are fully operational but it looks like some of their sites are still experiencing issues [1]. On the positive side: it was my second day as on-call and I already had the worst incident(no ssh, provider site is down). [1]: http://status.ovh.com/ http://status.ovh.com/
- nicolaslem 5y ago
  I don't think this counts as a worst incident. Since there is nothing you can do you might as well go take a nap and wait for it to pass. On the other hand something like Gitlab's "We deleted the production database and our backups were not working for months" sounds much more stressful to me.
joshxyz 5y ago
Internet hiccups
svdr 5y ago
Statuspage is down with https, but not without: http://status.ovh.com/ http://status.ovh.com/
blago 5y ago
Jitpack is down for me too. I wonder if this is related.
bluedino 5y ago
Died in a middle of a game of Lichess
sodality2 5y ago
This seems to have brought down Snapchat as well... (not the target audience particularly, here, but still relevant); didn't expect snapchat to rely on OVH at all, so it could be a coincidence.
- beta2k 5y ago
  i can confirm that this is not true. talked to an employee. snap hosts on google cloud.
  - beta2k 5y ago
    obviously I cant rule out that it's not related in some way or the other, but at least not directly.
protecto 5y ago
migadu.com is also down, connected? edit: also can't get access to ovh vps via ssh
- alibert 5y ago
  Migadu is indeed using OVH servers.
- chloratine 5y ago
  Lot of websites will probably be impacted
usrme 5y ago
Azure is also having issues with deploying and starting virtual machines. EDIT: To those having issues starting/deploying Windows virtual machines in Azure and are using ARM to do so: change the OS type to Linux instead of Windows. This seems to resolve the following issue: > Error: No version found in the artifact repository that satisfies the requested version '' for VM extension with publisher 'Microsoft.WindowsAzure.GuestAgent' and type 'CRPProd'
- cm2187 5y ago
  That may be an interesting effect. People test their failover solutions if one of their provider is down when the provider is not actually down. But when it is actually down, everyone else is executing their failover plan at the same time, and compete for other clouds resources.
  - qaq 5y ago
    Hence the BS of cloud marketing imagine AWS US-East going down there is no capacity to absorb that anywhere.
    - fragmede 5y ago
      Do you have any numbers/sources to back that up? I find that hard to believe, mainly because US-east1 goes down so much. It's never been hard-down across three AZs level of being down with no EC2 no VPC no nothing, but S3 being down in the whole region is pretty freaking close!
      - daviddever23box 5y ago
        AWS us-east-2 (Cincinnati) is generally pretty reliable; it pays to use your head and not simply use the defaults.
      - qaq 5y ago
        There was a leaked dock that detailed AWS datacenters a while back and US-east dwarfed everything else.
kilburn 5y ago
Their entire network is down. It seems connected to some routing "improvements" they had planned that went REALLY wrong: https://twitter.com/ovh_status/status/1448185498812485633 https://twitter.com/ovh_status/status/1448185498812485633 ... and yeah, lots of sites impacted
unilynx 5y ago
looks like funny routing.. packets to ovh.nl go to Asia, and then suddenly find a faster way back to me 3 0.ae21.xr4.1d12.xs4all.net (194.109.7.169) 7.324 ms 0.ae21.xr3.3d12.xs4all.net (194.109.7.173) 5.929 ms 0.ae21.xr4.1d12.xs4all.net (194.109.7.169) 5.864 ms 4 0.et-1-1-0.xr1.tc2.xs4all.net (194.109.5.7) 7.562 ms 5.487 ms 0.et-7-1-0.xr1.tc2.xs4all.net (194.109.5.5) 5.551 ms 5 asd-s8-rou-1041.nl.as286.net (134.222.94.216) 6.034 ms 5.759 ms 5.608 ms 6 ae11.cr6-ams1.ip4.gtt.net (213.200.117.178) 6.620 ms 7.938 ms 6.670 ms 7 80.231.85.162 (80.231.85.162) 7.348 ms 6.170 ms 6.320 ms 8 if-ae-45-2.tcore2.av2-amsterdam.as6453.net (80.231.152.50) 310.705 ms 257.678 ms 257.278 ms 9 if-ae-14-2.tcore2.l78-london.as6453.net (80.231.131.160) 257.217 ms 256.759 ms 259.044 ms 10 if-ae-2-2.tcore1.l78-london.as6453.net (80.231.131.2) 258.850 ms 256.723 ms 256.735 ms 11 if-ae-12-2.tcore2.mlv-mumbai.as6453.net (180.87.39.22) 257.349 ms 255.042 ms 257.338 ms 12 if-ae-16-2.tcore1.svw-singapore.as6453.net (180.87.12.226) 248.550 ms 257.559 ms 329.240 ms 13 if-ae-2-2.tcore2.svw-singapore.as6453.net (180.87.12.2) 307.149 ms 255.582 ms be101.mrs-mrs1-sbb1-nc5.fr.eu (54.36.50.135) 181.205 ms 14 be101.mrs-mrs2-sbb1-nc5.fr.eu (54.36.50.159) 180.612 ms * 179.738 ms 15 * * par-gsw-sbb1-nc5.fr.eu (54.36.50.228) 186.377 ms
- y4mi 5y ago
  Maybe they accidentally misconfigured the data siphon to the Chinese state to not only send duplicate packets but the originals too! (This is an unsubstantiated thesis that has no collaborating evidence)
  - sva_ 5y ago
    Singapore != China
    - b112 5y ago
      Have you asked China this question lately?
mjw1007 5y ago
Seems to be back 07:41 UTC. I became able to reach our servers in London, and www.ovh.com, from UK locations and from DigitalOcean.
- rgun 5y ago
  Still down from here
- ro_bit 5y ago
  Both my Montreal based servers are still down. Seems like their resellers are also, unsurprisingly, impacted
- mjw1007 5y ago
  2 10.80.71.15 (10.80.71.15) 1.095 ms 10.80.70.253 (10.80.70.253) 0.919 ms 10.80.71.15 (10.80.71.15) 1.067 ms 3 138.197.249.124 (138.197.249.124) 1.365 ms 138.197.249.102 (138.197.249.102) 1.354 ms 138.197.249.100 (138.197.249.100) 1.549 ms 4 138.197.251.140 (138.197.251.140) 1.179 ms 1.168 ms 138.197.251.136 (138.197.251.136) 1.114 ms 5 * 138.197.249.217 (138.197.249.217) 1.167 ms 1.155 ms 9 vl1272.lon1-eri1-d1-a75.uk.eu (213.186.32.253) 2.594 ms be101.lon1-eri1-g2-nc5.uk.eu (91.121.215.119) 2.654 ms 2.466 ms 14 (51.89.234.11) 2.102 ms 2.091 ms 2.076 ms
mikevm 5y ago
"No impact expected" - famous last words
d--b 5y ago
Ah, I was wondering why my scraper was failing a lot this morning. good to know
sschueller 5y ago
Let's hope it is not another container fire. [1] https://thestack.technology/ovhcloud-fire-strasbourg/ https://thestack.technology/ovhcloud-fire-strasbourg/
cute_boi 5y ago
probably bgp? It seems bgp is always the issue?
- jhugo 5y ago
  Well, that's a bit like saying SMTP or IMAP is always the issue when your email stops working. If they screwed up routing, BGP was very likely to be involved, but the cause was probably human error, failure of process design, failure of system design, etc etc.
raverbashing 5y ago
Let me guess: BGP? DNS?
bratch 5y ago
It looks like only IPv4 is affected. IPv6 is working fine to a couple of test endpoints. 3 2a02:c28:1:6506::106 2.131 ms 2.124 ms 2.117 ms 4 2a02:c28:11:6::100 15.695 ms 15.689 ms 15.682 ms 5 2a02:c28:1:1900::19 15.998 ms 15.991 ms 15.985 ms 6 2a02:c28:0:1819::18 15.259 ms 15.051 ms 16.975 ms 7 2a02:c28:0:1718::17 16.968 ms 16.403 ms 15.619 ms 8 2a02:c28:0:1731::31 15.406 ms 18.620 ms 18.554 ms 9 2001:7f8:4::3f94:2 12.025 ms * * 10 * * * 11 2001:41d0:aaaa:100::5 21.622 ms 38.813 ms 2001:41d0:aaaa:100::3 37.231 ms 12 * * * 13 * 2001:41d0::25f1 19.507 ms 2001:41d0::c68 20.892 ms 14 2001:41d0::513 19.685 ms 19.674 ms 2001:41d0::50d 18.248 ms 15 2001:41d0:0:50::5:10a1 19.092 ms 19.917 ms 2001:41d0:0:50::5:10a5 20.126 ms 16 2001:41d0:0:50::1:143f 19.375 ms 2001:41d0:0:50::1:143b 24.609 ms 2001:41d0:0:50::1:143d 18.933 ms
- gorgoiler 5y ago
  Ugh. You reminded me that while OVH used to hand out a /128, if you needed more address space you had to pay more for “enterprise” IPv6 or something.
  - megous 5y ago
    Is their enterprise prefix /64? That would be even funnier. :)
  - minaguib 5y ago
    lolwut ? Don't consumer ISPs hand out /64s or /48s ?
    - jaywalk 5y ago
      Yeah, Comcast hands out /64s on residential connections. /128 is literally just a single address, which is absurd if true.
      - zinekeller 5y ago
        Unfortunately, it was(is?) true. Not just OVH though, even Scaleway also had these silly /128 shenanigans.
vadfa 5y ago
When you use OVH you have to be ready for this kind of stuff. They are cheap for a reason.
- rgun 5y ago
  Well, even if you don't use OVH you should be ready for this kind of stuff.
- hdjjhhvvhga 5y ago
  Hetzner is also cheap but for some reason they don't seem to suffer from these kinds of issues.
  - vadfa 5y ago
    Well they are not French.
    - bsaul 5y ago
      i guess facebook must be french then, judging by what happened last week.
      - vadfa 5y ago
        Facebook does not go down several times a year and their datacenters don't literally burn down because of how cheap and useless they are.
        drdebug 5y ago
        Facebook once let it rain in their data centers, https://www.theregister.com/2013/06/08/facebook_cloud_versus_cloud/ https://www.theregister.com/2013/06/08/facebook_cloud_versus... and they actually went down at least twice during the last year https://www.cnbc.com/2021/10/05/facebook-says-sorry-for-mass-outage-and-reveals-why-it-happened.html https://www.cnbc.com/2021/10/05/facebook-says-sorry-for-mass... https://gadgets.ndtv.com/apps/news/facebook-messenger-instagram-whatsapp-down-outage-2336939 https://gadgets.ndtv.com/apps/news/facebook-messenger-instag...
ryanmjacobs 5y ago
https://pbs.twimg.com/media/FBkA6ZrWEAA9olm?format=jpg&name=large https://pbs.twimg.com/media/FBkA6ZrWEAA9olm?format=jpg&name=...
gadrev 5y ago
I was just going for a peaceful morning walk, trying to start the day with some sunlight. As soon as I sit on the first bench with sun I can find, the dreaded message comes: "everything is down". Thank you! I had almost forgotten about my beloved provider since the last outage :)
- hinkley 5y ago
  One more and you get to make a sign. It has been [17] days since the last outage.
crypt1d 5y ago
LOL. OVH has turned into the laughing stock of the industry. Its a shame really, I always used to gravitate towards their offerings vs competition because I liked the fact that they have servers literally everywhere...but what good are servers everywhere when u get a major outage every few months? Perhaps I'd tolerate the constant outages if they were actually any good at communicating. Their status page sucks. They rarely update it properly, and when they do, its hard to find the information you need and that you know relates to your servers.
- encryptluks2 5y ago
  If you have to deal with a few outages for a 90% cost reduction, then you just make sure you have a plan. I've found OVH to be mostly reliable.
  - wg0 5y ago
    Things do happen even with AWS. Minus the fire incident, can't recall any outage. Not the cheapest offering out there but been running many things on it on autopilot. Support is solid too, always had a good experience.
- sdevonoes 5y ago
  Well, some products/businesses don't need 99.999% uptime and that's fine. Hell, I'm working on a small shop and we had to update some service that is not used constantly by our customers and when I suggested to have a downtime of 5 minutes everybody looked at me as if I were a caveman. The other alternative was to come up with some custom strategy to have two instances of the same service running at the same time and kill one of them when we know the other is healthy. Now, for a service that is constantly used that's a good idea, but for our scenario a 5 min. downtime is fine... we are not Google.
- acatton 5y ago
  I'm not affiliated with them in any way, but historically, Online.net (now rebranded as "Scaleway") has always been the main competitor of OVH in France. They were known for their "Dedibox" when I was younger. OVH kinda won because their VPS pricing was always dirt-cheap compared to the cheap dedicated servers provided by Online.net/Scaleway. Since then OVH VPSes got expensive, and Scaleway started offering cheap VPSes, the prices got aligned. I know a few people who used to be OVH-fanboys and they all migrated to Scaleway and Vultr the last 4 years. But I agree, the competition for unmetered reliable VPSes is getting thinner and thinner… People are even started to consider metered VPS solutions like Vultr… Rasmus Lerdorf (the author of PHP) did an objective non-sponsored comparison[1] in 2019, I'm using IONOS personally, but since then strato.de appeared. (Both are German) I'm thinking about adding another node from them to avoid relying on only one vendor. [1] https://toys.lerdorf.com/low-cost-vps-testing https://toys.lerdorf.com/low-cost-vps-testing
throwaway81523 5y ago
Router maintenance, no impact predicted. Many people saying that if "all servers down everywhere" is not impact, they wonder what actual impact is. https://twitter.com/ovh_status/status/1448185498812485633 https://twitter.com/ovh_status/status/1448185498812485633
- tdeck 5y ago
  Which is stupid. If I say I'm walking to the store and will be gone for 30 minutes, then get hit by a truck and spend a week in the hospital, nobody would say "30 minutes, huh?"
multun 5y ago
It seems like (only?) their name servers are down. I got my website back up by avoiding to use these (it's hosted on a dedicated server at OVH). They'll probably have the same kind of issues as facebook did while trying to get things back up.
benoitg 5y ago
They just posted an update : https://twitter.com/olesovhcom/status/1448196879020433409 https://twitter.com/olesovhcom/status/1448196879020433409 Following a human error during the reconfiguration of the network on our DC to VH (US-EST), we have a problem on the entire backbone. We will isolate the DC VH and then fix the conf.
- deleted 5y ago
  [deleted]
amanzi 5y ago
Mxroute is also down as a result of this outage.
nowahe 5y ago
Octave Klaba (OVH CEO) on Twitter [0] : Following a human error during the reconfiguration of the network in our DC in VH (US-EST), we had an issue on all the backbone. We're gonna isolate the DC VH then fix the configuration Those last few days, DDoS attacks intensity has grown a lot. We decided to increase our DDoS handling capacity by adding new infrastructures in our DC VH (US-EST). A bad router configuration caused the network crash. [0]: https://twitter.com/olesovhcom/status/1448196879020433409 https://twitter.com/olesovhcom/status/1448196879020433409
- deleted 5y ago
  [deleted]
- rahen 5y ago
  It's now back: https://status.ovh.com https://status.ovh.com Short lived incident fortunately...
  - Kelteseth 5y ago
    Does not work for me
    - rahen 5y ago
      Apparently it's only the status page. My mail servers have been down for just about a hour.
  - sbierwagen 5y ago
    Down here. Comcast west coast.
    - rgun 5y ago
      Down for me too. Although, seems like this time only the status page is down.
  - tonylemesmer 5y ago
    down from the UK
deleted 5y ago
[deleted]
rbut 5y ago
Back up for our servers in Australia. Edit: Our US server also. Total downtime around 50minutes.
rgun 5y ago
It is up here at 08:27 am UTC
gadrev 5y ago
It seems to be back up, I can access the Control Panel and my servers. So around 1 hour downtime for me, 7:20 to 8:20 UTC.
areski 5y ago
It's back! http://status.ovh.es/ http://status.ovh.es/
WesolyKubeczek 5y ago
Uh-oh, seems like greybeards who know how to do routing properly are dying off, and the young generation is not up to snuff. Also, people who know the number 1 principle of OOB networking. These are challenging times indeed.
- rand49an 5y ago
  We don't know the ages of the people involved with this incident, so this just comes off as gatekeeping.
  - WesolyKubeczek 5y ago
    Not gatekeeping, rather lament that the next generation doesn't tend to dive deeper into the nuts and bolts of how those things actually work and what the failure modes are. Maybe the greybeards don't do a good job at educating their successors, or maybe it's the "tl;dr, YOLO" attitude at work. I don't know. I suspect a combination of both.
- ewidar 5y ago
  Yes, because no network issue ever happened in the past with the "greybeards". Do we really need these kind of comments?
  - WesolyKubeczek 5y ago
    > Yes, because no network issue ever happened in the past with the "greybeards". These issues, one might think, should have turned into tomes of extensive information on why they happened and how to avoid them, and become an integral part of showing the ropes to new sysadmins, or operation persons, you name them. It seems, however, that by and large the knowledge of the actual working systems does get irretrievably lost once some Kevin gets retired. P. S. Would you be just as cavalier if lives were lost as a result of such incidents?
Octabrain 5y ago
I don’t know how is OVH doing on average nowadays in relation to downtimes, but I worked for a company around 8 years ago that had part of its infrastructure running on OVH. That thing was a sad joke and we ended up migrating everything to AWS. Pretty much every month we had issues with them.
- whitepoplar 5y ago
  What sorts of issues?
  - Octabrain 5y ago
    Downtimes due to incidents on their data centres. We used to joke saying things like “oh, that must be the OVH guys having some fun on the datacenter again”.
rozenmd 5y ago
As someone that runs an uptime monitoring service, it was wild to see so many sites go down that I wondered whether it was my monitoring that was screwing up!
- 1cvmask 5y ago
  Where do you host your monitoring service. And how do you maintain uptime?
  - rozenmd 5y ago
    AWS, and the monitors themselves are geographically redundant. So if ap-southeast-2 goes down, it uses another region like ap-southeast-1. The web app that configures the monitors is deployed separately, so the web app can go down while the monitors + alerting keeps running.
    - 1cvmask 5y ago
      So it is solely running on AWS? Or are there other clouds as well?
      - rozenmd 5y ago
        Solely on AWS for now.
curiousgal 5y ago
Does anyone have an alternative for a hobbyist VPS provider (around 5€/month) ? OVH keeps reducing their offering and bumping their prices. Plus they dropped support for Arch images.
- adevx 5y ago
  Good experience with Hetzner Cloud, they support Arch. Only EU data centers though.
- draugadrotten 5y ago
  You can often find deals on https://lowendbox.com/ https://lowendbox.com/
- reactor 5y ago
  https://www.ssdnodes.com/ https://www.ssdnodes.com/ Quite attractive pricing and been an user for about 2+ yrs, never faced any issue so far, probably luck me?
  - PixyMisa 5y ago
    I've been using SSDNodes for some stuff for the past year. Had an issue with my VPS in Dallas, contacted support, they fixed it right away - response in a few minutes, fixed within half an hour.
- stillblue 5y ago
  Linode and Digital Ocean are pretty good.
  - deleted 5y ago
    [deleted]
- dlkmp 5y ago
  I use hetzner, their smallest VPS is around 2,xx Euro. Has been very reliable for me in the last couple of years. I'm not sure whether they support arch, though.
janmo 5y ago
OVH has many issues. One of their four data center burned down in march. A few days ago they sent me an email that they were deleting my account soon, and a few hours later another E-Mail stating that it was an error. Now this. Tomorrow is their IPO, it will be at a very high valuation, lets see how that plays out.
- sofixa 5y ago
  One of their four datacenters in Strasbourg. They have many others accross the world.
mst 5y ago
We had a couple network blips but all our systems seem basically fine now. The status page is still hosed, but I'll take "my systems are up but the status page is down" over the converse any day ;)
mdiesel 5y ago
What's the rough downtime an email server can handle without losing any emails? The greylist filter relies on most people's emails resending after a few mins if the first is rejected, so guessing a short enough downtime has no impact?
- LinuxBender 5y ago
  Defaults depend on the application and actual settings depend on the admin/postmaster. So retry queues will typically be somewhere between 3 to 7 days, depending. High volume sending servers may be shorter. It is rare for short outages to cause emails to be lost unless the outage is due to a fire and the server holding your message melted.
block_dagger 5y ago
Prior to the outage I received an erroneous email from them about an old account, followed by an apology about it. It seems they were having other issues than just the router config.
itaibo 5y ago
You can follow the status page here: http://travaux.ovh.net http://travaux.ovh.net
qeternity 5y ago
Has OVH gotten substantially worse in the past couple of years? We ran a decent cluster in GRA-3 for a few years, and I can't recall having any real downtime. But lots of comments here lead me to believe that other people have very different experiences with OVH.
champtar 5y ago
Here the bad copy/paste https://mobile.twitter.com/fred21946537/status/1448219047494098944 https://mobile.twitter.com/fred21946537/status/1448219047494...

234 points by rgun 5y ago

What could be the best way for status page apart from third party services? I remember one project that used IPFS as distributed status page. https://news.ycombinator.com/item?id=16273609 https://news.ycombinator.com/item?id=16273609

fragmede 5y ago
That third party service has still got to be hosted somewhere though. What happens when the cloud they're running on goes down? The rule should be host your status page on your competitor's cloud. If you're AWS, host it in Azure, if you're Azure, host it in GCP, if you're GCP, host it in AWS. (Linode, Digital Ocean, OVH, etc can do their own dance.)

lerela 5y ago

Tough timing, considering their IPO is tomorrow [1]. [1]: https://ipo.ovhcloud.com/sites/default/files/2021-10/OVHcloud%20-%20Press%20Release%20-%20Oct%205%202021%20-%20OVHcloud%20launches%20its%20initial%20public%20offering%20on%20Euronext%20Paris%20and%20announces%20its%20indicative%20price%20range.pdf https://ipo.ovhcloud.com/sites/default/files/2021-10/OVHclou...

SiempreViernes 5y ago
So ovhcloud seems up, where's that hosted?
- throwaway81523 5y ago
  Stuff is apparently starting to come back up. Mostly still inaccessible though.
speedgoose 5y ago
Wasn't the fire in one of the Strasbourg's datacenter the night after the announcement of the IPO ? They have bad luck.
- cyanydeez 5y ago
  they also have garbage it support
  - malikNF 5y ago
    This is one of those things I keep hearing but in my case I experienced the opposite. I have been using OVH for a long time, and their paid support is actually really good. One of our projects used to get ddosed and as a small company the only reason we managed to survive was thanks to OVH. When the attackers kept changing their attacks, we were direct to talk to their engineers and they helped fix the issue, all we had to do was send a bunch of packet captures to them. Over the years since then we had a few issues and every single time their customer support was really helpful. I have used paid support from google, azure and amazon, but I have had a better customer service experience with OVH overall.
oars 5y ago
Pure coincidence.
- jackhalford 5y ago
  not really, from Octave’s twitter it seems that DDoS attempts have increased which is usual right before an IPO. They were installing new gear because of the higher DDoS.
  - MrWiffles 5y ago
    Could you please elaborate on this? I’ve never heard of that being related to an IPO before, so I’m wondering if this is a standard practice for competitors to get known groups to do this sort of thing, or if this is an economy destabilization measure that’s in the playbook of some nation state actors. Any insight there?
    - tdeck 5y ago
      It's more likely to be a "DDoS until you pay a ransom" sort of deal where the attacker thinks the company will value uptime more highly during that period.
    - MrWiffles 5y ago
      *NATION state actors. Damn mobile keyboards. Stupid app wouldn’t let me edit the reply either, hence this.
      - Symbiote 5y ago
        The term is state actor, anyway. A state is a sovereign political division, or a subdivision within a federation (France, Arizona). A nation is a community of people with a common culture, language, ethnicity etc (the Navaho, the Welsh, the Japanese). When these are the same entity, it's a nation state (Japan), but there are many cases where they aren't the same (Belgium, China, USA, Russia).
        5y ago
jacquesm 5y ago
If this is an unforced error that would be monumentally stupid.
- mhaberl 5y ago
  > Those last few days, DDoS attacks intensity has grown a lot. not unforced, prolly ransom requests before IPO
  - jacquesm 5y ago
    Yes, that would be the expected thing. Around an IPO you should have your house in order otherwise you will get burned.
HelloNurse 5y ago
Maybe competitors doing their worst to cripple the IPO? Are hosting/CDN firms evil enough?

Shamanoid 5y ago

Our whole OVH infra is down, composed of OVH baremetal + SoYouStart an OVH sister company. http://travaux.ovh.net/ http://travaux.ovh.net/ is also down which is their WIP tasks dashboard, having worked there I can tell that something definitely went very very wrong

Shamanoid 5y ago
Update: We recovered our OVH sandbox env (yay) but prod is still very down, guess they are rolling back DC by DC
Shamanoid 5y ago
Update 2: Our call center provider has VoIP provided by OVH as well, they are down as well... It's going to be a very long day...

didymospl 5y ago

Our servers in Strasburg were down for around 10 minutes, now they are fully operational but it looks like some of their sites are still experiencing issues [1]. On the positive side: it was my second day as on-call and I already had the worst incident(no ssh, provider site is down). [1]: http://status.ovh.com/ http://status.ovh.com/

nicolaslem 5y ago
I don't think this counts as a worst incident. Since there is nothing you can do you might as well go take a nap and wait for it to pass. On the other hand something like Gitlab's "We deleted the production database and our backups were not working for months" sounds much more stressful to me.

joshxyz 5y ago

Internet hiccups

svdr 5y ago

Statuspage is down with https, but not without: http://status.ovh.com/ http://status.ovh.com/

blago 5y ago

Jitpack is down for me too. I wonder if this is related.

bluedino 5y ago

Died in a middle of a game of Lichess

sodality2 5y ago

This seems to have brought down Snapchat as well... (not the target audience particularly, here, but still relevant); didn't expect snapchat to rely on OVH at all, so it could be a coincidence.

beta2k 5y ago
i can confirm that this is not true. talked to an employee. snap hosts on google cloud.
- beta2k 5y ago
  obviously I cant rule out that it's not related in some way or the other, but at least not directly.

protecto 5y ago

migadu.com is also down, connected? edit: also can't get access to ovh vps via ssh

alibert 5y ago
Migadu is indeed using OVH servers.
chloratine 5y ago
Lot of websites will probably be impacted

usrme 5y ago

Azure is also having issues with deploying and starting virtual machines. EDIT: To those having issues starting/deploying Windows virtual machines in Azure and are using ARM to do so: change the OS type to Linux instead of Windows. This seems to resolve the following issue: > Error: No version found in the artifact repository that satisfies the requested version '' for VM extension with publisher 'Microsoft.WindowsAzure.GuestAgent' and type 'CRPProd'

cm2187 5y ago
That may be an interesting effect. People test their failover solutions if one of their provider is down when the provider is not actually down. But when it is actually down, everyone else is executing their failover plan at the same time, and compete for other clouds resources.
- qaq 5y ago
  Hence the BS of cloud marketing imagine AWS US-East going down there is no capacity to absorb that anywhere.
  - fragmede 5y ago
    Do you have any numbers/sources to back that up? I find that hard to believe, mainly because US-east1 goes down so much. It's never been hard-down across three AZs level of being down with no EC2 no VPC no nothing, but S3 being down in the whole region is pretty freaking close!
    - daviddever23box 5y ago
      AWS us-east-2 (Cincinnati) is generally pretty reliable; it pays to use your head and not simply use the defaults.
    - qaq 5y ago
      There was a leaked dock that detailed AWS datacenters a while back and US-east dwarfed everything else.

kilburn 5y ago

Their entire network is down. It seems connected to some routing "improvements" they had planned that went REALLY wrong: https://twitter.com/ovh_status/status/1448185498812485633 https://twitter.com/ovh_status/status/1448185498812485633 ... and yeah, lots of sites impacted

unilynx 5y ago

looks like funny routing.. packets to ovh.nl go to Asia, and then suddenly find a faster way back to me 3 0.ae21.xr4.1d12.xs4all.net (194.109.7.169) 7.324 ms 0.ae21.xr3.3d12.xs4all.net (194.109.7.173) 5.929 ms 0.ae21.xr4.1d12.xs4all.net (194.109.7.169) 5.864 ms 4 0.et-1-1-0.xr1.tc2.xs4all.net (194.109.5.7) 7.562 ms 5.487 ms 0.et-7-1-0.xr1.tc2.xs4all.net (194.109.5.5) 5.551 ms 5 asd-s8-rou-1041.nl.as286.net (134.222.94.216) 6.034 ms 5.759 ms 5.608 ms 6 ae11.cr6-ams1.ip4.gtt.net (213.200.117.178) 6.620 ms 7.938 ms 6.670 ms 7 80.231.85.162 (80.231.85.162) 7.348 ms 6.170 ms 6.320 ms 8 if-ae-45-2.tcore2.av2-amsterdam.as6453.net (80.231.152.50) 310.705 ms 257.678 ms 257.278 ms 9 if-ae-14-2.tcore2.l78-london.as6453.net (80.231.131.160) 257.217 ms 256.759 ms 259.044 ms 10 if-ae-2-2.tcore1.l78-london.as6453.net (80.231.131.2) 258.850 ms 256.723 ms 256.735 ms 11 if-ae-12-2.tcore2.mlv-mumbai.as6453.net (180.87.39.22) 257.349 ms 255.042 ms 257.338 ms 12 if-ae-16-2.tcore1.svw-singapore.as6453.net (180.87.12.226) 248.550 ms 257.559 ms 329.240 ms 13 if-ae-2-2.tcore2.svw-singapore.as6453.net (180.87.12.2) 307.149 ms 255.582 ms be101.mrs-mrs1-sbb1-nc5.fr.eu (54.36.50.135) 181.205 ms 14 be101.mrs-mrs2-sbb1-nc5.fr.eu (54.36.50.159) 180.612 ms * 179.738 ms 15 * * par-gsw-sbb1-nc5.fr.eu (54.36.50.228) 186.377 ms

y4mi 5y ago
Maybe they accidentally misconfigured the data siphon to the Chinese state to not only send duplicate packets but the originals too! (This is an unsubstantiated thesis that has no collaborating evidence)
- sva_ 5y ago
  Singapore != China
  - b112 5y ago
    Have you asked China this question lately?

mjw1007 5y ago

Seems to be back 07:41 UTC. I became able to reach our servers in London, and www.ovh.com, from UK locations and from DigitalOcean.

rgun 5y ago
Still down from here
ro_bit 5y ago
Both my Montreal based servers are still down. Seems like their resellers are also, unsurprisingly, impacted
mjw1007 5y ago
2 10.80.71.15 (10.80.71.15) 1.095 ms 10.80.70.253 (10.80.70.253) 0.919 ms 10.80.71.15 (10.80.71.15) 1.067 ms 3 138.197.249.124 (138.197.249.124) 1.365 ms 138.197.249.102 (138.197.249.102) 1.354 ms 138.197.249.100 (138.197.249.100) 1.549 ms 4 138.197.251.140 (138.197.251.140) 1.179 ms 1.168 ms 138.197.251.136 (138.197.251.136) 1.114 ms 5 * 138.197.249.217 (138.197.249.217) 1.167 ms 1.155 ms 9 vl1272.lon1-eri1-d1-a75.uk.eu (213.186.32.253) 2.594 ms be101.lon1-eri1-g2-nc5.uk.eu (91.121.215.119) 2.654 ms 2.466 ms 14 (51.89.234.11) 2.102 ms 2.091 ms 2.076 ms

mikevm 5y ago

"No impact expected" - famous last words

d--b 5y ago

Ah, I was wondering why my scraper was failing a lot this morning. good to know

sschueller 5y ago

Let's hope it is not another container fire. [1] https://thestack.technology/ovhcloud-fire-strasbourg/ https://thestack.technology/ovhcloud-fire-strasbourg/

cute_boi 5y ago

probably bgp? It seems bgp is always the issue?

jhugo 5y ago
Well, that's a bit like saying SMTP or IMAP is always the issue when your email stops working. If they screwed up routing, BGP was very likely to be involved, but the cause was probably human error, failure of process design, failure of system design, etc etc.

raverbashing 5y ago

Let me guess: BGP? DNS?

bratch 5y ago

It looks like only IPv4 is affected. IPv6 is working fine to a couple of test endpoints. 3 2a02:c28:1:6506::106 2.131 ms 2.124 ms 2.117 ms 4 2a02:c28:11:6::100 15.695 ms 15.689 ms 15.682 ms 5 2a02:c28:1:1900::19 15.998 ms 15.991 ms 15.985 ms 6 2a02:c28:0:1819::18 15.259 ms 15.051 ms 16.975 ms 7 2a02:c28:0:1718::17 16.968 ms 16.403 ms 15.619 ms 8 2a02:c28:0:1731::31 15.406 ms 18.620 ms 18.554 ms 9 2001:7f8:4::3f94:2 12.025 ms * * 10 * * * 11 2001:41d0:aaaa:100::5 21.622 ms 38.813 ms 2001:41d0:aaaa:100::3 37.231 ms 12 * * * 13 * 2001:41d0::25f1 19.507 ms 2001:41d0::c68 20.892 ms 14 2001:41d0::513 19.685 ms 19.674 ms 2001:41d0::50d 18.248 ms 15 2001:41d0:0:50::5:10a1 19.092 ms 19.917 ms 2001:41d0:0:50::5:10a5 20.126 ms 16 2001:41d0:0:50::1:143f 19.375 ms 2001:41d0:0:50::1:143b 24.609 ms 2001:41d0:0:50::1:143d 18.933 ms

gorgoiler 5y ago
Ugh. You reminded me that while OVH used to hand out a /128, if you needed more address space you had to pay more for “enterprise” IPv6 or something.
- megous 5y ago
  Is their enterprise prefix /64? That would be even funnier. :)
- minaguib 5y ago
  lolwut ? Don't consumer ISPs hand out /64s or /48s ?
  - jaywalk 5y ago
    Yeah, Comcast hands out /64s on residential connections. /128 is literally just a single address, which is absurd if true.
    - zinekeller 5y ago
      Unfortunately, it was(is?) true. Not just OVH though, even Scaleway also had these silly /128 shenanigans.

vadfa 5y ago

When you use OVH you have to be ready for this kind of stuff. They are cheap for a reason.

rgun 5y ago
Well, even if you don't use OVH you should be ready for this kind of stuff.
hdjjhhvvhga 5y ago
Hetzner is also cheap but for some reason they don't seem to suffer from these kinds of issues.
- vadfa 5y ago
  Well they are not French.
  - bsaul 5y ago
    i guess facebook must be french then, judging by what happened last week.
    - vadfa 5y ago
      Facebook does not go down several times a year and their datacenters don't literally burn down because of how cheap and useless they are.
      - drdebug 5y ago
        Facebook once let it rain in their data centers, https://www.theregister.com/2013/06/08/facebook_cloud_versus_cloud/ https://www.theregister.com/2013/06/08/facebook_cloud_versus... and they actually went down at least twice during the last year https://www.cnbc.com/2021/10/05/facebook-says-sorry-for-mass-outage-and-reveals-why-it-happened.html https://www.cnbc.com/2021/10/05/facebook-says-sorry-for-mass... https://gadgets.ndtv.com/apps/news/facebook-messenger-instagram-whatsapp-down-outage-2336939 https://gadgets.ndtv.com/apps/news/facebook-messenger-instag...

ryanmjacobs 5y ago

https://pbs.twimg.com/media/FBkA6ZrWEAA9olm?format=jpg&name=large https://pbs.twimg.com/media/FBkA6ZrWEAA9olm?format=jpg&name=...

gadrev 5y ago

I was just going for a peaceful morning walk, trying to start the day with some sunlight. As soon as I sit on the first bench with sun I can find, the dreaded message comes: "everything is down". Thank you! I had almost forgotten about my beloved provider since the last outage :)

hinkley 5y ago
One more and you get to make a sign. It has been [17] days since the last outage.

crypt1d 5y ago

LOL. OVH has turned into the laughing stock of the industry. Its a shame really, I always used to gravitate towards their offerings vs competition because I liked the fact that they have servers literally everywhere...but what good are servers everywhere when u get a major outage every few months? Perhaps I'd tolerate the constant outages if they were actually any good at communicating. Their status page sucks. They rarely update it properly, and when they do, its hard to find the information you need and that you know relates to your servers.

encryptluks2 5y ago
If you have to deal with a few outages for a 90% cost reduction, then you just make sure you have a plan. I've found OVH to be mostly reliable.
- wg0 5y ago
  Things do happen even with AWS. Minus the fire incident, can't recall any outage. Not the cheapest offering out there but been running many things on it on autopilot. Support is solid too, always had a good experience.
sdevonoes 5y ago
Well, some products/businesses don't need 99.999% uptime and that's fine. Hell, I'm working on a small shop and we had to update some service that is not used constantly by our customers and when I suggested to have a downtime of 5 minutes everybody looked at me as if I were a caveman. The other alternative was to come up with some custom strategy to have two instances of the same service running at the same time and kill one of them when we know the other is healthy. Now, for a service that is constantly used that's a good idea, but for our scenario a 5 min. downtime is fine... we are not Google.
acatton 5y ago
I'm not affiliated with them in any way, but historically, Online.net (now rebranded as "Scaleway") has always been the main competitor of OVH in France. They were known for their "Dedibox" when I was younger. OVH kinda won because their VPS pricing was always dirt-cheap compared to the cheap dedicated servers provided by Online.net/Scaleway. Since then OVH VPSes got expensive, and Scaleway started offering cheap VPSes, the prices got aligned. I know a few people who used to be OVH-fanboys and they all migrated to Scaleway and Vultr the last 4 years. But I agree, the competition for unmetered reliable VPSes is getting thinner and thinner… People are even started to consider metered VPS solutions like Vultr… Rasmus Lerdorf (the author of PHP) did an objective non-sponsored comparison[1] in 2019, I'm using IONOS personally, but since then strato.de appeared. (Both are German) I'm thinking about adding another node from them to avoid relying on only one vendor. [1] https://toys.lerdorf.com/low-cost-vps-testing https://toys.lerdorf.com/low-cost-vps-testing

throwaway81523 5y ago

Router maintenance, no impact predicted. Many people saying that if "all servers down everywhere" is not impact, they wonder what actual impact is. https://twitter.com/ovh_status/status/1448185498812485633 https://twitter.com/ovh_status/status/1448185498812485633

tdeck 5y ago
Which is stupid. If I say I'm walking to the store and will be gone for 30 minutes, then get hit by a truck and spend a week in the hospital, nobody would say "30 minutes, huh?"

multun 5y ago

It seems like (only?) their name servers are down. I got my website back up by avoiding to use these (it's hosted on a dedicated server at OVH). They'll probably have the same kind of issues as facebook did while trying to get things back up.

benoitg 5y ago

They just posted an update : https://twitter.com/olesovhcom/status/1448196879020433409 https://twitter.com/olesovhcom/status/1448196879020433409 Following a human error during the reconfiguration of the network on our DC to VH (US-EST), we have a problem on the entire backbone. We will isolate the DC VH and then fix the conf.

deleted 5y ago
[deleted]

amanzi 5y ago

Mxroute is also down as a result of this outage.

nowahe 5y ago

Octave Klaba (OVH CEO) on Twitter [0] : Following a human error during the reconfiguration of the network in our DC in VH (US-EST), we had an issue on all the backbone. We're gonna isolate the DC VH then fix the configuration Those last few days, DDoS attacks intensity has grown a lot. We decided to increase our DDoS handling capacity by adding new infrastructures in our DC VH (US-EST). A bad router configuration caused the network crash. [0]: https://twitter.com/olesovhcom/status/1448196879020433409 https://twitter.com/olesovhcom/status/1448196879020433409

deleted 5y ago
[deleted]
rahen 5y ago
It's now back: https://status.ovh.com https://status.ovh.com Short lived incident fortunately...
- Kelteseth 5y ago
  Does not work for me
  - rahen 5y ago
    Apparently it's only the status page. My mail servers have been down for just about a hour.
- sbierwagen 5y ago
  Down here. Comcast west coast.
  - rgun 5y ago
    Down for me too. Although, seems like this time only the status page is down.
- tonylemesmer 5y ago
  down from the UK

deleted 5y ago

[deleted]

rbut 5y ago

Back up for our servers in Australia. Edit: Our US server also. Total downtime around 50minutes.

rgun 5y ago

It is up here at 08:27 am UTC

gadrev 5y ago

It seems to be back up, I can access the Control Panel and my servers. So around 1 hour downtime for me, 7:20 to 8:20 UTC.

areski 5y ago

It's back! http://status.ovh.es/ http://status.ovh.es/

WesolyKubeczek 5y ago

Uh-oh, seems like greybeards who know how to do routing properly are dying off, and the young generation is not up to snuff. Also, people who know the number 1 principle of OOB networking. These are challenging times indeed.

rand49an 5y ago
We don't know the ages of the people involved with this incident, so this just comes off as gatekeeping.
- WesolyKubeczek 5y ago
  Not gatekeeping, rather lament that the next generation doesn't tend to dive deeper into the nuts and bolts of how those things actually work and what the failure modes are. Maybe the greybeards don't do a good job at educating their successors, or maybe it's the "tl;dr, YOLO" attitude at work. I don't know. I suspect a combination of both.
ewidar 5y ago
Yes, because no network issue ever happened in the past with the "greybeards". Do we really need these kind of comments?
- WesolyKubeczek 5y ago
  > Yes, because no network issue ever happened in the past with the "greybeards". These issues, one might think, should have turned into tomes of extensive information on why they happened and how to avoid them, and become an integral part of showing the ropes to new sysadmins, or operation persons, you name them. It seems, however, that by and large the knowledge of the actual working systems does get irretrievably lost once some Kevin gets retired. P. S. Would you be just as cavalier if lives were lost as a result of such incidents?

Octabrain 5y ago

I don’t know how is OVH doing on average nowadays in relation to downtimes, but I worked for a company around 8 years ago that had part of its infrastructure running on OVH. That thing was a sad joke and we ended up migrating everything to AWS. Pretty much every month we had issues with them.

whitepoplar 5y ago
What sorts of issues?
- Octabrain 5y ago
  Downtimes due to incidents on their data centres. We used to joke saying things like “oh, that must be the OVH guys having some fun on the datacenter again”.

rozenmd 5y ago

As someone that runs an uptime monitoring service, it was wild to see so many sites go down that I wondered whether it was my monitoring that was screwing up!

1cvmask 5y ago
Where do you host your monitoring service. And how do you maintain uptime?
- rozenmd 5y ago
  AWS, and the monitors themselves are geographically redundant. So if ap-southeast-2 goes down, it uses another region like ap-southeast-1. The web app that configures the monitors is deployed separately, so the web app can go down while the monitors + alerting keeps running.
  - 1cvmask 5y ago
    So it is solely running on AWS? Or are there other clouds as well?
    - rozenmd 5y ago
      Solely on AWS for now.

curiousgal 5y ago

Does anyone have an alternative for a hobbyist VPS provider (around 5€/month) ? OVH keeps reducing their offering and bumping their prices. Plus they dropped support for Arch images.

adevx 5y ago
Good experience with Hetzner Cloud, they support Arch. Only EU data centers though.
draugadrotten 5y ago
You can often find deals on https://lowendbox.com/ https://lowendbox.com/
reactor 5y ago
https://www.ssdnodes.com/ https://www.ssdnodes.com/ Quite attractive pricing and been an user for about 2+ yrs, never faced any issue so far, probably luck me?
- PixyMisa 5y ago
  I've been using SSDNodes for some stuff for the past year. Had an issue with my VPS in Dallas, contacted support, they fixed it right away - response in a few minutes, fixed within half an hour.
stillblue 5y ago
Linode and Digital Ocean are pretty good.
- deleted 5y ago
  [deleted]
dlkmp 5y ago
I use hetzner, their smallest VPS is around 2,xx Euro. Has been very reliable for me in the last couple of years. I'm not sure whether they support arch, though.

janmo 5y ago

OVH has many issues. One of their four data center burned down in march. A few days ago they sent me an email that they were deleting my account soon, and a few hours later another E-Mail stating that it was an error. Now this. Tomorrow is their IPO, it will be at a very high valuation, lets see how that plays out.

sofixa 5y ago
One of their four datacenters in Strasbourg. They have many others accross the world.

mst 5y ago

We had a couple network blips but all our systems seem basically fine now. The status page is still hosed, but I'll take "my systems are up but the status page is down" over the converse any day ;)

mdiesel 5y ago

What's the rough downtime an email server can handle without losing any emails? The greylist filter relies on most people's emails resending after a few mins if the first is rejected, so guessing a short enough downtime has no impact?

LinuxBender 5y ago
Defaults depend on the application and actual settings depend on the admin/postmaster. So retry queues will typically be somewhere between 3 to 7 days, depending. High volume sending servers may be shorter. It is rare for short outages to cause emails to be lost unless the outage is due to a fire and the server holding your message melted.

block_dagger 5y ago

Prior to the outage I received an erroneous email from them about an old account, followed by an apology about it. It seems they were having other issues than just the router config.

itaibo 5y ago

You can follow the status page here: http://travaux.ovh.net http://travaux.ovh.net

qeternity 5y ago

Has OVH gotten substantially worse in the past couple of years? We ran a decent cluster in GRA-3 for a few years, and I can't recall having any real downtime. But lots of comments here lead me to believe that other people have very different experiences with OVH.

champtar 5y ago

Here the bad copy/paste https://mobile.twitter.com/fred21946537/status/1448219047494098944 https://mobile.twitter.com/fred21946537/status/1448219047494...