10 ms·

Sourcehut network outage post-mortem

286 points by ggpsv 3y ago

kyrra 3y ago
I've been using Sourcehut for a couple years now. One thing this outage taught me about the service that I didn't know is that Mercurial (hg) is community maintained: > We also did our best with hg.sr.ht, but it is community maintained It looks like git.sr.ht is hosted on OVH in France, while hg.sr.ht is hosted on High5! in the Netherlands. It's not entirely clear to me how this affects their product roadmap or support, but definitely good to know.
- skywal_l 3y ago
  > It looks like git.sr.ht is hosted on OVH in France They explain it here: > However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS
  - pelagicAustral 3y ago
    That's such an odd choice for this type of infra. I've had horrendous experiences with OVH in the past and what even worse, terrible customer service. Yes, this was about 8 years ago, and not with France based metal, but still... Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with an anti-establishment manifesto than any practicality. Then again, I might be wrong.
    - DistractionRect 3y ago
      Well, it's certainly better than their last provider who they couldn't reach during a critical time, and still cannot reasonably communicate with. They can at least reach and reason with OVH, as mentioned when they got flagged as an out bound DDoS. > Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with a anti-establishment manifesto than any practicality I feel this is a pretty unfair barb considering one of their first moves was reaching out to Cloudflare. Unfortunately, non-http traffic + the need for tls termination on their own servers (pretty sure cloudflare calls this Keyless SSL) squarely lands them as an enterprise customer w/ enterprise pricing. Drew probably had already entered into agreements with OVH when cloudflare came back around, and we don't have insight on the terms or period for which Cloudflare's second offer was good for.
    - elpy1 3y ago
      > Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with a anti-establishment manifesto than any practicality Not wanting your traffic MITM'd is anti-establishment. That's where we're at LOL.
- twic 3y ago
  This also came as a surprise to me! Not only that but: > restoring service was delayed until we could get the community maintainer, Ludovic Chabant, online to help Maintainer, singular! The only reason i use Sourcehut, and the main reason i pay for it, is because i stubbornly still use Mercurial, and want first-class support for it. With the utmost of respect to M. Chabant, that is not exactly first-class.
  - nequo 3y ago
    > With the utmost of respect to M. Chabant, that is not exactly first-class. It would appear that Ludovic Chabant is working full-time at Epic Games. He is unlikely to have the capacity to be on call for Sourcehut.
    - vanderZwan 3y ago
      I think the complaint was aimed at Sourcehut leaning on a sole volunteer for this service, not at Ludovic Chabant
      - deleted 3y ago
        [deleted]
      - drewdevault 3y ago
        hg.sr.ht would not exist if not for being community maintained; the cost/benefit ratio does not work out in its favor were we to maintain it internally. The deal is that we provide infrastructure and operations but that the software itself is maintained by the community that needs it. I think this is an advantage of SourceHut's free software model and ethic that allows people to build what they need and to get its infrastructure needs met in a way that wouldn't be possible, for example, on GitHub.
- drewdevault 3y ago
  hg.sr.ht is operated by SourceHut, but the software is maintained by the community. Ludovic is the primary maintainer and various other Mercurial users participate in its development.
- moberley 3y ago
  For me there a bit of a language barrier with the terminology. After reading the sentences about hg.sr.ht and community maintenance it seems that some notable meaning is being conveyed about what that means for the operation of the service but its one I'm not smart enough to understand. I appreciate the service though so I hope the differences between maintained and operated doesn't mean anything in the long term.
hypeatei 3y ago
> As unfortunate as these events were, we welcome opportunities to stress-test our emergency procedures; This right here is invaluable and something you only get from experience. Planning and theory only get you so far. I extend this thinking to deploying large infrastructure changes you've never done before - you can only plan so much before pulling the trigger and just doing it and seeing what happens.
- 20after4 3y ago
  Wikimedia's operations team go through a full-datacenter failover regularly. That is, once every 6 months or so. It takes several hours of intense all-hands-on-deck operations. They do this repeatedly in order to be sure that all of the procedures are practiced and documentation is well maintained.
shrubble 3y ago
Would have liked to know what the difference was in response between Cogent and Level3. Did only Cogent respond at all, or was Cogent the one handling all their IPv4 space?
- zeroclicks 3y ago
  Seems only Cogent was advertising their routes. Once Cogent blackholed their prefixes, there'd be no way to reach their services via the internet.
scandox 3y ago
I'm still left not quite certain what would happen if they were hit with another L3 DDOS tomorrow. That said I'm very happy to use Sourcehut and I think they'll overcome these challenges over time. They seem to have the staying power.
- frakkingcylons 3y ago
  They’re on OVH now and should have protection from it by virtue of being on their network now. > However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.
  - treesknees 3y ago
    I'd consider it mostly protected, because no their servers are not on OVH, just a single box performing front-facing NAT/proxy essentially. The attacker now just needs to find the "secret" production subnet and attack it directly instead of through the front-facing NAT addresses.
    - frakkingcylons 3y ago
      That's true. I guess it'd be more accurate to say they're on OVH's network, not necessarily their compute and other infrastructure.
    - bombcar 3y ago
      That is very easy to mitigate, because you null route the production subnet except for a VPN that only can be reached by the proxy. You can even VPN over a completely different IPv6 route. They could still try to knock your entire datacenter offline but that is much harder.
      - 3y ago
OsrsNeedsf2P 3y ago
Unrelated, but TIL Drew DeVault is one of the SourceHut maintainers. His blog[0] is strongly opinionated and always an informative read. [0] https://drewdevault.com/ https://drewdevault.com/
- mortallywounded 3y ago
  Maintainer? More like creator.
  - trevyn 3y ago
    Opinionated? More like firebrand.
  - otachack 3y ago
    ¿Por qué no los dos?
  - j4yav 3y ago
    Arent open source project maintainers typically the creators?
- matthews2 3y ago
  You don't need to agree with all of his opinions to use SourceHut :) I'm not a big fan of some of his hot takes, but I still respect him and trust him with my data.
  - sneak 3y ago
    The reason I don’t use srht is because of his opinions about product development (of srht itself), not his personal opinions. Social/collaboration features are explicitly deprioritized by design; I think this is a natural consequence of srht being built by and for lone wolf developers. GitHub and Gitea (which is basically a github clone) seem much more geared toward collaboration by groups, something most small-time f/oss developers don’t need. Also, the emphasis on email and irc is bad, imo. The web won because it is better. A lot of the anti-web stuff is just tradition.
oldpersonintx 3y ago
[dead]
rezmason 3y ago
Wild speculation: maybe the attacker's motive was to usher specific Sourcehut hosted repositories to the jurisdiction of the EU.
- ploum 3y ago
  On a more serious note, I’m really wondering about the motivations. I see the following hypothesis: 1) Test/demonstration of a DDOS against a random target. 2) Attack against a project hosted on sourcehut to make it unavailable (there was even the speculation of disabling a master repository so an end-user could not check that his own local version was the correct one, thus using it with a security hole or a trojan) 3) Attack against a page hosted on sourcehut (I joke that someone wrote "Putin = Fag" on his sourcehut hosted blog). 4) What else ?
  - SSLy 3y ago
    1A) Test/demonstration of a DDOS against a service well know among some circles
  - miki123211 3y ago
    5) Somebody very vehemently disagreeing with Drew's (often radical) views. This could also be a simple transphobe, though I expect those people to have far juicier targets to attack.
    - ploum 3y ago
      I’m following Drew’s blog and don’t remember anything particular about trans rights (except some "I condemn those attacking LGBT’s rights", which is not particularly radical). Also, the attack (which had an obvious cost) would have no impact regarding this cause. So, to me, this is quite unlikely.
      - tourmalinetaco 3y ago
        It’s more likely than you might think overall, though I’ve seen more of it from the other side of the aisle.
svieira 3y ago
Looks like Cloudflare did change their minds later and offered to mitigate the attack pro bono: > Following our initial quote from CloudFlare, we understand that some CloudFlare employees undertook a grassroots effort internally to convince the leadership to sponsor our needs, and eventually CloudFlare came back to us with an offer to sponsor our services for us free of charge. This was a very generous offer for which we are very appreciative; in the end we did not take them up on it as we had made substantial inroads towards an alternative solution by that time. I have had my reservations about CloudFlare in the past, but they were there for us in a time of need and I am grateful for that.
- zeroclicks 3y ago
  Typical "corporate pricing"--they offer a really high price they'll expect you'll negotiate downwards to something reasonable. The Sourcehut negotiators probably never dealt with this kind of "sales model" before. That said, what will happen when more companies publish their experiences with "enterprise sales"? There's an article from HEY[1] about how broken the sales process is. To get a quote, you normally have to endure 2 or 3 zoom calls before the price is unveiled. There's probably room for an innovator to fix all of this. 1: https://world.hey.com/dhh/the-only-thing-worse-than-cloud-pricing-is-the-enterprisey-alternatives-854e98f3 https://world.hey.com/dhh/the-only-thing-worse-than-cloud-pr...
  - drewdevault 3y ago
    We did negotiate them down a bit but we didn't feel that we could come to an agreement within our budget and decided to move on. Apparently this was an excellent negotiation tactic because they came back with an offer of $0!
  - ralph84 3y ago
    It’s not broken per se. Charging different customers different prices based on their willingness to pay maximizes profits. Heck even Sourcehut does it, but they don’t require interacting with sales because at their pricing they don’t have enough margin for a sales team.
tetha 3y ago
I find it somewhat chilling how their original colo left them to hang and dry. Maybe I'm weird, but I'd consider colo to be a closer cooperation than just renting some virtual servers from wherever. And just getting told "Yupp, your null-routed. No, we can't give you access for specific sources over a different path. Get fucked" - or, in fact, not getting told that - is ... one of our ex-hosters was like that. And as a service provider, I have strong feelings about the customer service there. Maybe I don't know big infrastructures, but this just leaves me with a weird feeling in my guts. But hell. Make sure to give your engineers - and their family - something. After some hell-weeks, we've given people some budget to do something fun with their family, because the company had to take so much private time during those weeks.
- downrightmike 3y ago
  being null routed is really the only thing they can do. Then then undid it and they had to do it again. This wasn't a standard DDOS attack, which they normally handle just fine. Good coverage of the event: Security Now! Podcast https://www.youtube.com/watch?v=ehfV7cRLkFE https://www.youtube.com/watch?v=ehfV7cRLkFE
  - aidenn0 3y ago
    About 29minutes in is where it picks up after reading verbatim the status report from Drew.
  - MaxBarraclough 3y ago
    Drew's second mention on Security Now, for what it's worth. Gibson quoted one of his blog posts back in September. https://www.grc.com/sn/sn-941.htm https://www.grc.com/sn/sn-941.htm
  - iso8859-1 3y ago
    Subscribed and the unsubscribed shortly after when they mistook Greenland for Iceland (42m55s), Italy for ??? (43m42s) and then Equatorial Guinea for Chad (44m55s).
  - AlexeyBelov 3y ago
vander_elst 3y ago
Mostly curious about the k8s plans. From some past posts it seems that the team was strongly against employing containerization [0]. However, it seems something changed. If anyone has more info about this if love to hear more. [0] https://news.ycombinator.com/item?id=23030489 https://news.ycombinator.com/item?id=23030489
doublerabbit 3y ago
> This outcome was unacceptable No it wasn't. The outcome is due to major networks being shite. Not accommodating newer technologies and gate keeping services to resolve DDoS attacks. All major network upstreams could do so much more to make the net more reliable and resilient to small ISP. Myself included. peer neutral networking, not having tons upon tons of e-waste prone to botnet behaviour, it wouldn't be like this.
imiric 3y ago
I admire Drew's work and SourceHut, but I wonder if choosing colocation instead of a cloud provider, and not adopting modern cloud tooling hurt the recovery time. The article mentions how setting up SourceHut from scratch is a complex undertaking with hundreds of small tasks. Some of those are understandable, especially given the amount of data they surely handle, but setting up a complete environment and restoring from a backup should be a simple and mostly automated procedure, not a gargantuan undertaking that needs to involve the entire team several days. There are always difficulties when production is down, and you're trying to restore a full system while undergoing a DDoS attack, I get that, but the reason we have modern cloud providers and tooling is to make creating new environments as painless as possible. It seems foolish not to take advantage of that. I'm not a fan of Kubernetes either, but it's good that they're experimenting with it. Hopefully it leads to quicker deployments if this happens again.
- nullwarp 3y ago
  Cloud isn't a dependency to accomplish this. I can stand up new instances of our primary infrastructure within minutes on any VM or server running linux and our agent. This sounds like a case of treating your infra as pets and getting stuck when it suddenly needs to be replaced.
  - Aeolun 3y ago
    That’s not really a surprise considering how they started? To some extend I think it’s unavoidable if you use colocation.
xwolfi 3y ago
[flagged]
grolegat 3y ago
[flagged]
rony989 3y ago
[dead]

286 points by ggpsv 3y ago

I've been using Sourcehut for a couple years now. One thing this outage taught me about the service that I didn't know is that Mercurial (hg) is community maintained: > We also did our best with hg.sr.ht, but it is community maintained It looks like git.sr.ht is hosted on OVH in France, while hg.sr.ht is hosted on High5! in the Netherlands. It's not entirely clear to me how this affects their product roadmap or support, but definitely good to know.

skywal_l 3y ago
> It looks like git.sr.ht is hosted on OVH in France They explain it here: > However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS
- pelagicAustral 3y ago
  That's such an odd choice for this type of infra. I've had horrendous experiences with OVH in the past and what even worse, terrible customer service. Yes, this was about 8 years ago, and not with France based metal, but still... Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with an anti-establishment manifesto than any practicality. Then again, I might be wrong.
  - DistractionRect 3y ago
    Well, it's certainly better than their last provider who they couldn't reach during a critical time, and still cannot reasonably communicate with. They can at least reach and reason with OVH, as mentioned when they got flagged as an out bound DDoS. > Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with a anti-establishment manifesto than any practicality I feel this is a pretty unfair barb considering one of their first moves was reaching out to Cloudflare. Unfortunately, non-http traffic + the need for tls termination on their own servers (pretty sure cloudflare calls this Keyless SSL) squarely lands them as an enterprise customer w/ enterprise pricing. Drew probably had already entered into agreements with OVH when cloudflare came back around, and we don't have insight on the terms or period for which Cloudflare's second offer was good for.
  - elpy1 3y ago
    > Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with a anti-establishment manifesto than any practicality Not wanting your traffic MITM'd is anti-establishment. That's where we're at LOL.
twic 3y ago
This also came as a surprise to me! Not only that but: > restoring service was delayed until we could get the community maintainer, Ludovic Chabant, online to help Maintainer, singular! The only reason i use Sourcehut, and the main reason i pay for it, is because i stubbornly still use Mercurial, and want first-class support for it. With the utmost of respect to M. Chabant, that is not exactly first-class.
- nequo 3y ago
  > With the utmost of respect to M. Chabant, that is not exactly first-class. It would appear that Ludovic Chabant is working full-time at Epic Games. He is unlikely to have the capacity to be on call for Sourcehut.
  - vanderZwan 3y ago
    I think the complaint was aimed at Sourcehut leaning on a sole volunteer for this service, not at Ludovic Chabant
    - deleted 3y ago
      [deleted]
    - drewdevault 3y ago
      hg.sr.ht would not exist if not for being community maintained; the cost/benefit ratio does not work out in its favor were we to maintain it internally. The deal is that we provide infrastructure and operations but that the software itself is maintained by the community that needs it. I think this is an advantage of SourceHut's free software model and ethic that allows people to build what they need and to get its infrastructure needs met in a way that wouldn't be possible, for example, on GitHub.
drewdevault 3y ago
hg.sr.ht is operated by SourceHut, but the software is maintained by the community. Ludovic is the primary maintainer and various other Mercurial users participate in its development.
moberley 3y ago
For me there a bit of a language barrier with the terminology. After reading the sentences about hg.sr.ht and community maintenance it seems that some notable meaning is being conveyed about what that means for the operation of the service but its one I'm not smart enough to understand. I appreciate the service though so I hope the differences between maintained and operated doesn't mean anything in the long term.

hypeatei 3y ago

> As unfortunate as these events were, we welcome opportunities to stress-test our emergency procedures; This right here is invaluable and something you only get from experience. Planning and theory only get you so far. I extend this thinking to deploying large infrastructure changes you've never done before - you can only plan so much before pulling the trigger and just doing it and seeing what happens.

20after4 3y ago
Wikimedia's operations team go through a full-datacenter failover regularly. That is, once every 6 months or so. It takes several hours of intense all-hands-on-deck operations. They do this repeatedly in order to be sure that all of the procedures are practiced and documentation is well maintained.

shrubble 3y ago

Would have liked to know what the difference was in response between Cogent and Level3. Did only Cogent respond at all, or was Cogent the one handling all their IPv4 space?

zeroclicks 3y ago
Seems only Cogent was advertising their routes. Once Cogent blackholed their prefixes, there'd be no way to reach their services via the internet.

scandox 3y ago

I'm still left not quite certain what would happen if they were hit with another L3 DDOS tomorrow. That said I'm very happy to use Sourcehut and I think they'll overcome these challenges over time. They seem to have the staying power.

frakkingcylons 3y ago
They’re on OVH now and should have protection from it by virtue of being on their network now. > However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.
- treesknees 3y ago
  I'd consider it mostly protected, because no their servers are not on OVH, just a single box performing front-facing NAT/proxy essentially. The attacker now just needs to find the "secret" production subnet and attack it directly instead of through the front-facing NAT addresses.
  - frakkingcylons 3y ago
    That's true. I guess it'd be more accurate to say they're on OVH's network, not necessarily their compute and other infrastructure.
  - bombcar 3y ago
    That is very easy to mitigate, because you null route the production subnet except for a VPN that only can be reached by the proxy. You can even VPN over a completely different IPv6 route. They could still try to knock your entire datacenter offline but that is much harder.
    - 3y ago

OsrsNeedsf2P 3y ago

Unrelated, but TIL Drew DeVault is one of the SourceHut maintainers. His blog[0] is strongly opinionated and always an informative read. [0] https://drewdevault.com/ https://drewdevault.com/

mortallywounded 3y ago
Maintainer? More like creator.
- trevyn 3y ago
  Opinionated? More like firebrand.
- otachack 3y ago
  ¿Por qué no los dos?
- j4yav 3y ago
  Arent open source project maintainers typically the creators?
matthews2 3y ago
You don't need to agree with all of his opinions to use SourceHut :) I'm not a big fan of some of his hot takes, but I still respect him and trust him with my data.
- sneak 3y ago
  The reason I don’t use srht is because of his opinions about product development (of srht itself), not his personal opinions. Social/collaboration features are explicitly deprioritized by design; I think this is a natural consequence of srht being built by and for lone wolf developers. GitHub and Gitea (which is basically a github clone) seem much more geared toward collaboration by groups, something most small-time f/oss developers don’t need. Also, the emphasis on email and irc is bad, imo. The web won because it is better. A lot of the anti-web stuff is just tradition.

oldpersonintx 3y ago

[dead]

rezmason 3y ago

Wild speculation: maybe the attacker's motive was to usher specific Sourcehut hosted repositories to the jurisdiction of the EU.

ploum 3y ago
On a more serious note, I’m really wondering about the motivations. I see the following hypothesis: 1) Test/demonstration of a DDOS against a random target. 2) Attack against a project hosted on sourcehut to make it unavailable (there was even the speculation of disabling a master repository so an end-user could not check that his own local version was the correct one, thus using it with a security hole or a trojan) 3) Attack against a page hosted on sourcehut (I joke that someone wrote "Putin = Fag" on his sourcehut hosted blog). 4) What else ?
- SSLy 3y ago
  1A) Test/demonstration of a DDOS against a service well know among some circles
- miki123211 3y ago
  5) Somebody very vehemently disagreeing with Drew's (often radical) views. This could also be a simple transphobe, though I expect those people to have far juicier targets to attack.
  - ploum 3y ago
    I’m following Drew’s blog and don’t remember anything particular about trans rights (except some "I condemn those attacking LGBT’s rights", which is not particularly radical). Also, the attack (which had an obvious cost) would have no impact regarding this cause. So, to me, this is quite unlikely.
    - tourmalinetaco 3y ago
      It’s more likely than you might think overall, though I’ve seen more of it from the other side of the aisle.

svieira 3y ago

Looks like Cloudflare did change their minds later and offered to mitigate the attack pro bono: > Following our initial quote from CloudFlare, we understand that some CloudFlare employees undertook a grassroots effort internally to convince the leadership to sponsor our needs, and eventually CloudFlare came back to us with an offer to sponsor our services for us free of charge. This was a very generous offer for which we are very appreciative; in the end we did not take them up on it as we had made substantial inroads towards an alternative solution by that time. I have had my reservations about CloudFlare in the past, but they were there for us in a time of need and I am grateful for that.

zeroclicks 3y ago
Typical "corporate pricing"--they offer a really high price they'll expect you'll negotiate downwards to something reasonable. The Sourcehut negotiators probably never dealt with this kind of "sales model" before. That said, what will happen when more companies publish their experiences with "enterprise sales"? There's an article from HEY[1] about how broken the sales process is. To get a quote, you normally have to endure 2 or 3 zoom calls before the price is unveiled. There's probably room for an innovator to fix all of this. 1: https://world.hey.com/dhh/the-only-thing-worse-than-cloud-pricing-is-the-enterprisey-alternatives-854e98f3 https://world.hey.com/dhh/the-only-thing-worse-than-cloud-pr...
- drewdevault 3y ago
  We did negotiate them down a bit but we didn't feel that we could come to an agreement within our budget and decided to move on. Apparently this was an excellent negotiation tactic because they came back with an offer of $0!
- ralph84 3y ago
  It’s not broken per se. Charging different customers different prices based on their willingness to pay maximizes profits. Heck even Sourcehut does it, but they don’t require interacting with sales because at their pricing they don’t have enough margin for a sales team.

tetha 3y ago

I find it somewhat chilling how their original colo left them to hang and dry. Maybe I'm weird, but I'd consider colo to be a closer cooperation than just renting some virtual servers from wherever. And just getting told "Yupp, your null-routed. No, we can't give you access for specific sources over a different path. Get fucked" - or, in fact, not getting told that - is ... one of our ex-hosters was like that. And as a service provider, I have strong feelings about the customer service there. Maybe I don't know big infrastructures, but this just leaves me with a weird feeling in my guts. But hell. Make sure to give your engineers - and their family - something. After some hell-weeks, we've given people some budget to do something fun with their family, because the company had to take so much private time during those weeks.

downrightmike 3y ago
being null routed is really the only thing they can do. Then then undid it and they had to do it again. This wasn't a standard DDOS attack, which they normally handle just fine. Good coverage of the event: Security Now! Podcast https://www.youtube.com/watch?v=ehfV7cRLkFE https://www.youtube.com/watch?v=ehfV7cRLkFE
- aidenn0 3y ago
  About 29minutes in is where it picks up after reading verbatim the status report from Drew.
- MaxBarraclough 3y ago
  Drew's second mention on Security Now, for what it's worth. Gibson quoted one of his blog posts back in September. https://www.grc.com/sn/sn-941.htm https://www.grc.com/sn/sn-941.htm
- iso8859-1 3y ago
  Subscribed and the unsubscribed shortly after when they mistook Greenland for Iceland (42m55s), Italy for ??? (43m42s) and then Equatorial Guinea for Chad (44m55s).
- AlexeyBelov 3y ago

vander_elst 3y ago

Mostly curious about the k8s plans. From some past posts it seems that the team was strongly against employing containerization [0]. However, it seems something changed. If anyone has more info about this if love to hear more. [0] https://news.ycombinator.com/item?id=23030489 https://news.ycombinator.com/item?id=23030489

doublerabbit 3y ago

> This outcome was unacceptable No it wasn't. The outcome is due to major networks being shite. Not accommodating newer technologies and gate keeping services to resolve DDoS attacks. All major network upstreams could do so much more to make the net more reliable and resilient to small ISP. Myself included. peer neutral networking, not having tons upon tons of e-waste prone to botnet behaviour, it wouldn't be like this.

imiric 3y ago

I admire Drew's work and SourceHut, but I wonder if choosing colocation instead of a cloud provider, and not adopting modern cloud tooling hurt the recovery time. The article mentions how setting up SourceHut from scratch is a complex undertaking with hundreds of small tasks. Some of those are understandable, especially given the amount of data they surely handle, but setting up a complete environment and restoring from a backup should be a simple and mostly automated procedure, not a gargantuan undertaking that needs to involve the entire team several days. There are always difficulties when production is down, and you're trying to restore a full system while undergoing a DDoS attack, I get that, but the reason we have modern cloud providers and tooling is to make creating new environments as painless as possible. It seems foolish not to take advantage of that. I'm not a fan of Kubernetes either, but it's good that they're experimenting with it. Hopefully it leads to quicker deployments if this happens again.

nullwarp 3y ago
Cloud isn't a dependency to accomplish this. I can stand up new instances of our primary infrastructure within minutes on any VM or server running linux and our agent. This sounds like a case of treating your infra as pets and getting stuck when it suddenly needs to be replaced.
- Aeolun 3y ago
  That’s not really a surprise considering how they started? To some extend I think it’s unavoidable if you use colocation.

xwolfi 3y ago

[flagged]

grolegat 3y ago

[flagged]

rony989 3y ago

[dead]