11 ms·
Statement regarding the ongoing Sourcehut outage
- okasaki 3y agoThis site can’t be reached DNS_PROBE_FINISHED_NXDOMAIN
- callalex 3y agoAm I correct in assuming these kind of problems are not possible if you are using a major cloud provider instead of renting rack space?
- eterps 3y agoYou are probably correct in assuming these kinds of problems are not possible given enough $$$
- preisschild 3y agoDepends if you buy their ddos protection services or not.
- esskay 3y agoStill very much possible, in most cases all you're doing at the cloud provider is running virtual hardware through load balancers so they can very much become overwhelmed. Even with more advanced setups, it's still all just hitting a variety of servers, each of which can be affected in different ways. There are (expensive) ways of mitigating but a project like Sourcehut couldn't afford or justify what will likely be a 5 to 6 figure sum.
- daenney 3y agoNo, that is not a correct assumption. It's also worth remembering that cloud providers are also servers in racks, but they own the building around it to. For DDoS attacks, you need to have enough capacity to absorb the attack. Major cloud providers tend to have that, as do DDoS mitigation services (Cloudflare amongst others). > We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means [..] Typically what you want to do though is stop the traffic from reaching you at all, so ideally your network provider, who is upstream from you, blocks the illegitimate traffic so your servers never see it and don't get overwhelmed. What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage. Imagine if your ISP decided to stop routing traffic to Google. Being hosted on GCP, a major cloud provider, would be of no use, since there wouldn't be a network path to them in the first place. In general, Cogent seems to be doing a rather bad job at dealing with this attack and there's been fallout for many services beyond Sourchut. Google or AWS or Microsoft might've handled it more gracefully, or might not. Though major cloud providers tend to have their own connectivity between their datacenters, they too have peering/transit agreements with other major network providers. If those upstreams stop forwarding traffic to them, the same thing would happen. It's just less likely to go unnoticed. Cogent is a massive provider, so you'd think they'd be a bit better at this. But they also have a reputation for being awful.
- dewey 3y ago> What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage. That's not what happened based on my understanding. The provider nullrouted their traffic (which is common if a customer is under attack), but Sourcehut couldn't talk to the customer support as their support panel wasn't working for them.
- fmajid 3y agoCogent is also a low-cost provider, and it shows in their customer service.
- JoshTriplett 3y agoIf you're using a major cloud provider, a DDoS might mean either that your service hits scaling limits or that you get a massive bill.
- tgsovlerkhgsel 3y agoFor any company with a complicated cloud footprint, how do they avoid "denial-of-money" by having the attacker pick some externally reachable piece of the cloud setup where the attacker can take actions that the company pays for (e.g. download large files from a cloud storage bucket, generating huge bandwidth charges)?
- miki123211 3y agoThis is a major problem, particularly for those offering any kind of outbound SMS capability, whether for 2FA or just phone number verification. That service is the easiest and most profitable to abuse, there are certain providers in certain countries that price inbound SMS very steeply, and are willing to share the profits with you. If you if you can get an attack going.
- justsomehnguy 3y agoCodeberg went down to DDoS ~12h ago too: https://status.codeberg.eu/status/codeberg https://status.codeberg.eu/status/codeberg
- EasyMark 3y agoI wonder why they're thinking about moving to EU as their compatriot seems to be having the same issue and moving to Europe wouldn't fix it. It was mentioned in their public notice.
- karmarepellent 3y agoTheir plan to move to the EU via their AMS site is not related to the attacks, but seems to be a general plan they have had for a while. At least this is how I understand their notice.
- bayindirh 3y agoThey were planning to moving to EU for the last 6-8 months. They were making it gracefully. This attack expedited it greatly.
- mrweasel 3y agoThe EU move wouldn't exactly help, if the attack is specifically directed at Sourcehut. That is unless the ISP and hosting provider in AMS is offering some form of protection as part of their offering. I can't see why the attack wouldn't just follow the services.
- bayindirh 3y agoYou're right, but they were already planning it, so why not? Maybe they have another plans under that, a better server or set of servers, some hand-rolled mitigations, etc. I have no idea. I'm a user as everybody else.
- 3y ago
- febeling 3y agoWhat is the motivation of ddos attacks, in general? I assume it comes with some risk of being criminally prosecuted, so there must be some upside. Is it shakedown attempts, or competitor's sabotage? Neither seems too plausible in this case.
- mardifoufs 3y agoFor dark net markets, it was commonly used to extract a ransom or to make other websites aware of their DDoS capabilities so that others pay the ransom too. But for a clear web website it's probably not a monetary incentive
- raverbashing 3y ago> it comes with some risk of being criminally prosecuted Has there been actual cases of prosecution?
- dewey 3y agoOne recent example would be Zeekill, who ran a lot of DDoS attacks: https://krebsonsecurity.com/2023/02/finlands-most-wanted-hacker-nabbed-in-france/ https://krebsonsecurity.com/2023/02/finlands-most-wanted-hac...
- Propa_ganda 3y agoMost of lulzsec, for example. https://en.wikipedia.org/wiki/LulzSec https://en.wikipedia.org/wiki/LulzSec
- wildrhythms 3y agohttps://www.justice.gov/opa/pr/former-operator-illegal-booter-services-sentenced-conspiracy-commit-computer-damage-and-abuse https://www.justice.gov/opa/pr/former-operator-illegal-boote... https://www.justice.gov/opa/pr/criminal-charges-filed-los-angeles-and-alaska-conjunction-seizures-15-websites-offering-ddos https://www.justice.gov/opa/pr/criminal-charges-filed-los-an... https://www.justice.gov/opa/pr/individual-pleads-guilty-participating-internet-things-cyberattack-2016 https://www.justice.gov/opa/pr/individual-pleads-guilty-part...
- MichaelMug 3y ago1. Do "PHL, FRE, and AMS" mean something? Or are these just codenames for each site? 2. If I host a service on AWS, Azure, Linode, DigitalOcean am I also susceptible to layer 3 DDoS?
- emersion 3y ago1. PHL means Philadelphia, AMS means Amsterdam. Yes, just codenames for sites. 2. Depends if DDoS protection is part of the offer, I suppose.
- cr3ative 3y agoThey're generally airport codes, so Philadelphia and Amsterdam... unsure about FRE.
- Y_Y 3y agoThe IATA code FRE is assigned to Fera in the Solomon Islands. Freemont Airport does not have an IATA code.
- fmajid 3y agoNaming conventions can be inconsistent, and not every datacenter location has an airport. ASD is often used instead of AMS for Amsterdam.
- Y_Y 3y agoASD is the IATA code assigned to Andros Town Airport, in the Bahamas. It might also be a commonly used abbreviation for a medical condition.
- gslin 3y agoTheir topology page has information about PHL/AMS and FRE (seems like Fremont datacenter): https://web.archive.org/web/20240111132224/https://man.sr.ht/ops/topology.md https://web.archive.org/web/20240111132224/https://man.sr.ht...
- csunbird 3y ago> We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means, but we are investigating other solutions which may be more affordable and have a few avenues for research today, though we cannot disclose too many details without risking alerting the attackers to our plans. Maybe someone higher up in Cloudflare here can escalate this subject and help them along for a proper quote?
- that_guy_iain 3y agoJust because the quote is expensive doesn't make it not a proper quote. Either CloudFlare does it for PR for free or they need paid for a proper quote.
- csunbird 3y agoLet me rephrase: Maybe they can negotiate a different quote? The PR gained in the development community to help a service beloved by hackers in this site could be very beneficial.
- okasaki 3y ago[flagged]
- arghwhat 3y agoYeah, that post is not at all related to getting a discount from Cloudflare on a product they need to buy from somewhere regardless. The owner is in general against slowly having all traffic on the internet routed through Cloudflare, but considering that they asked Cloudflare they seem to be interested in that compromise to stay online.
- plagiarist 3y agoI don't understand how the linked image had anything to do with Cloudflare. Are DDoS attacks happening specifically to marginalized people, and Cloudflare is the techbro getting rich, and Cloudflare is "scanning eyeballs?" IMO that quote is more about, say, Facebook blimping in internet for parts of Africa. The person posting it here as if the author is against all tech companies seems like a deliberate misinterpretation.
- Aissen 3y agoInteresting to learn the reason HN was down: > You may have noticed that Hacker News was down on January 10th; we believe that was ultimately due to Cogent’s heavy handed approach to mitigating the DDoS targetting SourceHut (sorry, HN, glad you got it sorted). It was also in https://news.ycombinator.com/item?id=38939532 https://news.ycombinator.com/item?id=38939532 but I did not see it earlier.
- kazen44 3y agoto be fair, you reach a certain point when the only options left for DDoS protection are Remote Triggered blackholing on your edge. This results in an entire /24 network not being routed to your network and being dropped by your peer instead.
- mariusor 3y agoWhich is a totally reasonable way of going about the problem, of course. /s
- coldacid 3y agoNobody has ever accused Cogent of being reasonable or competent.
- bityard 3y agoThat's not true. I work for a company that sells DDoS mitigation products to large network operators like Cogent for helping them deal with exactly these kinds of attacks in a much more sane manner than, "oy, just blackhole the destination and head out to lunch." Either Cogent didn't buy our product (or a competitor's equivalent), or they have a network op who's a fresher and only knows how to blackhole things. Either way, it's a bad look for Cogent.
- dfc 3y agoThat is not a new look for cogent.
- beretguy 3y agoGodot, codeberg, sourcehut… any other websites? Looks like somebody is targeting open source related websites.
- nusl 3y agoSomebody is demonstrating their DDOS capabilities to be able to sell capacity to malicious buyers
- numbsafari 3y agoOr someone is trying to make something being hosted there unavailable.
- sam_lowry_ 3y agoMicrosoft's Github?
- internetter 3y agoMicrosoft has very little to gain from DDoSing sourcehut as at the end of the day GitHub’s competition is like flies, AND Microsoft has a lot to loose if it was discovered they did it
- sam_lowry_ 3y agoCompanies are made of people. Those whose wealth and wellbeing depend on Github being the monopoly may be interested in destroying the competition before it grows into a concern.
- dingnuts 3y agoIt would be really stupid for Microsoft, the country's richest company, to go after a tiny competitor via extralegal means. Absolutely would make no sense. The liability they would incur would be astronomical
- andersa 3y ago> We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means I don't get it. Cloudflare proudly advertises unmetered DDoS protection on any plan level. Is that just a lie, or what am I missing here? They don't need to be on a custom Enterprise plan.
- tpetry 3y agoThey are only doing unmetered DDoS for HTTP traffic - for websites. File storage etc. is not allowed. And git falls into that realm.
- andersa 3y agoI see, that explains it.
- Deathmax 3y agoTo expand on that, Cloudflare's standard product is their HTTP reverse proxy. To proxy arbitrary TCP/UDP traffic, you need to use their Cloudflare Spectrum service (https://www.cloudflare.com/en-gb/application-services/products/cloudflare-spectrum/ https://www.cloudflare.com/en-gb/application-services/produc...), which is metered.
- blibble 3y agowow $1/GB? would be cheaper to pay a developer to add websocket support to openssh
- matteocontrini 3y agoI think they might be looking in something more similar to Magic Transit, but that depends on how they set up their infrastructure.
- donor20 3y agoUnlimited free is almost always a lie.
- cutner 3y ago[flagged]
- tristan957 3y agoWhen has Drew said he was a fan of DDoS attacks? When you make claims like these you should really bring receipts. SourceHut doesn't blackhole content. It is in their terms of service and documentation what they will not accept on their website.
- gavinhoward 3y ago[flagged]
- deleted 3y ago[deleted]
- eesmith 3y agoYes, he doesn't like cryptocurrency projects. I don't see much vague about it. The other clauses about bigotry are pretty much aligned with GitHub's ToS, which has "GitHub does not tolerate speech that attacks or promotes hate toward an individual or group of people on the basis of who they are, including age, body size, ability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, sexual identity, or sexual orientation", and "We have the right to refuse or remove any User-Generated Content that, in our sole discretion, violates any laws or GitHub terms or policies." While people who don't like these policies call it "deplatforming", in the US the freedom of association is a protected First Amendment right, with only limited exceptions for certain protected classes like race, ethnicity, and national origin where there is a strong tension between the First Amendment protections and other constitutional protections. I don't know about Dutch law though.
- gavinhoward 3y agoIn private communication, he told me that he would ban me for being Christian. Yeah, his policies are "aligned" with GitHub's (and I avoid that too), but Drew is on the record saying he will ban people for their opinions, not their actions.
- throw9999away 3y agoI run a service which was recently DDOS'd. It came immediately after a request from the police services in India to give them data on a specific user. They claimed it was terrorism related. We explained our policy on that (there is a procedure but it requires lengthy legal formalities in our jurisdiction). Within hours our service was taken down. It lasted about 12 hours. We never received any ransom request. We were left with 3 possibilities: 1. Indian police/government 2. Original user 3. Total coincidence When it happens you just realize: this should not be possible. This is a game that hugely benefits the most powerful players whether they just don't bother solving the problem or actively play in it.
- teddyh 3y agoHow sure are you that the request came from the actual police in India, and not somebody else? I.e. someone who might get irrationally upset when you did not give them what you wanted?
- mattw2121 3y ago"someone who might get irrationally upset when you did not give them what you wanted" Like the police in India?
- teddyh 3y agoI would assume that the actual police have their own traditional methods of being irrationally vindictive, and would not normally restort to DDoS.
- alright2565 3y agoIndian police will have a lot of trouble beating you if you live in another country.
- m-p-3 3y agoTheir government might though..
- MartijnBraam 3y agoI wish the sr.ht employees the best of luck with the AMS migration. Luckily git is decentralized enough that I've been able to practically do everything the last few days except that I can't push new releases of software to the canonical upstream repository.
- mariusor 3y agoErm... AWS migration? What am I missing? A DDoS on AWS would most likely bankrupt Drew and Sourcehut, even if Amazon managed to absorb the traffic. Not to mention the principial issues that they have with AWS in the first place.
- internetter 3y agoAlso, the fact sourcehut does not baby users about git send-email means that collaborative development happening on the platform is uniquely posed to continue
- colinsane 3y agosure, i can `git send-email` to the particular developers i've collaborated with before, but by default my send-email goes to just the list-serve at `~user/project@lists.sr.ht`. pulling out `nc lists.sr.ht 25` seems to show that it's also offline. so, yes for really important stuff i could get patches through to projects i care about. but in practice the handful of projects i'm involved in on sr.ht seem to be mostly stalled.
- scopeh 3y agoFeel bad for the team to have to deal with this, but I have to say its a great example of how to communicate with your customers so hats off to you!
- deleted 3y ago[deleted]
- Phelinofist 3y agoSad to see, best of luck to them
- pierat 3y ago[flagged]
- monero-xmr 3y agoReminder that source hut makes a value judgement on your source code and will ban repositories they don't like https://news.ycombinator.com/item?id=33403780 https://news.ycombinator.com/item?id=33403780
- mwcampbell 3y agoAnd that's a good thing. A bootstrapped company doesn't have to be a slave to money.
- red_admiral 3y agoI followed that link to see what "don't like" means, and it means "no cryptocurrency projects". That's +1 to SourceHut in my book.
- crotchfire 3y ago[flagged]
- red_admiral 3y agoMy karma farming is working better than a lot of people's crypto farming these days :D
- plagiarist 3y agoWhy do you need to convert "goes against their principles" to "doesn't like" to make your argument? Is it because it is a weak argument without twisting words? They're allowed to decide what they'll host, and it looks like they're just being clear about a boundary. Unless they're playing favorites and allowing some crypto, what is the problem?
- monero-xmr 3y agoI don't want my source code vendor to tell me what source code I am allowed to write
- jordemort 3y agoOof, my heart goes out to them. My very first week at [large popular public code forge], we were attacked by [state level actor], probably for hosting something that [state] didn't like. In a way, SourceHut and Codeberg getting this kind of attention is an encouraging sign that these alternative forges are starting to gain traction.
- rapnie 3y agoI wonder how you could figure out it was [state]. Was there some clear threat made? A blackmail "do this, or else.."?
- meredydd 3y agoThis is likely a reference to GitHub, which was DoSed by the Chinese government for hosting a repository that was mirroring uncensored Western media on a domain that could not be blocked by the Great Firewall without hobbling the Chinese software industry. The attribution wasn't subtle - a substantial fraction of Baidu's ads/analytics traffic served to domestic Chinese users was rewritten to hammer that specific repository directly. NYT coverage at the time: https://www.nytimes.com/2015/03/31/technology/china-appears-to-attack-github-by-diverting-web-traffic.html https://www.nytimes.com/2015/03/31/technology/china-appears-...
- throwawaaarrgh 3y agoSo it turns out that running your own servers isn't always the best idea
- betimsl 3y ago>“what if the primary datacenter just disappeared tomorrow?” We ask this question of ourselves seriously, and make serious plans for what we’d do if this were to pass, and we are executing those plans now – though we had hoped that we would never have to. Respectfully, if you guys asked this question...how come you don't have a cluster slave as a replica in another data center where the whole thing is synced? A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.
- Rantenki 3y ago> A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users. If you want a Klugey non-production IoT solution that bodges up something really important THESE days, all the cool kids are using ESP32s. And as much as I think that would be a totally inappropriate solution for src.ht, I kinda wanna go make a "black-hole" switch for my office.
- sytse 3y agoHugOps to the Drew, the rest of the Sourcehut crew, and anyone else working on this. Compliments on the clear blog post.
- simonebrunozzi 3y ago> My name is Drew, I’m the founder of SourceHut and one of three SourceHut staff members working on the outage, alongside my colleagues Simon and Conrad. As you have noticed, SourceHut is down. I offer my deepest apologies for this situation. We have made a name for ourselves for reliability, and this is the most severe and prolonged outage we have ever faced. We spend a lot of time planning to make sure this does not happen, and we failed. We have all hands on deck working the problem to restore service as soon as possible. Drew, you're great, Simon and Conrad are great, you'll get through this, and you will be fine. Keep doing your great work, forever.
- 127361 3y agoI suspect a possible DDoS attack from something Kiwi Farms related, it seems they really don't like the guy over there? https://archive.is/yOObX https://archive.is/yOObX
- mardifoufs 3y agoIf kiwifarms ddosed everyone that they didn't like... plus, if anything KF has been surprisingly resilient to massive DDoS without any 3rd party protection service since they lost CloudFlare and other providers. So for this one specific thing, I think lessons can be learnt from kf on how to mitigate DDoS without 3rd party providers.
- 127361 3y agoThey developed their own proof-of-work system to rate limit requests, using SHA256, implementation details here: https://archive.is/dfBVN https://archive.is/dfBVN They also have servers in multiple countries. Some of these servers being blackholed by Cogent, by the way.