5 ms·

I'm still left not quite certain what would happen if they were hit with another L3 DDOS tomorrow. That said I'm very happy to use Sourcehut and I think they'l

I'm still left not quite certain what would happen if they were hit with another L3 DDOS tomorrow. That said I'm very happy to use Sourcehut and I think they'll overcome these challenges over time. They seem to have the staying power.

frakkingcylons 3y ago
They’re on OVH now and should have protection from it by virtue of being on their network now. > However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.
- treesknees 3y ago
  I'd consider it mostly protected, because no their servers are not on OVH, just a single box performing front-facing NAT/proxy essentially. The attacker now just needs to find the "secret" production subnet and attack it directly instead of through the front-facing NAT addresses.
  - frakkingcylons 3y ago
    That's true. I guess it'd be more accurate to say they're on OVH's network, not necessarily their compute and other infrastructure.
  - bombcar 3y ago
    That is very easy to mitigate, because you null route the production subnet except for a VPN that only can be reached by the proxy. You can even VPN over a completely different IPv6 route. They could still try to knock your entire datacenter offline but that is much harder.
    - treesknees 3y ago
      That's not much harder, that's exactly what happened to them in the initial attack. You're still depending on either a "secret" IPv6 network, or your upstream provider performing some source-based routing to only route packets from the VPN connection. I doubt that's available to a simple colo customer.
makeworld 3y ago
My reading is that OVH would handle it.
mrled 3y ago
I am really curious if the DDOS tried to follow them to the new infra and failed to cause an outage or not. Apparently the perpetrator noticed when they got Cogent to narrow the null route, but the blog post notes they still can't access the original subnet in that datacenter. Are they still trying to knock Sourcehut offline? Is the DDOS still pointing at now deprecated infra for some reason?
- caboteria 3y ago
  > At about 06:30 UTC the following morning, the DDoS escalated and broadened its targets to include other parts of our PHL subnet. In response, our colocation provider null routed our subnet once again. This subnet has been unreachable ever since.
  - mrled 3y ago
    Right, that's expanding to the rest of the subnet in their old DC. They've since migrated to the new DC with new countermeasures. Did the DDOS follow and the countermeasures are working? Or if it didn't follow, why not? There's also the question of whether the DDOS is still even trying the old infrastructure. The post says it's unreachable, but that would be true if the null route hadn't been removed yet.
    - drewdevault 3y ago
      Yes, the DDoS followed us to networks with countermeasures, and yes, the countermeasures worked. We don't want to disclose too much about that, though.
- wpm 3y ago
  When they switched DNS over to point to the AMS datacenter, the DDOS attack followed it until it got smacked down by the OVH NAT.

frakkingcylons 3y ago

They’re on OVH now and should have protection from it by virtue of being on their network now. > However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.

treesknees 3y ago
I'd consider it mostly protected, because no their servers are not on OVH, just a single box performing front-facing NAT/proxy essentially. The attacker now just needs to find the "secret" production subnet and attack it directly instead of through the front-facing NAT addresses.
- frakkingcylons 3y ago
  That's true. I guess it'd be more accurate to say they're on OVH's network, not necessarily their compute and other infrastructure.
- bombcar 3y ago
  That is very easy to mitigate, because you null route the production subnet except for a VPN that only can be reached by the proxy. You can even VPN over a completely different IPv6 route. They could still try to knock your entire datacenter offline but that is much harder.
  - treesknees 3y ago
    That's not much harder, that's exactly what happened to them in the initial attack. You're still depending on either a "secret" IPv6 network, or your upstream provider performing some source-based routing to only route packets from the VPN connection. I doubt that's available to a simple colo customer.

makeworld 3y ago

My reading is that OVH would handle it.

mrled 3y ago

I am really curious if the DDOS tried to follow them to the new infra and failed to cause an outage or not. Apparently the perpetrator noticed when they got Cogent to narrow the null route, but the blog post notes they still can't access the original subnet in that datacenter. Are they still trying to knock Sourcehut offline? Is the DDOS still pointing at now deprecated infra for some reason?

caboteria 3y ago
> At about 06:30 UTC the following morning, the DDoS escalated and broadened its targets to include other parts of our PHL subnet. In response, our colocation provider null routed our subnet once again. This subnet has been unreachable ever since.
- mrled 3y ago
  Right, that's expanding to the rest of the subnet in their old DC. They've since migrated to the new DC with new countermeasures. Did the DDOS follow and the countermeasures are working? Or if it didn't follow, why not? There's also the question of whether the DDOS is still even trying the old infrastructure. The post says it's unreachable, but that would be true if the null route hadn't been removed yet.
  - drewdevault 3y ago
    Yes, the DDoS followed us to networks with countermeasures, and yes, the countermeasures worked. We don't want to disclose too much about that, though.
wpm 3y ago
When they switched DNS over to point to the AMS datacenter, the DDOS attack followed it until it got smacked down by the OVH NAT.