3 ms·

Does anyone know details on the throughput or packets per second? This smells so much of gross negligence on the part of Dyn executives and all the unicorn web

by kev009 10y ago

Does anyone know details on the throughput or packets per second? This smells so much of gross negligence on the part of Dyn executives and all the unicorn web executive teams for single sourcing to me. I present as a counterexample and encourage people to go research the architecture of Verisign. I attended a talk by Verisign, which runs .com and .org as well as root servers. They are a constant DDOS target. They are necessarily a single entity of failure and appointed by ICANN to perform these services for the most common TLD in the world. If they mess up, they will probably lose that status. They've had over 16 years of uptime. Every layer of the stack is dual or triple sourced. Two server makers, two generations, two router vendors, two switch vendors, two network architectures, POP diversity, peering diversity. Services and capacity always added in pairs. Two separate NOC and Ops teams. FreeBSD, Linux and Solaris. nsd, bind, and internally developed userspace on Netmap. Code upgrade deployed in halves. Everything is structured for at least a full halving for The Big One zero day I don't think we've really seen yet. They were doing 10gbps of DNS on a single commodity server 3 years ago. This makes it easy to absorb any DDOS and gradually clamp it at the public peering points. The above should be standard operations structure for any breakaway success web business. It's not that hard, but you have to claw the charlatans out of the management chain and put in professionals that take the career seriously. Professionals. What's really appalling is hearing some unicorn web biz uses one cloud vendor, one service provider like Dyn. This is absolutely trivial stuff to multi-source and has almost no effect on OpEx. Last I heard from a Dyn insider was they were eliminating FreeBSD and had a brain drain a few years ago. Overall pretty unsurprising outcome. Internet reliability no more complex and certainly much cheaper than analogous critical infrastructure like electrical generation for a region. I am constantly disappointed in this field for its lack of recognizing and installing professionals into management. We've been doing Internet architecture in modern form for 30 years. Get your shit together.

detaro 10y ago
What kind of single-sourcing do you think was an issue on friday/how does double-sourcing help against pure DDOS attacks? I haven't read anything about a specific bug being exploited?
- kev009 10y ago
  It's hard to know until/unless they are fully transparent. It was almost certainly inside plant resource exhaustion, otherwise you wouldn't need an IoT botnet to cause it. Multisourcing gives you at least path diversity in this case. Hopefully application software performance as well. This is handled in the Verisign example by basic mechanical sympathy; when your machines can push 10gbps (probably 4x that in the years since the talk), it is economically silly not to simply eat the DDOS. Eating a DDOS is always the first mark of maturity. You can then improve with additional network intelligence. The most common way is with using sample based packet analysis. You match an attack signature and feed that into your gear at the peering points using BGP with blackhole destination. You are now filtering at ASIC speed, terabits per second on modern chassis switch or router.
  - deleted 10y ago
    [deleted]
  - detaro 10y ago
    I re-read your initial comment and apparently I misinterpreted what you wrote, so thanks for the high-quality answer despite this. (You didn't explicitly call out Dyn for not double-sourcing, but their customers, and only used it as an example for good practices in the Verisign example. Since you mentioned it lot, I gave it to much weight and read it as "Dyn failed because they didn't double-source", for which I see no evidence.)
youeeeeeediot 10y ago
So much this - but even worse - a particular unicorn who's primary business is identity management. DNS glue records ALL point to AWS/Route53, actual zone file NS records ALL point to Dyn. I looked at that and went WTF were you thinking?!?! It's one thing to give yourself a SPOF, but why would you give yourself 2 SPOFs?
drieddust 10y ago
Top Management don't care because they do not have a downside. By optimizing the cost they can get additional bonuses. Even if they sink the company, they still walk away rich and ready to screw someone else. Middle manager are too busy to save their own asses to ask questions. If one off poor guy musters courage to speak up, they are immediately shut down in the name of falling in line. This is the usual response from top management when they don'have an answer. "This isn't a Democracy."
- kev009 10y ago
  This mirrors my experience. And for public companies, the Board of Directors are not attuned to these outcomes. For the most part B2B customers are in the same boat so everybody feigns an uncomfortable laugh, some moderate outrage or displeasure, put out a silly postmortem without intent to rectify the culture that enabled it, and continue collecting an outsized salary. I adore computers, but I really wish I could lateral into some career like construction management where the layman is much more aware of success and failure, who is good and who is bad at their profession.
- deleted 10y ago
  [deleted]
makmanalp 10y ago
Where are you getting all this? I have a thought - perhaps it's not that everyone but you and your favourite organization is automatically incompetent! Perhaps there are reasons people do things! Perhaps there is more to something than what you can immediately glean! You could wait to learn more to pass judgement, but it seems you don't feel like doing that. > they were eliminating FreeBSD and had a brain drain a few years ago. Overall pretty unsurprising outcome. Thus they must be incompetent, because in their narrow minded foolishness, they don't understand that it is the Superior Operating System. There could be no other reason to do what they did. It couldn't have made sense for some other reason. Right? I also appreciate the BSDs for what they are, but this is so condescending. This whole post is so condescending. > Internet reliability no more complex and certainly much cheaper than analogous critical infrastructure like electrical generation for a region Not if you're trying to also make it cost effective? Electrical generation has well understood usage patterns, hasn't grown that much in the last few decades and doesn't have the kind of demand spikes that happen on the internet. Plus, power companies are mostly monopolies and charge whatever they want. It's an apples and oranges comparison. Also, electrical grids fail from time to time too! See the northeast blackout of 2003. Verisign has a similar monopoly to power companies and charges a huge tax margin on domains for all the services it provides. Without arguing about whether this is good or bad, we can say that Verisign is in effect pre-subsidized to provide however complex / redundant / reliable a service they want to provide. edit: Here's Krebs' breakdown of the attack: https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/ https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twit... Looks like OVH, the latest mirai attack, was around 1.2tbps. That's nothing to scoff at.
- kev009 10y ago
  Where am I getting what? My argument is that weak leadership causes the problems we see. You sound personally affronted because the I am pointing out a view of the industry that is not rosy glasses where we are all geniuses and Silicon Valley inspired culture solves all the hard problems with slam dunks. These problems are preventable and the tech industry is actually a wasteland of poor understanding and repeating organizational and managerial failure. These are ignored because everyone else is as almost as bad, and most people can't judge the good from the bad. This isn't cutting edge research, you carbon copy a competent company's design and they will probably explain it publicly if you ask nicely. There are forums like IETF and NANOG where you can exchange these practices, and professionals do that to build the profession. Swap out BSD with anything you please, just don't single source when you've hit breakout success and revenue loss from failure is material. I'm not a zealot, I even think WinNT is a pretty nice kernel and could fit into this picture with the new Microsoft. The point is you aren't holding up whole digit numbers of worldwide GDP on one kernel with a very poor security track record. Every modern widely used kernel applies to this statement. This is insane, I hope you can take that away from this thread at least. The cost of designing against common mode failure is inconsequential when properly budgeted against cost of actual failure. Any company in the Fortune 500, non-tech included, could afford to create a DDOS-proof DNS and web platform. Luckily they shouldn't have to if the service providers acted more like Verisign. And it scales down too, it would cost at most $100s of dollars a year for a single person company to multisource cloud providers, but likely it will be a wash since IAAS is most commonly usage billing.
dmourati 10y ago
Two sources claim 1.2 Tbps. http://hosted.ap.org/dynamic/stories/D/DISRUPTIVE_CYBERATTACK?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT&CTIME=2016-10-21-17-26-28 http://hosted.ap.org/dynamic/stories/D/DISRUPTIVE_CYBERATTAC... http://www.cbc.ca/news/technology/hackers-ddos-attacks-1.3817392 http://www.cbc.ca/news/technology/hackers-ddos-attacks-1.381...

by kev009 10y ago

detaro 10y ago

What kind of single-sourcing do you think was an issue on friday/how does double-sourcing help against pure DDOS attacks? I haven't read anything about a specific bug being exploited?

kev009 10y ago
It's hard to know until/unless they are fully transparent. It was almost certainly inside plant resource exhaustion, otherwise you wouldn't need an IoT botnet to cause it. Multisourcing gives you at least path diversity in this case. Hopefully application software performance as well. This is handled in the Verisign example by basic mechanical sympathy; when your machines can push 10gbps (probably 4x that in the years since the talk), it is economically silly not to simply eat the DDOS. Eating a DDOS is always the first mark of maturity. You can then improve with additional network intelligence. The most common way is with using sample based packet analysis. You match an attack signature and feed that into your gear at the peering points using BGP with blackhole destination. You are now filtering at ASIC speed, terabits per second on modern chassis switch or router.
- deleted 10y ago
  [deleted]
- detaro 10y ago
  I re-read your initial comment and apparently I misinterpreted what you wrote, so thanks for the high-quality answer despite this. (You didn't explicitly call out Dyn for not double-sourcing, but their customers, and only used it as an example for good practices in the Verisign example. Since you mentioned it lot, I gave it to much weight and read it as "Dyn failed because they didn't double-source", for which I see no evidence.)

youeeeeeediot 10y ago

So much this - but even worse - a particular unicorn who's primary business is identity management. DNS glue records ALL point to AWS/Route53, actual zone file NS records ALL point to Dyn. I looked at that and went WTF were you thinking?!?! It's one thing to give yourself a SPOF, but why would you give yourself 2 SPOFs?

drieddust 10y ago

Top Management don't care because they do not have a downside. By optimizing the cost they can get additional bonuses. Even if they sink the company, they still walk away rich and ready to screw someone else. Middle manager are too busy to save their own asses to ask questions. If one off poor guy musters courage to speak up, they are immediately shut down in the name of falling in line. This is the usual response from top management when they don'have an answer. "This isn't a Democracy."

kev009 10y ago
This mirrors my experience. And for public companies, the Board of Directors are not attuned to these outcomes. For the most part B2B customers are in the same boat so everybody feigns an uncomfortable laugh, some moderate outrage or displeasure, put out a silly postmortem without intent to rectify the culture that enabled it, and continue collecting an outsized salary. I adore computers, but I really wish I could lateral into some career like construction management where the layman is much more aware of success and failure, who is good and who is bad at their profession.
deleted 10y ago
[deleted]

makmanalp 10y ago

Where are you getting all this? I have a thought - perhaps it's not that everyone but you and your favourite organization is automatically incompetent! Perhaps there are reasons people do things! Perhaps there is more to something than what you can immediately glean! You could wait to learn more to pass judgement, but it seems you don't feel like doing that. > they were eliminating FreeBSD and had a brain drain a few years ago. Overall pretty unsurprising outcome. Thus they must be incompetent, because in their narrow minded foolishness, they don't understand that it is the Superior Operating System. There could be no other reason to do what they did. It couldn't have made sense for some other reason. Right? I also appreciate the BSDs for what they are, but this is so condescending. This whole post is so condescending. > Internet reliability no more complex and certainly much cheaper than analogous critical infrastructure like electrical generation for a region Not if you're trying to also make it cost effective? Electrical generation has well understood usage patterns, hasn't grown that much in the last few decades and doesn't have the kind of demand spikes that happen on the internet. Plus, power companies are mostly monopolies and charge whatever they want. It's an apples and oranges comparison. Also, electrical grids fail from time to time too! See the northeast blackout of 2003. Verisign has a similar monopoly to power companies and charges a huge tax margin on domains for all the services it provides. Without arguing about whether this is good or bad, we can say that Verisign is in effect pre-subsidized to provide however complex / redundant / reliable a service they want to provide. edit: Here's Krebs' breakdown of the attack: https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/ https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twit... Looks like OVH, the latest mirai attack, was around 1.2tbps. That's nothing to scoff at.

kev009 10y ago
Where am I getting what? My argument is that weak leadership causes the problems we see. You sound personally affronted because the I am pointing out a view of the industry that is not rosy glasses where we are all geniuses and Silicon Valley inspired culture solves all the hard problems with slam dunks. These problems are preventable and the tech industry is actually a wasteland of poor understanding and repeating organizational and managerial failure. These are ignored because everyone else is as almost as bad, and most people can't judge the good from the bad. This isn't cutting edge research, you carbon copy a competent company's design and they will probably explain it publicly if you ask nicely. There are forums like IETF and NANOG where you can exchange these practices, and professionals do that to build the profession. Swap out BSD with anything you please, just don't single source when you've hit breakout success and revenue loss from failure is material. I'm not a zealot, I even think WinNT is a pretty nice kernel and could fit into this picture with the new Microsoft. The point is you aren't holding up whole digit numbers of worldwide GDP on one kernel with a very poor security track record. Every modern widely used kernel applies to this statement. This is insane, I hope you can take that away from this thread at least. The cost of designing against common mode failure is inconsequential when properly budgeted against cost of actual failure. Any company in the Fortune 500, non-tech included, could afford to create a DDOS-proof DNS and web platform. Luckily they shouldn't have to if the service providers acted more like Verisign. And it scales down too, it would cost at most $100s of dollars a year for a single person company to multisource cloud providers, but likely it will be a wash since IAAS is most commonly usage billing.

dmourati 10y ago

Two sources claim 1.2 Tbps. http://hosted.ap.org/dynamic/stories/D/DISRUPTIVE_CYBERATTACK?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT&CTIME=2016-10-21-17-26-28 http://hosted.ap.org/dynamic/stories/D/DISRUPTIVE_CYBERATTAC... http://www.cbc.ca/news/technology/hackers-ddos-attacks-1.3817392 http://www.cbc.ca/news/technology/hackers-ddos-attacks-1.381...