6 ms·

Skype Outage – I run a supernode and here’s what I think happened

99 points by jdupree 16y ago

st3fan 16y ago
I seriously doubt the Skype supernodes actually run in regular clients on people's desktops. I'm pretty sure these are special servers placed strategically on the net. If you see thousands of connections on your Skype at home then there probably is some weird P2P problem going on.
- zb 16y ago
  I'm sorry to be the one who has to point this out, but you have no idea what you are talking about. http://www.skype.com/intl/en-us/security/universities/ http://www.skype.com/intl/en-us/security/universities/
eps 16y ago
Well, the big question is not if Skype went down becase lots of supernodes crashed. The question is why they crashed, and if the trigger was external. There are messages on p2p-hackers list arguing just that: I have evidence that suggests this is being done by a ddos attack on the supernodes' object list cmd parameter.
Groxx 16y ago
A wonderful reason to have pre-release testers for every piece of software that communicates with other software, any time you plan to push updates. With pulls, your early updaters are your test beds; with push, you have to create your testing groups.
- Daniel_Newby 16y ago
  Push software is even better about testing: you can forcibly upgrade 1% of installs, monitor them intensely, and let the (non-upgraded) system executive roll back the version if something bad happens. The system executive can even automatically roll back if it loses contact with the vendor, in case the upgrade goes so horribly wrong that it brings down networking.
- guelo 16y ago
  The problem with a big adaptive p2p network is that it is close to impossible to test under real production loads.
  - pyre 16y ago
    They could have setup their 'mega-supernodes' running the new client in a production environment, so that if it crashed horribly, it wasn't a client machine that failed (or their entire network). Upgrading (relatively) untested software network-wide on what is essentially their critical infrastructure is bad news. If this didn't happen now, it would happen sometime. It was just a question of when.
  - jrockway 16y ago
    My guess is that the average supernode is on something like 6M/1.5M DSL. This is easy to simulate. Now, if most of your nodes are on machines directly connected to 10GbE, then that's a problem. But most Skype nodes aren't. (I do imagine there are a few connected to 100M+ connections. But only a few.)
    - ohashi 16y ago
      I imagine quite a lot in countries like Sweden. I had 100Mbit in my apartment for cheaper than my shitty comcast which barely breaks 5.
buro9 16y ago
Just checked my Windows event logs... as I also run a supernode and yup, one Application Error at 2010-12-22 15:51:10 from Skype. Tech data shows version 5: Faulting application name: Skype.exe, version: 5.0.0.152, time stamp: 0x4cb31516 Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdbdf Exception code: 0xe0fafafa Fault offset: 0x0000b727 Faulting process id: 0xd20 Faulting application start time: 0x01cba1ea3b70b5ef Faulting application path: C:\Program Files (x86)\Skype\Phone\Skype.exe Faulting module path: C:\Windows\syswow64\KERNELBASE.dll Report Id: 4b2781d7-0de3-11e0-bbdc-005056c00008 I didn't upgrade voluntarily.
jallmann 16y ago
Nice to see the CEO of a large company that knows how to walk the talk. I've also seen him post on HN. Does Steve Ballmer know the difference between a micro and monolithic kernel?
- js3309 16y ago
  I am also surprised that a company as large as his, uses Skype instant messaging for their communication.
  - akalsey 16y ago
    We're a very distributed company, with some projects being worked on by people in a dozen locations, so some form of IM is needed. I'm a big XMPP proponent (and Voxeo acquired my xmpp-based company) and we do use XMPP for a lot of things inside Voxeo (like Phono, our jquery softphone: http://phono.com/ http://phono.com/). Every voxeo employee has an XMPP ID, but they don't tend to get used. People gravitate to Skype naturally. It's the user experience on the client side of XMPP that keeps people from using it extensively for internal communications. At any given time I've got 40+ group chats going on and we create and destroy group chats many times a week for specific needs. The ability to make a voice call to one or more participants of the chat is also a winner. The fact that everyone on the planet has a Skype ID also helps. We can easily pull partners and customers into chats as needed. They're already on Skype, so we don't need to teach them anything new. It's a lesson in usability, certainly. Create a client that allows users, without any assistance, to create groups, move seamlessly between text, voice, and video, and has a foolproof signup and setup process, and folks will use it.
- ra 16y ago
  I was thinking the same thing. What other big company CEO's would even think of investigating why their Skype was more jittery in some hotels, let alone actually know how to do it.
  - bl4k 16y ago
    probably busy running big companies
cookiecaper 16y ago
Maybe this will cause Skype to take Linux and Mac more seriously. If you've used the Windows version, you know almost all development has occurred there, whereas the Mac and Linux clients have looked the same for the last four or five years. The Linux client did have a major version bump about a year ago iirc, and that brought some needed features, but it was mostly the same. Also, hopefully this will teach Skype to do more with open-source. I really hope they open the client up. This bug may have been caught, and things would definitely have turned out differently if Skype ran freely on other platforms. Maybe someone could even factor out a "Skype server" instead of an exclusive policy of client supernodes. Even serious torrenters rent a server somewhere to host their torrents -- P2P doesn't have to be strictly consumer-level connection, and really shouldn't be.
- anon114 16y ago
  Or, maybe this will cause users to take Jabber/XMPP more seriously and stop using proprietary technology for corporate IM. I've worked at places where management is completely gaga over Skype and would push me to support it despite the fact that I had no ability to block spam, troubleshoot messaging problems or integrate our IM system into existing Asterisk, SSO, monitoring and collaboration solutions. IMHO, Openfire is far more flexible, extensible, secure, reliable and most importantly - manageable as a service. http://www.igniterealtime.org/projects/openfire/ http://www.igniterealtime.org/projects/openfire/ disclaimer: I do not work for Ignite and have no vested interest in their business.
  - cookiecaper 16y ago
    When an XMPP solution that's as easy to use as Skype comes out I'll be all for this. As it stands, even Google's video chat is much more difficult for end-users to install and use. I used to use Ekiga for video chat and it was much shoddier than Skype, constantly dropping calls, refusing to release the audio or video device so that we couldn't call back, and other serious bugs. Skype "just works". I really hope that someone comes up with a decent free software competitor, but it doesn't really exist at this point. The fastest way to solve this problem would be for Skype to become free software.
- 16y ago
billpg 16y ago
Are Skype relying on third parties with no contractual relationship for an essential component of the network?
- ars 16y ago
  Yup. But they do mention it when installing. You can also disable it. If you are not firewalled or NATd you are a supernode automatically - on the plus side you get better sound quality for voice calls. Normally the bandwidth used is pretty low since you are mostly forwarding text messages. When they said mega-supernodes they meant machines controlled by them that do nothing else, and are on high bandwidth connections. (My bet is lots of amazon instances.)
  - yellowbkpk 16y ago
    I've looked around for the ability to turn off supernode on Linux and Mac clients for a while but have come up empty. Do you know which section of the preferences/settings it's supposed to be in?
    - NikkiA 16y ago
      I don't think there's a UI element for it, but you have to add a registry key, see here: https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration#Disable_Supernode https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration...
      - pyre 16y ago
        How pray-tell, does a Windows registry entry disable super node functionality on Linux and/or MacOSX?
        neworbit 16y ago
        Well barring effective virtualization, booting into Windows disables supernode functionality on Windows and MacOSX in the same sense that a brownout does
_stephan 16y ago
I had the same issue: Shortly before Skype went completely offline, my Windows Skype client suddenly started crashing every few minutes. The Mac client didn't have that problem. Some bug in the new version is probably the likeliest explanation, but if someone deliberately attacked the Skype network, it would probably look similar, right?
- jvagner 16y ago
  ...or vice versa. a bug could easily look like an attack.
- Dylan16807 16y ago
  You're not going to get full client crashes without a major bug. Some amount of attack is possible but I've been getting crashes for a couple weeks now and skype should have fixed things.
schtono 16y ago
tl,dr; a forced software upgrade kept crashing all skype windows clients, taking down supernodes as well.
deleted 16y ago
[deleted]
hackermom 16y ago
I, too, run a Skype supernode - completely unvoluntarily, barring the possibility of something hidden behind an asterisk in the EULA. Does anyone know if there is a way to "opt out" of this? Everytime I start skype up it makes a million connections to everywhere and without even having a call on the line my measly 2.5 mbps upstream is permanently choked.
- citricsquid 16y ago
  https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration#Disable_Supernode https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration... Found this link lower down :-)
  - hackermom 16y ago
    Sorry, I should've mentioned that I'm an OS X user :) With that said, I've had a look-see in Skype's .plist and its various Application Support files, but oddly there's nothing "grep'able" to be found anywhere...
johndyer 16y ago
Nice write up, pretty interesting that Skype has not come out and said anything about WTF happened. Also ironic that their security blog's last posting it titled "The importance of updating" :) ( http://blogs.skype.com/garage/ http://blogs.skype.com/garage/ ) -John
bsg75 16y ago
http://arstechnica.com/software/news/2010/12/skype-brought-down-by-double-whammy-of-overloaded-servers-client-bugs.ars http://arstechnica.com/software/news/2010/12/skype-brought-d...

99 points by jdupree 16y ago

I seriously doubt the Skype supernodes actually run in regular clients on people's desktops. I'm pretty sure these are special servers placed strategically on the net. If you see thousands of connections on your Skype at home then there probably is some weird P2P problem going on.

zb 16y ago
I'm sorry to be the one who has to point this out, but you have no idea what you are talking about. http://www.skype.com/intl/en-us/security/universities/ http://www.skype.com/intl/en-us/security/universities/

eps 16y ago

Well, the big question is not if Skype went down becase lots of supernodes crashed. The question is why they crashed, and if the trigger was external. There are messages on p2p-hackers list arguing just that: I have evidence that suggests this is being done by a ddos attack on the supernodes' object list cmd parameter.

Groxx 16y ago

A wonderful reason to have pre-release testers for every piece of software that communicates with other software, any time you plan to push updates. With pulls, your early updaters are your test beds; with push, you have to create your testing groups.

Daniel_Newby 16y ago
Push software is even better about testing: you can forcibly upgrade 1% of installs, monitor them intensely, and let the (non-upgraded) system executive roll back the version if something bad happens. The system executive can even automatically roll back if it loses contact with the vendor, in case the upgrade goes so horribly wrong that it brings down networking.
guelo 16y ago
The problem with a big adaptive p2p network is that it is close to impossible to test under real production loads.
- pyre 16y ago
  They could have setup their 'mega-supernodes' running the new client in a production environment, so that if it crashed horribly, it wasn't a client machine that failed (or their entire network). Upgrading (relatively) untested software network-wide on what is essentially their critical infrastructure is bad news. If this didn't happen now, it would happen sometime. It was just a question of when.
- jrockway 16y ago
  My guess is that the average supernode is on something like 6M/1.5M DSL. This is easy to simulate. Now, if most of your nodes are on machines directly connected to 10GbE, then that's a problem. But most Skype nodes aren't. (I do imagine there are a few connected to 100M+ connections. But only a few.)
  - ohashi 16y ago
    I imagine quite a lot in countries like Sweden. I had 100Mbit in my apartment for cheaper than my shitty comcast which barely breaks 5.

buro9 16y ago

Just checked my Windows event logs... as I also run a supernode and yup, one Application Error at 2010-12-22 15:51:10 from Skype. Tech data shows version 5: Faulting application name: Skype.exe, version: 5.0.0.152, time stamp: 0x4cb31516 Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdbdf Exception code: 0xe0fafafa Fault offset: 0x0000b727 Faulting process id: 0xd20 Faulting application start time: 0x01cba1ea3b70b5ef Faulting application path: C:\Program Files (x86)\Skype\Phone\Skype.exe Faulting module path: C:\Windows\syswow64\KERNELBASE.dll Report Id: 4b2781d7-0de3-11e0-bbdc-005056c00008 I didn't upgrade voluntarily.

jallmann 16y ago

Nice to see the CEO of a large company that knows how to walk the talk. I've also seen him post on HN. Does Steve Ballmer know the difference between a micro and monolithic kernel?

js3309 16y ago
I am also surprised that a company as large as his, uses Skype instant messaging for their communication.
- akalsey 16y ago
  We're a very distributed company, with some projects being worked on by people in a dozen locations, so some form of IM is needed. I'm a big XMPP proponent (and Voxeo acquired my xmpp-based company) and we do use XMPP for a lot of things inside Voxeo (like Phono, our jquery softphone: http://phono.com/ http://phono.com/). Every voxeo employee has an XMPP ID, but they don't tend to get used. People gravitate to Skype naturally. It's the user experience on the client side of XMPP that keeps people from using it extensively for internal communications. At any given time I've got 40+ group chats going on and we create and destroy group chats many times a week for specific needs. The ability to make a voice call to one or more participants of the chat is also a winner. The fact that everyone on the planet has a Skype ID also helps. We can easily pull partners and customers into chats as needed. They're already on Skype, so we don't need to teach them anything new. It's a lesson in usability, certainly. Create a client that allows users, without any assistance, to create groups, move seamlessly between text, voice, and video, and has a foolproof signup and setup process, and folks will use it.
ra 16y ago
I was thinking the same thing. What other big company CEO's would even think of investigating why their Skype was more jittery in some hotels, let alone actually know how to do it.
- bl4k 16y ago
  probably busy running big companies

cookiecaper 16y ago

Maybe this will cause Skype to take Linux and Mac more seriously. If you've used the Windows version, you know almost all development has occurred there, whereas the Mac and Linux clients have looked the same for the last four or five years. The Linux client did have a major version bump about a year ago iirc, and that brought some needed features, but it was mostly the same. Also, hopefully this will teach Skype to do more with open-source. I really hope they open the client up. This bug may have been caught, and things would definitely have turned out differently if Skype ran freely on other platforms. Maybe someone could even factor out a "Skype server" instead of an exclusive policy of client supernodes. Even serious torrenters rent a server somewhere to host their torrents -- P2P doesn't have to be strictly consumer-level connection, and really shouldn't be.

anon114 16y ago
Or, maybe this will cause users to take Jabber/XMPP more seriously and stop using proprietary technology for corporate IM. I've worked at places where management is completely gaga over Skype and would push me to support it despite the fact that I had no ability to block spam, troubleshoot messaging problems or integrate our IM system into existing Asterisk, SSO, monitoring and collaboration solutions. IMHO, Openfire is far more flexible, extensible, secure, reliable and most importantly - manageable as a service. http://www.igniterealtime.org/projects/openfire/ http://www.igniterealtime.org/projects/openfire/ disclaimer: I do not work for Ignite and have no vested interest in their business.
- cookiecaper 16y ago
  When an XMPP solution that's as easy to use as Skype comes out I'll be all for this. As it stands, even Google's video chat is much more difficult for end-users to install and use. I used to use Ekiga for video chat and it was much shoddier than Skype, constantly dropping calls, refusing to release the audio or video device so that we couldn't call back, and other serious bugs. Skype "just works". I really hope that someone comes up with a decent free software competitor, but it doesn't really exist at this point. The fastest way to solve this problem would be for Skype to become free software.
16y ago

billpg 16y ago

Are Skype relying on third parties with no contractual relationship for an essential component of the network?

ars 16y ago
Yup. But they do mention it when installing. You can also disable it. If you are not firewalled or NATd you are a supernode automatically - on the plus side you get better sound quality for voice calls. Normally the bandwidth used is pretty low since you are mostly forwarding text messages. When they said mega-supernodes they meant machines controlled by them that do nothing else, and are on high bandwidth connections. (My bet is lots of amazon instances.)
- yellowbkpk 16y ago
  I've looked around for the ability to turn off supernode on Linux and Mac clients for a while but have come up empty. Do you know which section of the preferences/settings it's supposed to be in?
  - NikkiA 16y ago
    I don't think there's a UI element for it, but you have to add a registry key, see here: https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration#Disable_Supernode https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration...
    - pyre 16y ago
      How pray-tell, does a Windows registry entry disable super node functionality on Linux and/or MacOSX?
      - neworbit 16y ago
        Well barring effective virtualization, booting into Windows disables supernode functionality on Windows and MacOSX in the same sense that a brownout does

_stephan 16y ago

I had the same issue: Shortly before Skype went completely offline, my Windows Skype client suddenly started crashing every few minutes. The Mac client didn't have that problem. Some bug in the new version is probably the likeliest explanation, but if someone deliberately attacked the Skype network, it would probably look similar, right?

jvagner 16y ago
...or vice versa. a bug could easily look like an attack.
Dylan16807 16y ago
You're not going to get full client crashes without a major bug. Some amount of attack is possible but I've been getting crashes for a couple weeks now and skype should have fixed things.

schtono 16y ago

tl,dr; a forced software upgrade kept crashing all skype windows clients, taking down supernodes as well.

deleted 16y ago

[deleted]

hackermom 16y ago

I, too, run a Skype supernode - completely unvoluntarily, barring the possibility of something hidden behind an asterisk in the EULA. Does anyone know if there is a way to "opt out" of this? Everytime I start skype up it makes a million connections to everywhere and without even having a call on the line my measly 2.5 mbps upstream is permanently choked.

citricsquid 16y ago
https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration#Disable_Supernode https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration... Found this link lower down :-)
- hackermom 16y ago
  Sorry, I should've mentioned that I'm an OS X user :) With that said, I've had a look-see in Skype's .plist and its various Application Support files, but oddly there's nothing "grep'able" to be found anywhere...

johndyer 16y ago

Nice write up, pretty interesting that Skype has not come out and said anything about WTF happened. Also ironic that their security blog's last posting it titled "The importance of updating" :) ( http://blogs.skype.com/garage/ http://blogs.skype.com/garage/ ) -John

bsg75 16y ago

http://arstechnica.com/software/news/2010/12/skype-brought-down-by-double-whammy-of-overloaded-servers-client-bugs.ars http://arstechnica.com/software/news/2010/12/skype-brought-d...