7 ms·

Skype brief post-mortem

76 points by LiveTheDream 16y ago

gojomo 16y ago
But why did a "large umber of supernodes" today get "taken offline by a problem affecting some versions of Skype"? Synchronized bug in that software – some sort of clock overflow, or update from Skype gone awry? Or a flaw in that software discovered and exploited by others? There must be a lot more to this story.
terryjsmith 16y ago
This seems like a nightmare situation for P2P company. Your super nodes get knocked offline and there's no way to force update them until they come back online, which is sporadic at best (based on my own experience today), which means either manually updating them or completely re-seeding your network (which seems to be the route they're going -- though so far it doesn't appear to be going all that well). As other people have stated on other articles posted here, no discredit to them for having some downtime (I'm sure they're working round the clock on it), but I have to wonder if this could have been avoided with a phased roll out of new version to the super nodes.
- dennisgorelik 16y ago
  Why is it a problem to bring supernode back online?
  - barrkel 16y ago
    I can imagine it would self-DDoS from lack of peer supernodes to share the load.
    - dennisgorelik 16y ago
      But at least it would successfully reply to at least some of requests, right? And would still keep trying to serve the requests, right? So if sufficient number of supernodes is brought back online then the problems should disappear.
      - terryjsmith 16y ago
        When this happens to our websites (when all servers go down), we need to rate limit and/or shut down traffic at the load balancer level as we bring things back online, otherwise everything just continues to get swamped and goes right back down. This would be nearly impossible in a P2P network and coordinating it between locations would be an even bigger nightmare. I imagine this is why turning on an entirely new network is a more viable option for them.
        dennisgorelik 16y ago
        Why would overloaded server go down? Shouldn't it simply stop serving incoming requests if it's overloaded?
        barrkel 16y ago
        Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?
        dennisgorelik 16y ago
        There is a meaningful distinction between a server that serves 1% of requests and server that is down. I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge. Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node? Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?
        terryjsmith 16y ago
        It handling 1% of connections would assume that the network was the bottleneck. You are much more likely to use up the CPU, RAM, or other resources before hitting the maximum number of available sockets. In which case things are swapping or waiting for available CPU time and each individual requests becomes seconds or tens of seconds to get handled. For all intents and purposes, that machine is dead. As to your second point, I think you are right. I assume that is what the mega-supernode is: a network of machines who's resources are as high as can be to handle all of the connections and try to beat the bottlenecks.
    - tommi 16y ago
      That can be avoided by altering the logic by which supernodes are queried. Not all nodes must try all supernodes.
      - terryjsmith 16y ago
        Since it seems a good chunk of the super nodes went down, I imagine every running unconnected Skype instance is checking all of the super nodes it can find (or actively searching for them). Changing the logic at this point isn't really an option for them.
        tommi 16y ago
        Well, even if we were to talk about this point, it actually might be. I have no knowledge of Skype architecture, so I'm totally guessing here. But, those unconnected Skype instances have to have some kind of directory of supernodes: either dynamic or static. If it's dynamic, that is from Skype's servers, it can be affected.
        terryjsmith 16y ago
        That's fair, but even if it is dynamic, I doubt it's checked on a regular basis; I doubt it's a scenario they've even considered before now. This would be on par with the DNS root servers changing and somehow letting everyone know about it on the fly.
        tommi 16y ago
        I tend to disagree. Error handling exactly in such cases where you have to bring the network online from a complete halt or other catastrophe is at the very heart of the architecture of these systems. These scenarios should and are at the minds of the architects and coders.
        terryjsmith 16y ago
        That's a lot easier when it's your network. Skype is obviously designed to be used in "hostile" environments; super nodes are likely expected to go away and come back on a regular basis. Likewise, I don't think it's an unreasonable expectation that there will be a given number of nodes online at any given time. If they all disappear, I doubt there's a contingency plan for that.
  - dedward 16y ago
    Supernodes are just guys like you and me with the right amount of bandwidth and the right network connection - if bad code or bad data exposed an existing bug, causing the supernodes in the meshed p2p network to go down - you have a chicken and egg problem. Skype doens't own the supernodes.... skype works because it uses the user's own resources to help route calls for other users.
    - dennisgorelik 16y ago
      So the bug is in Skype client code that is responsible for handling supernode behavior. The fix should probably be about developing deploying new version of Skype client. Such fix can take a week. Especially considering that engineering might be not in a good shape in Skype few years after acquisition from original founders.
- ddlatham 16y ago
  There's an important lesson here for P2P system design. As in all systems, things will go wrong, whether it's your own bug or someone else's. Your system needs to not only be stable in the steady state, but to be able to return to the steady state when something interrupts it.
  - terryjsmith 16y ago
    Absolutely. We designed a P2P file system a few years ago and actually gleaned a good number of tricks from Skype for dealing with NATs and constructing your network in general. Dropbox (and many others I'm sure) have all said the same thing: you need to design your system to function in the most hostile conditions you can think of; for Skype this seems especially devastating because so many components are beyond their control.
    - amackera 16y ago
      Do you know of any resources where I could learn about Skype's network structure? I'm very interested in distributed systems, especially ones with the scope of Skype's network.
      - terryjsmith 16y ago
        Sure, so there were a lot of them that we compiled, here are the ones and topics I remember. Protocol: we used a custom psuedo-TCP protocol built on UDP based on libjingle from Google (used in Google Talk): http://code.google.com/apis/talk/libjingle/file_share.html http://code.google.com/apis/talk/libjingle/file_share.html. Libjingle's filesharing itself was a decent resource for learning some P2P stuff as well. NATing and routing: we used Skype's UDP hole punching: http://www.h-online.com/security/features/How-Skype-Co-get-round-firewalls-747314.html http://www.h-online.com/security/features/How-Skype-Co-get-r.... Skype configures each install to be a supernode by default and then disables it based on certain criteria (low speeds, behind a NAT/firewall, etc.). Discovery of other network nodes is an entire subject unto itself and a lot of other network discovery protocols are well documented; our system used a central server to track up-time statistically and map clients to one another based on finding a good fit. Happy to answer any questions here or at my e-mail address in profile.
ddlatham 16y ago
Under normal circumstances, there are a large number of supernodes available. Unfortunately, today, many of them were taken offline by a problem affecting some versions of Skype. So, what was the problem affecting versions of Skype?
- AgentConundrum 16y ago
  That's what I've been wondering all day. I didn't have a problem with Skype being down; I had a problem with Skype having exploded this morning, screaming about uncaught exceptions as it did. Your quote was an "aha" moment for me.
  - igravious 16y ago
    So did I, were you travelling at the time? I was in an airport and I figured that it is the weird you have an internet connection but it only takes you to a pay per usage landing page and no other page kind of problem that made Skype barf. I happened to a Java network client too at the same time. I can send you logs Skype!
    - AgentConundrum 16y ago
      Nope, not traveling. I was just sitting at home and it exploded.
- code_duck 16y ago
  I doubt that they'd want to tell everyone, since apparently triggering this problem effectively creates a DOS attack against the entire network.
zinssmeister 16y ago
does anyone know why skype picked German as the only translation of this message?
- mike4u2 16y ago
  Because Germans tend to complain the most if free stuff does not work ;-)
  - RP_Joe 16y ago
    Very funny.
  - preek 16y ago
    What free stuff? I have a paid subscription for international calls and my voicemail, my dear friend. Do you? If not, then stop mocking people you don't know. If you do, preferably also stop mocking people you don't know. Having said that, I have important business calls to do today and I'm certainly not going to throw money at my iPhone carrier. I hope Skype is coming back online for me, soon. But I have not yet started complaining anywhere, because Skype has offered me a great service and lots of savings in the last two years. But who am I to day, I'm just a grumpy German.
- corin_ 16y ago
  Obvious possibilities include German being the most used non-English language for Skype users, or that they're going to be adding other languages but their translators haven't yet written them. The only guess I have that isn't from an obvious reason is that Skype HQ is in Luxembourg, a small country right next to Germany. One of the three official languages there is German, and the other is Luxembourgish, which is closely related to German.
- switch007 16y ago
  My awful Spanish translation: A principios de hoy, nos dimos cuenta de que el número de usarios conectado a Skype empezó caer. Esto no es normal ni esperado así que nos pusimos a investigarlo. Skype no es como un red de telefonía convencional ni red de mensajería instantánea - en cambio, para que todo vaya bien depende de millones de conexiones entre los computadores y los teléfonos. Unos de esas computadores se llaman "supernodes" - funcionan como una agenda de direcciones por Skype. Si quieres hablar con alguien y Skype no lo puede localizar inmediatamente, tu computador o tu móvil intentará localizar un "supernode" para descubrir la manera de contactarle. Normalmente hay un montón de supernodes disponibles. Hoy, por desgracia, muchos no eran conectados debido a una problema que afectaba algunos versiones de Skype. ¿Qué estamos haciendo para mejorar la situación? Nuestros ingenieros están haciendo nuevos "mega-supernodes" lo cuanto antes, que con un poco de suerte va a hacer que todo se normalice.
johndbritton 16y ago
"Our engineers are creating new ‘mega-supernodes’ as fast as they can" What a brilliant solution.
- tommi 16y ago
  It's not a solution, it's a hot-fix.
  - dennisgorelik 16y ago
    It's a hotfix that does not work [yet].
- terryjsmith 16y ago
  Yea, see the above comments. Bringing up individual nodes isn't an option as they would likely be overwhelmed and rate limiting traffic from all of the running instances in a P2P network would be near impossible. Bringing up an entirely new super node network, while insane, is probably their best option right now. Edit: I initially read your post as sarcastic, but I realize now it may not have been. It's definitely outside-the-box thinking and after considering their options, this certainly seems the most viable.
  - dennisgorelik 16y ago
    HackerNews does not allow deep threads, (http://news.ycombinator.com/item?id=2032770 http://news.ycombinator.com/item?id=2032770) so I reply here. Why would supernode that has CPU as a bottleneck stop serving _any_ requests? Say, CPU is able to serve a thousand requests per second. If supernode receives a million requests per second, then supernode should serve first thousand requests and then ignore the rest, right?
    - terryjsmith 16y ago
      Correct, it will stop serving requests (ignore the rest), but the end result is no different (in fact I imagine this is exactly what's happening). That system is dead to the world. According to Wikipedia in Q3 2009 [1] Skype has over 500 million users (Q1 2009 says 42M active at any time); any super node that is processing a thousand requests and then dies (or only processes one request a second) isn't helping anyone. [1] http://en.wikipedia.org/wiki/Skype#Usage_and_traffic http://en.wikipedia.org/wiki/Skype#Usage_and_traffic
swolchok 16y ago
This isn't really a post-mortem; they're still in progress on a fix.
- tommi 16y ago
  The title also refers the whole Skype being dead.
anonymous246 16y ago
Does anybody know if these supernodes are paid for by Skype or they're simply users' computers being commandeered by Skype (after getting click-through permission). The explanation doesn't ring true to me because AFAIK supernodes are used only if a direct P2P path cannot be established between caller and callee. What we're observing is the inability to sign in and see contacts' status. I always thought that was centralized. Are supernodes involved in signin?
- oiuytriuytr 16y ago
  In theory they can be regular users computers - but unless you have the only PC connected to some small outpost on the internet it's unlikely that it would be you. Generally it's either a skype or ISP owned machine
- terryjsmith 16y ago
  This article is a little old now but I imagine the basic technique is still the same: http://www.h-online.com/security/features/How-Skype-Co-get-round-firewalls-747314.html http://www.h-online.com/security/features/How-Skype-Co-get-r... As I understand it, for corporate firewalls or routers/firewalls configured to not allow "hole punching", Skype will use the supernode as a relay for all communications.
- piramida 16y ago
  The user database is centralized (from a talk that lead Skype engineer given couple of years ago), so I too wonder how would a traffic routing nodes affect signing in. I am sure non of the client code has their user database credentials... So supernodes can only prevent communication, calls, video from happening. So there's something else they are hiding. My Mac skype dies on start since yesterday (trying to sign-in), supernodes would have nothing to do with that.
viraptor 16y ago
This part is pretty cool, but quite subtle: "Earlier today, we noticed that the number of people online on Skype was falling". I mean that they noticed the number going down, not "people started raising tickets saying it doesn't work".
car 16y ago
How is this a post-mortem? The blog entry does not explain in any way what really happened.
bystanderr 16y ago
Reading the discussion here reminded me of the outage of AT&T's long-distance network back in 1991. After a bit of searching, I found an interesting-looking document at http://faqs.org/rfcs/rfc3439.html http://faqs.org/rfcs/rfc3439.html entitled "RFC 3439 - Some Internet Architectural Guidelines and Philosophy," which seems relevant here somehow. Though this document outlines considerations pertaining to complexity in Internet backbone architecture, perhaps the overall philosophical questions it poses and guidelines it offers could be instructive as regards the recent Skype meltdown. As regards the aforementioned 1991 failure of the AT&T long-distance system, which resulted in a service outage of about six hours, the document says: "The PSTN's SS7 control network provides an interesting example of what can go wrong with a tightly coupled complex system. Outages such as the well publicized 1991 outage of AT&T's SS7 demonstrates the phenomenon: the outage was caused by software bugs in the switches' crash recovery code. In this case, one switch crashed due to a hardware glitch. When this switch came back up, it (plus a reasonably probable timing event) caused its neighbors to crash When the neighboring switches came back up, they caused their neighbors to crash, and so on [NEUMANN] (the root cause turned out to be a misplaced 'break' statement; this is an excellent example of cross- layer coupling). This phenomenon is similar to the phase-locking of weakly coupled oscillators, in which random variations in sequence times plays an important role in system stability [THOMPSON]."

76 points by LiveTheDream 16y ago

But why did a "large umber of supernodes" today get "taken offline by a problem affecting some versions of Skype"? Synchronized bug in that software – some sort of clock overflow, or update from Skype gone awry? Or a flaw in that software discovered and exploited by others? There must be a lot more to this story.

terryjsmith 16y ago

This seems like a nightmare situation for P2P company. Your super nodes get knocked offline and there's no way to force update them until they come back online, which is sporadic at best (based on my own experience today), which means either manually updating them or completely re-seeding your network (which seems to be the route they're going -- though so far it doesn't appear to be going all that well). As other people have stated on other articles posted here, no discredit to them for having some downtime (I'm sure they're working round the clock on it), but I have to wonder if this could have been avoided with a phased roll out of new version to the super nodes.

dennisgorelik 16y ago
Why is it a problem to bring supernode back online?
- barrkel 16y ago
  I can imagine it would self-DDoS from lack of peer supernodes to share the load.
  - dennisgorelik 16y ago
    But at least it would successfully reply to at least some of requests, right? And would still keep trying to serve the requests, right? So if sufficient number of supernodes is brought back online then the problems should disappear.
    - terryjsmith 16y ago
      When this happens to our websites (when all servers go down), we need to rate limit and/or shut down traffic at the load balancer level as we bring things back online, otherwise everything just continues to get swamped and goes right back down. This would be nearly impossible in a P2P network and coordinating it between locations would be an even bigger nightmare. I imagine this is why turning on an entirely new network is a more viable option for them.
      - dennisgorelik 16y ago
        Why would overloaded server go down? Shouldn't it simply stop serving incoming requests if it's overloaded?
        barrkel 16y ago
        Is there a meaningful distinction between a server that doesn't serve requests, and a server that is down?
        dennisgorelik 16y ago
        There is a meaningful distinction between a server that serves 1% of requests and server that is down. I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge. Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node? Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?
        terryjsmith 16y ago
        It handling 1% of connections would assume that the network was the bottleneck. You are much more likely to use up the CPU, RAM, or other resources before hitting the maximum number of available sockets. In which case things are swapping or waiting for available CPU time and each individual requests becomes seconds or tens of seconds to get handled. For all intents and purposes, that machine is dead. As to your second point, I think you are right. I assume that is what the mega-supernode is: a network of machines who's resources are as high as can be to handle all of the connections and try to beat the bottlenecks.
  - tommi 16y ago
    That can be avoided by altering the logic by which supernodes are queried. Not all nodes must try all supernodes.
    - terryjsmith 16y ago
      Since it seems a good chunk of the super nodes went down, I imagine every running unconnected Skype instance is checking all of the super nodes it can find (or actively searching for them). Changing the logic at this point isn't really an option for them.
      - tommi 16y ago
        Well, even if we were to talk about this point, it actually might be. I have no knowledge of Skype architecture, so I'm totally guessing here. But, those unconnected Skype instances have to have some kind of directory of supernodes: either dynamic or static. If it's dynamic, that is from Skype's servers, it can be affected.
        terryjsmith 16y ago
        That's fair, but even if it is dynamic, I doubt it's checked on a regular basis; I doubt it's a scenario they've even considered before now. This would be on par with the DNS root servers changing and somehow letting everyone know about it on the fly.
        tommi 16y ago
        I tend to disagree. Error handling exactly in such cases where you have to bring the network online from a complete halt or other catastrophe is at the very heart of the architecture of these systems. These scenarios should and are at the minds of the architects and coders.
        terryjsmith 16y ago
        That's a lot easier when it's your network. Skype is obviously designed to be used in "hostile" environments; super nodes are likely expected to go away and come back on a regular basis. Likewise, I don't think it's an unreasonable expectation that there will be a given number of nodes online at any given time. If they all disappear, I doubt there's a contingency plan for that.
- dedward 16y ago
  Supernodes are just guys like you and me with the right amount of bandwidth and the right network connection - if bad code or bad data exposed an existing bug, causing the supernodes in the meshed p2p network to go down - you have a chicken and egg problem. Skype doens't own the supernodes.... skype works because it uses the user's own resources to help route calls for other users.
  - dennisgorelik 16y ago
    So the bug is in Skype client code that is responsible for handling supernode behavior. The fix should probably be about developing deploying new version of Skype client. Such fix can take a week. Especially considering that engineering might be not in a good shape in Skype few years after acquisition from original founders.
ddlatham 16y ago
There's an important lesson here for P2P system design. As in all systems, things will go wrong, whether it's your own bug or someone else's. Your system needs to not only be stable in the steady state, but to be able to return to the steady state when something interrupts it.
- terryjsmith 16y ago
  Absolutely. We designed a P2P file system a few years ago and actually gleaned a good number of tricks from Skype for dealing with NATs and constructing your network in general. Dropbox (and many others I'm sure) have all said the same thing: you need to design your system to function in the most hostile conditions you can think of; for Skype this seems especially devastating because so many components are beyond their control.
  - amackera 16y ago
    Do you know of any resources where I could learn about Skype's network structure? I'm very interested in distributed systems, especially ones with the scope of Skype's network.
    - terryjsmith 16y ago
      Sure, so there were a lot of them that we compiled, here are the ones and topics I remember. Protocol: we used a custom psuedo-TCP protocol built on UDP based on libjingle from Google (used in Google Talk): http://code.google.com/apis/talk/libjingle/file_share.html http://code.google.com/apis/talk/libjingle/file_share.html. Libjingle's filesharing itself was a decent resource for learning some P2P stuff as well. NATing and routing: we used Skype's UDP hole punching: http://www.h-online.com/security/features/How-Skype-Co-get-round-firewalls-747314.html http://www.h-online.com/security/features/How-Skype-Co-get-r.... Skype configures each install to be a supernode by default and then disables it based on certain criteria (low speeds, behind a NAT/firewall, etc.). Discovery of other network nodes is an entire subject unto itself and a lot of other network discovery protocols are well documented; our system used a central server to track up-time statistically and map clients to one another based on finding a good fit. Happy to answer any questions here or at my e-mail address in profile.

ddlatham 16y ago

Under normal circumstances, there are a large number of supernodes available. Unfortunately, today, many of them were taken offline by a problem affecting some versions of Skype. So, what was the problem affecting versions of Skype?

AgentConundrum 16y ago
That's what I've been wondering all day. I didn't have a problem with Skype being down; I had a problem with Skype having exploded this morning, screaming about uncaught exceptions as it did. Your quote was an "aha" moment for me.
- igravious 16y ago
  So did I, were you travelling at the time? I was in an airport and I figured that it is the weird you have an internet connection but it only takes you to a pay per usage landing page and no other page kind of problem that made Skype barf. I happened to a Java network client too at the same time. I can send you logs Skype!
  - AgentConundrum 16y ago
    Nope, not traveling. I was just sitting at home and it exploded.
code_duck 16y ago
I doubt that they'd want to tell everyone, since apparently triggering this problem effectively creates a DOS attack against the entire network.

zinssmeister 16y ago

does anyone know why skype picked German as the only translation of this message?

mike4u2 16y ago
Because Germans tend to complain the most if free stuff does not work ;-)
- RP_Joe 16y ago
  Very funny.
- preek 16y ago
  What free stuff? I have a paid subscription for international calls and my voicemail, my dear friend. Do you? If not, then stop mocking people you don't know. If you do, preferably also stop mocking people you don't know. Having said that, I have important business calls to do today and I'm certainly not going to throw money at my iPhone carrier. I hope Skype is coming back online for me, soon. But I have not yet started complaining anywhere, because Skype has offered me a great service and lots of savings in the last two years. But who am I to day, I'm just a grumpy German.
corin_ 16y ago
Obvious possibilities include German being the most used non-English language for Skype users, or that they're going to be adding other languages but their translators haven't yet written them. The only guess I have that isn't from an obvious reason is that Skype HQ is in Luxembourg, a small country right next to Germany. One of the three official languages there is German, and the other is Luxembourgish, which is closely related to German.
switch007 16y ago
My awful Spanish translation: A principios de hoy, nos dimos cuenta de que el número de usarios conectado a Skype empezó caer. Esto no es normal ni esperado así que nos pusimos a investigarlo. Skype no es como un red de telefonía convencional ni red de mensajería instantánea - en cambio, para que todo vaya bien depende de millones de conexiones entre los computadores y los teléfonos. Unos de esas computadores se llaman "supernodes" - funcionan como una agenda de direcciones por Skype. Si quieres hablar con alguien y Skype no lo puede localizar inmediatamente, tu computador o tu móvil intentará localizar un "supernode" para descubrir la manera de contactarle. Normalmente hay un montón de supernodes disponibles. Hoy, por desgracia, muchos no eran conectados debido a una problema que afectaba algunos versiones de Skype. ¿Qué estamos haciendo para mejorar la situación? Nuestros ingenieros están haciendo nuevos "mega-supernodes" lo cuanto antes, que con un poco de suerte va a hacer que todo se normalice.

johndbritton 16y ago

"Our engineers are creating new ‘mega-supernodes’ as fast as they can" What a brilliant solution.

tommi 16y ago
It's not a solution, it's a hot-fix.
- dennisgorelik 16y ago
  It's a hotfix that does not work [yet].
terryjsmith 16y ago
Yea, see the above comments. Bringing up individual nodes isn't an option as they would likely be overwhelmed and rate limiting traffic from all of the running instances in a P2P network would be near impossible. Bringing up an entirely new super node network, while insane, is probably their best option right now. Edit: I initially read your post as sarcastic, but I realize now it may not have been. It's definitely outside-the-box thinking and after considering their options, this certainly seems the most viable.
- dennisgorelik 16y ago
  HackerNews does not allow deep threads, (http://news.ycombinator.com/item?id=2032770 http://news.ycombinator.com/item?id=2032770) so I reply here. Why would supernode that has CPU as a bottleneck stop serving _any_ requests? Say, CPU is able to serve a thousand requests per second. If supernode receives a million requests per second, then supernode should serve first thousand requests and then ignore the rest, right?
  - terryjsmith 16y ago
    Correct, it will stop serving requests (ignore the rest), but the end result is no different (in fact I imagine this is exactly what's happening). That system is dead to the world. According to Wikipedia in Q3 2009 [1] Skype has over 500 million users (Q1 2009 says 42M active at any time); any super node that is processing a thousand requests and then dies (or only processes one request a second) isn't helping anyone. [1] http://en.wikipedia.org/wiki/Skype#Usage_and_traffic http://en.wikipedia.org/wiki/Skype#Usage_and_traffic

swolchok 16y ago

This isn't really a post-mortem; they're still in progress on a fix.

tommi 16y ago
The title also refers the whole Skype being dead.

anonymous246 16y ago

Does anybody know if these supernodes are paid for by Skype or they're simply users' computers being commandeered by Skype (after getting click-through permission). The explanation doesn't ring true to me because AFAIK supernodes are used only if a direct P2P path cannot be established between caller and callee. What we're observing is the inability to sign in and see contacts' status. I always thought that was centralized. Are supernodes involved in signin?

oiuytriuytr 16y ago
In theory they can be regular users computers - but unless you have the only PC connected to some small outpost on the internet it's unlikely that it would be you. Generally it's either a skype or ISP owned machine
terryjsmith 16y ago
This article is a little old now but I imagine the basic technique is still the same: http://www.h-online.com/security/features/How-Skype-Co-get-round-firewalls-747314.html http://www.h-online.com/security/features/How-Skype-Co-get-r... As I understand it, for corporate firewalls or routers/firewalls configured to not allow "hole punching", Skype will use the supernode as a relay for all communications.
piramida 16y ago
The user database is centralized (from a talk that lead Skype engineer given couple of years ago), so I too wonder how would a traffic routing nodes affect signing in. I am sure non of the client code has their user database credentials... So supernodes can only prevent communication, calls, video from happening. So there's something else they are hiding. My Mac skype dies on start since yesterday (trying to sign-in), supernodes would have nothing to do with that.

viraptor 16y ago

This part is pretty cool, but quite subtle: "Earlier today, we noticed that the number of people online on Skype was falling". I mean that they noticed the number going down, not "people started raising tickets saying it doesn't work".

car 16y ago

How is this a post-mortem? The blog entry does not explain in any way what really happened.

bystanderr 16y ago

Reading the discussion here reminded me of the outage of AT&T's long-distance network back in 1991. After a bit of searching, I found an interesting-looking document at http://faqs.org/rfcs/rfc3439.html http://faqs.org/rfcs/rfc3439.html entitled "RFC 3439 - Some Internet Architectural Guidelines and Philosophy," which seems relevant here somehow. Though this document outlines considerations pertaining to complexity in Internet backbone architecture, perhaps the overall philosophical questions it poses and guidelines it offers could be instructive as regards the recent Skype meltdown. As regards the aforementioned 1991 failure of the AT&T long-distance system, which resulted in a service outage of about six hours, the document says: "The PSTN's SS7 control network provides an interesting example of what can go wrong with a tightly coupled complex system. Outages such as the well publicized 1991 outage of AT&T's SS7 demonstrates the phenomenon: the outage was caused by software bugs in the switches' crash recovery code. In this case, one switch crashed due to a hardware glitch. When this switch came back up, it (plus a reasonably probable timing event) caused its neighbors to crash When the neighboring switches came back up, they caused their neighbors to crash, and so on [NEUMANN] (the root cause turned out to be a misplaced 'break' statement; this is an excellent example of cross- layer coupling). This phenomenon is similar to the phase-locking of weakly coupled oscillators, in which random variations in sequence times plays an important role in system stability [THOMPSON]."