11 ms·

Why is IRC distributed across multiple servers?

134 points by rain1 5y ago

jimjams 5y ago
In reality only guys in the same channel get sent the messages... if messages are spread between even a few channels the autual numbers are much more manageable for one server.
LinuxBender 5y ago
I can somewhat answer this. Apologies, this became a bit long winded and I have barely touched on several historical, technical and logistical reasons. Part of the answer is historical and part of this was technical. IRC has been around for a very long time. As such, the earlier versions of the servers and daemons could not accept tens of thousands of client connections ePoll vs Select. The connections between servers are multiplexed and not directly related to the number of people connected to the server. There was also a matter of latency. Servers in a region would keep the messages local to that region, as only people in the same channel get the messages and it was less common to have people in the same channel all over the world. This also changed with time. If there was a split, you lost other regions. This was not always the case, so of course I am over-generalizing since there were many different IRC networks designed by many different people. Being long running services, I had seen a great deal of hesitation to re-architect anything on the fly on at least some of the networks, even after ePoll and modern hardware made it possible to have tens of thousands of people on one server. Some of the smaller IRC networks indeed consolidated into fewer or a single server. Another facet is logistics and ownership. Many of the bigger networks are comprised of servers owned and managed by different people and organizations. The servers are linked as a matter of trust. That trust can be revoked. Most of the early IRC networks were run by people doing this in their free time with their own money and/or limited resources. In some other cases some organizations prefer to have their own servers so that their own people indeed to not suffer splits for their local communication. There are a myriad of other use-cases and reasons why some organizations had their own servers. Sometimes there was a need to give LocalOps special permissions that would not be permitted network-wide. Despite the technical capability to have less servers, some organizations are not going to give up their local nodes. One issue not mentioned is permission losses on splits. The issue with splits and permission changes has more to do with the way services are integrated into IRC, or more specifically, aren't. Services are treated like bots with higher privilege and most if not all of them were not written to be multi-master. Rather than dealing with moving services around or pushing for read-only daemons, they just lived with the possibility that there would be splits and they would eventually resolve themselves. I personally would have preferred to see a more common integration with OpenLDAP. Some of the IRC daemons can use LDAP, but it is more of an after-thought, or bolt on. This would have allowed splits to occur without losing channel permissions and clients could be configured to quickly attach to another server in another region and that is just DNS management. This could have been further improved by amending or replacing the IRC RFC's to allow SRV records. This may have been done by now for all I know. I shut down my last public server some time ago. There is a lot more to this than I could sum up on HN. Anyway, today you can fire up an IRCd of your choice on modern hardware and accept tens of thousands if not hundreds of thousands of people on a single server if you wish. It is technically possible. I would still design the network to have multiple servers, as you will eventually hit a bottleneck. If you really want to do this, you will have to de-tune the anti-ddos counter measures to allow the thundering herd to join your standby server or make code changes to permit the thundering herd briefly on fail-over.
- rain1 5y ago
  thanks, really appreciate this comment.
  - LinuxBender 5y ago
    My pleasure. I am sure others could add a great deal more. There is a very long history and there are many pieces of history I am leaving out. A big part I left out is the individual server rate limits vs. network link rate limits and network topology and that is both a technical and logistical issue.
    - spinax 5y ago
      One positive thing I'd add - as a user - under logistics is high availability. Life is messy, servers go down planned or unplanned for whatever reasons - IRC networks are in a sense truly 'federated' in that the client will get a new server on reconnect attempts much like webservers behind a load balancer. You never have to worry about your 'home instance' being unavailable, as they're all your home instance. (I speak about the public networks like Libera or OFTC)
- wayoutthere 5y ago
  As someone who ran IRC servers in the 90s the technical limitation was the number of file descriptors. I think Linux at the time was limited to 1024 and the biggest server on our network was a DEC Alpha with 4096. The entire network (DALnet at the time) was in the 20-30k user range so we absolutely needed multiple servers.
  - blibble 5y ago
    there was also no efficient io multiplexing an ircd with a few thousand clients was cpu bound on poll()/select() /dev/poll and kqueue/epoll were game changing
    - jlokier 5y ago
mvanbaak 5y ago
> via round robin DNS (meaning that when people resolve the DNS it gives them a random server from the set of 20 to connect to) Most of the times, it's not simple round-robin, but also geo-based. This means clients will get ip addresses of the servers closest to them.
- magila 5y ago
  My experience with Freenode/Libra Chat is that they either don't implement geo DNS or don't do a very good job of it. I'm on the US west coast and lookups to irc.libera.chat often return servers in Europe. Edit: Double checking Libra Chat's website I see that they have added regional hostnames so I guess that's their solution.
  - pvtmert 5y ago
    if they're using aws route53, your isp needs to support edns. otherwise, your netblock might have been falsely advertised in the dns provider's geoip database. (eg. maxmind)
    - pushrax 5y ago
      They're using Cloudflare. When I resolve them from the east coast, I got a San Francisco server once and a server in Budapest once. They have a server in Toronto, Ashburn, Montreal, and other places that are closer. I know geodns works here since I use it for some of my own deployments.
- melony 5y ago
  Does IRC predates distributed state machines? Why can't the servers sync up the chat via Paxos or Raft?
  - sterlind 5y ago
    hard to do Paxos over large geographical distances efficiently, but... it's IRC, so.. I just assume it was from an earlier internet where distributed systems weren't as well understood. I don't think it necessarily predates Paxos but it definitely predates Paxos being a household name.
300bps 5y ago
One thing I haven’t seen covered is multiple servers = redundancy. If a server goes down, having a net split is a lot better than having the entire network down.
unixhero 5y ago
Engineering IRC networks is si much fun.
k__ 5y ago
So its users can get fun netsplits. I remember we would all try to get on the same server in our channel, but some less technical people would use a web client that assigned different ones every time.
hcykb 5y ago
Because when IRC was popular servers and routes went down often and a single server couldn't handle all the users a network would have. Neither of those are a concern anymore.
rawoke083600 5y ago
Would be fun to visit the old problems(like this one) with modern toolset. Say golang with channels(not the /join type of channels) :p
prdonahue 5y ago
So you can nick collide people, obviously.
nathias 5y ago
The side effect was also great for communities on .net servers that didn't have services like user accounts and channels. Channel ops were battle-won and people who had them were much better at not sucking completely.
- Sunspark 5y ago
  ChanOps have always been a problem. Anyone who becomes one, is a dictator for life. There is no recourse, the only option is to either be on their good side, or go to a different channel or network. I like IRC as an open technology. I don't like the lack of accountability from the gatekeepers. It is the same problem on online forums like reddit. If the mods do not look upon you with favour, you are banned, even if the rules have not been broken.
  - nathias 5y ago
    Yes, chanops make channels into properties of individuals, but without them they are property of the community that uses them.
Ologn 5y ago
> One of the problems of having multiple servers is that netsplits can occur. In the early/mid 1990s, the IRC servers in Australia would split from the IRC servers in the US all of the time (sometimes Europe would break from the US as well). The Internet connection between the US and Australia was slower and flakier back then. It made lots of sense for Australians to be on Australian IRC servers and Americans to be on US IRC servers, and to all be talking together when the link was working (the majority of the time) and to not be when the link broke (fairly regularly). The CAP theorem says something has to go in those cases, and the thing that went was consistency between US and Australian (or European) messages sent to a channel - the messages from the other side of the split would be dropped during the split. I don't remember many technical netsplits on Freenode or Libera in recent years, so it is less of a thing now. IRC servers were always federated, so there was the original split of Anet and EFnet, and the Undernet split, then the EFnet/IRCnet split which revolved around those US/Europe/Australia issues. More recently there was the Freenode/Libera split. IRC's model always worked for me.
jchw 5y ago
Well, from my historical reading of it, initially, IRC was a federated network of servers that were essentially one network, the way email is one network: there was no shared administration or anything. Anyone could run a server and jump into the network. Due to abuse, servers began restricting who they peered with, and it fractured into multiple networks. So really, I suspect it was designed to be distributed and federated, and it just became what it is by accident.
- Ekaros 5y ago
  Many other services also used to be like this. Think of Usenet aka. news. It is effective model when you think of Internet as network of networks. When there was real difference between connecting to your local area network, metropolitan area network or even wide area network. Actually we have come quite far from those days and full speed point-to-point links between most points is somewhat realistic.
- unilynx 5y ago
  It was never open to attach your server to a network, unlike email. A server connection was way too powerful for that. You needed an existing server admin to allow your server to connect.
  - ghancock 5y ago
    I wasn’t there but I have seen multiple histories say that there were servers that accepted connections from anyone (most famously eris.berkeley.edu but not only that one). For example, https://about.psyc.eu/IRC https://about.psyc.eu/IRC
- albertgoeswoof 5y ago
  What would be the abuse issues from open peering? How were we able to solve them for email, but not IRC?
  - nickelpro 5y ago
    We didn't, email spam exists to this day. The solution has been to ban entire swaths of domains and even IP ranges by chucking all mail from them into spam folders
jbverschoor 5y ago
Because many of the early protocols, including IP, we’re designed with network failures in mind.
throwthere 5y ago
I don’t know if the numbers are realistic here. First and most importantly, messages are only sent to clients in the same chatroom, not sever wide. Second, 10% of users are only very rarely going to send messages at once. By rare you can probably substitute never. This, this is simple very small text messages where seconds of lag don’t really matter— why would it be hard to manage tens of thousands of concurrent connections?whatsapp crushed millions of connections on single server back in 2012— https://web.archive.org/web/20140501234954/https://blog.whatsapp.com/196/1-million-is-so-2011 https://web.archive.org/web/20140501234954/https://blog.what...
throwaway20371 5y ago
Why are Linux distributions hosted on multiple mirror servers that they don't own? 1) money 2) availability 3) trust 4) security 1) If you don't have a lot of money, you take the servers you can get. Donated mirrors means you don't have to pay the bandwidth or hosting bills. 2) If you have multiple servers, it's less likely that one server going down will tank your project. When GitHub, AWS, or even Level3 has an outage, Linux distros keep on chugging like nothing happened. Traditional server maintenance is also easier when everyone can just switch to a different server. 3) Maintainers can use their PGP keys to create signed packages and downloads. Their public keys are distributed on mirrors, as well as embedded in the downloads they've signed. Once downloaded by users, the distribution can verify its own integrity. But how does the user know they started with the real maintainers' public key? The public key is distributed on a hundred geographically-distributed servers all owned by different people; the user can check them all. So other than compromising a maintainer's key, it's logistically impossible to compromise end-user security. (this one is more Linux-specific than IRC-specific) 4) If you only have one server & it gets compromised, it can be hard to tell. By comparing its operating state to the other servers, you can sometimes more quickly identify the compromise. And if you do find a compromise, you can remove the compromised server quickly, close the hole on the other servers, and start regenerating keys. It's an eventuality every large project should be prepared for, and IRC servers do get compromised. Linux mirrors don't matter in this regard, but the build servers etc do matter. IRC comes from the same time and place, and has some (but not all) of the same considerations.
H8crilA 5y ago
Just remember that "netsplits" exist in every distributed system, be it a chat app or a database. It's just the CAP theorem. IRC has chosen to sacrifice C (consistency). The only thing that changed in the modern times is that the P (partitions) are extremely rare in modern high octane cloud infrastructures. Also, modern solutions often decide to sacrifice A (availability), by returning an error saying "we're aware of the problem and we're working on a solution". This is what happened quite recently when Google authentication went out and half of the internet went dark, while under the hood they had a simple out-of-quota situation on one of the replicas of their core authentication systems. The system was programmed to sacrifice A (availability) and reject all authentication requests.
- throwaway20371 5y ago
  > "netsplits" exist in every distributed system, be it a chat app or a database, it's just the CAP theorem Well let's not get carried away. Network partitions happen everywhere, but everything is not about the CAP theorem. CAP theorem is a very specific model that a lot of apps (even ACID databases) don't conform with. Comparing IRC to CAP theorem is like comparing it to ACID and saying, "IRC decided to sacrifice transaction integrity". IRC didn't explicitly sacrifice the C in CAP, they designed a simple server protocol. They could have added a bunch of weirdness to hide splits from users, but it would have been unnecessarily complicated and not contributed significantly to the user experience.
  - H8crilA 5y ago
    I'm sorry but I don't think you realise how simple and fundamental the CAP theorem is. It's almost a tautology. And yes it applies fully. The most basic case is if there's absolutely no method of exchanging information from point A to point B. Then agents at A and B will not be able to communicate. That's it. Any system built to facilitate information exhange will either have to deliver incomplete information (C) or will have to refuse to operate (A). Now then, as I said, nowadays it's extremely unlikely that there's truly no connection between any two major Internet hubs (though it can happen, hello BGP). It still happens in specific systems that do not work on any method of information transfer but rather on specific methods of information transfer. The IRC example requires specific servers to be up, not just a functioning IP routing between the end clients. If some server is not up then (at least temporarily) from IRC's point of view there's no way to deliver information from A to B. The Google auth outage example requires (among most likely many other things) disk space availability on specific servers for information exchange to happen.
sdfs6645d 5y ago
https://www.reddit.com/live/17n32347h2coa https://www.reddit.com/live/17n32347h2coa https://www.reddit.com/live/17n32eybvox8j https://www.reddit.com/live/17n32eybvox8j https://www.reddit.com/live/17n33zpj7ujrq https://www.reddit.com/live/17n33zpj7ujrq https://www.reddit.com/live/17n34xytz9su9 https://www.reddit.com/live/17n34xytz9su9 https://www.reddit.com/live/17n35ablgxldg https://www.reddit.com/live/17n35ablgxldg https://www.reddit.com/live/17n35grmkeeqo https://www.reddit.com/live/17n35grmkeeqo https://www.reddit.com/live/17n35walz24op https://www.reddit.com/live/17n35walz24op https://www.reddit.com/live/17n364wbm5m7n https://www.reddit.com/live/17n364wbm5m7n https://www.reddit.com/live/17n36fxxw5f06 https://www.reddit.com/live/17n36fxxw5f06 https://www.reddit.com/live/17n3bisi2bly2 https://www.reddit.com/live/17n3bisi2bly2 https://www.reddit.com/live/17n3bx98vndyl https://www.reddit.com/live/17n3bx98vndyl https://www.reddit.com/live/17n3c3kq8vgi1 https://www.reddit.com/live/17n3c3kq8vgi1 https://www.reddit.com/live/17n3cbuq2sawu https://www.reddit.com/live/17n3cbuq2sawu https://www.reddit.com/live/17n3clc0kf3qk https://www.reddit.com/live/17n3clc0kf3qk https://www.reddit.com/live/17n3crh9t9hba https://www.reddit.com/live/17n3crh9t9hba https://www.reddit.com/live/17n3d4tgfroo8 https://www.reddit.com/live/17n3d4tgfroo8 https://www.reddit.com/live/17n3dcxpjb6mz https://www.reddit.com/live/17n3dcxpjb6mz https://www.reddit.com/live/17n3dj906t21a https://www.reddit.com/live/17n3dj906t21a https://www.reddit.com/live/17n3drj836arb https://www.reddit.com/live/17n3drj836arb https://www.reddit.com/r/nflChiefsvBrownstvs/ https://www.reddit.com/r/nflChiefsvBrownstvs/ https://www.reddit.com/r/nflChiefsvBrowns/ https://www.reddit.com/r/nflChiefsvBrowns/ https://www.reddit.com/live/17n3g9kcuru7t https://www.reddit.com/live/17n3g9kcuru7t https://www.reddit.com/live/17n3gg2056pzw https://www.reddit.com/live/17n3gg2056pzw https://www.reddit.com/live/17n3grv476zr0 https://www.reddit.com/live/17n3grv476zr0 https://www.reddit.com/live/17n3h0p5701cy https://www.reddit.com/live/17n3h0p5701cy https://www.reddit.com/live/17n3iascrq39o https://www.reddit.com/live/17n3iascrq39o https://www.reddit.com/live/17n3i4nkj2vwg https://www.reddit.com/live/17n3i4nkj2vwg https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk5.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk4.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk3.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk2.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk1.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk0.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv3.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv2.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv1.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv0.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv.html https://links.uky.edu/sites/default/files/webform/marketing-...

134 points by rain1 5y ago

In reality only guys in the same channel get sent the messages... if messages are spread between even a few channels the autual numbers are much more manageable for one server.

LinuxBender 5y ago

I can somewhat answer this. Apologies, this became a bit long winded and I have barely touched on several historical, technical and logistical reasons. Part of the answer is historical and part of this was technical. IRC has been around for a very long time. As such, the earlier versions of the servers and daemons could not accept tens of thousands of client connections ePoll vs Select. The connections between servers are multiplexed and not directly related to the number of people connected to the server. There was also a matter of latency. Servers in a region would keep the messages local to that region, as only people in the same channel get the messages and it was less common to have people in the same channel all over the world. This also changed with time. If there was a split, you lost other regions. This was not always the case, so of course I am over-generalizing since there were many different IRC networks designed by many different people. Being long running services, I had seen a great deal of hesitation to re-architect anything on the fly on at least some of the networks, even after ePoll and modern hardware made it possible to have tens of thousands of people on one server. Some of the smaller IRC networks indeed consolidated into fewer or a single server. Another facet is logistics and ownership. Many of the bigger networks are comprised of servers owned and managed by different people and organizations. The servers are linked as a matter of trust. That trust can be revoked. Most of the early IRC networks were run by people doing this in their free time with their own money and/or limited resources. In some other cases some organizations prefer to have their own servers so that their own people indeed to not suffer splits for their local communication. There are a myriad of other use-cases and reasons why some organizations had their own servers. Sometimes there was a need to give LocalOps special permissions that would not be permitted network-wide. Despite the technical capability to have less servers, some organizations are not going to give up their local nodes. One issue not mentioned is permission losses on splits. The issue with splits and permission changes has more to do with the way services are integrated into IRC, or more specifically, aren't. Services are treated like bots with higher privilege and most if not all of them were not written to be multi-master. Rather than dealing with moving services around or pushing for read-only daemons, they just lived with the possibility that there would be splits and they would eventually resolve themselves. I personally would have preferred to see a more common integration with OpenLDAP. Some of the IRC daemons can use LDAP, but it is more of an after-thought, or bolt on. This would have allowed splits to occur without losing channel permissions and clients could be configured to quickly attach to another server in another region and that is just DNS management. This could have been further improved by amending or replacing the IRC RFC's to allow SRV records. This may have been done by now for all I know. I shut down my last public server some time ago. There is a lot more to this than I could sum up on HN. Anyway, today you can fire up an IRCd of your choice on modern hardware and accept tens of thousands if not hundreds of thousands of people on a single server if you wish. It is technically possible. I would still design the network to have multiple servers, as you will eventually hit a bottleneck. If you really want to do this, you will have to de-tune the anti-ddos counter measures to allow the thundering herd to join your standby server or make code changes to permit the thundering herd briefly on fail-over.

rain1 5y ago
thanks, really appreciate this comment.
- LinuxBender 5y ago
  My pleasure. I am sure others could add a great deal more. There is a very long history and there are many pieces of history I am leaving out. A big part I left out is the individual server rate limits vs. network link rate limits and network topology and that is both a technical and logistical issue.
  - spinax 5y ago
    One positive thing I'd add - as a user - under logistics is high availability. Life is messy, servers go down planned or unplanned for whatever reasons - IRC networks are in a sense truly 'federated' in that the client will get a new server on reconnect attempts much like webservers behind a load balancer. You never have to worry about your 'home instance' being unavailable, as they're all your home instance. (I speak about the public networks like Libera or OFTC)
wayoutthere 5y ago
As someone who ran IRC servers in the 90s the technical limitation was the number of file descriptors. I think Linux at the time was limited to 1024 and the biggest server on our network was a DEC Alpha with 4096. The entire network (DALnet at the time) was in the 20-30k user range so we absolutely needed multiple servers.
- blibble 5y ago
  there was also no efficient io multiplexing an ircd with a few thousand clients was cpu bound on poll()/select() /dev/poll and kqueue/epoll were game changing
  - jlokier 5y ago

mvanbaak 5y ago

> via round robin DNS (meaning that when people resolve the DNS it gives them a random server from the set of 20 to connect to) Most of the times, it's not simple round-robin, but also geo-based. This means clients will get ip addresses of the servers closest to them.

magila 5y ago
My experience with Freenode/Libra Chat is that they either don't implement geo DNS or don't do a very good job of it. I'm on the US west coast and lookups to irc.libera.chat often return servers in Europe. Edit: Double checking Libra Chat's website I see that they have added regional hostnames so I guess that's their solution.
- pvtmert 5y ago
  if they're using aws route53, your isp needs to support edns. otherwise, your netblock might have been falsely advertised in the dns provider's geoip database. (eg. maxmind)
  - pushrax 5y ago
    They're using Cloudflare. When I resolve them from the east coast, I got a San Francisco server once and a server in Budapest once. They have a server in Toronto, Ashburn, Montreal, and other places that are closer. I know geodns works here since I use it for some of my own deployments.
melony 5y ago
Does IRC predates distributed state machines? Why can't the servers sync up the chat via Paxos or Raft?
- sterlind 5y ago
  hard to do Paxos over large geographical distances efficiently, but... it's IRC, so.. I just assume it was from an earlier internet where distributed systems weren't as well understood. I don't think it necessarily predates Paxos but it definitely predates Paxos being a household name.

300bps 5y ago

One thing I haven’t seen covered is multiple servers = redundancy. If a server goes down, having a net split is a lot better than having the entire network down.

unixhero 5y ago

Engineering IRC networks is si much fun.

k__ 5y ago

So its users can get fun netsplits. I remember we would all try to get on the same server in our channel, but some less technical people would use a web client that assigned different ones every time.

hcykb 5y ago

Because when IRC was popular servers and routes went down often and a single server couldn't handle all the users a network would have. Neither of those are a concern anymore.

rawoke083600 5y ago

Would be fun to visit the old problems(like this one) with modern toolset. Say golang with channels(not the /join type of channels) :p

prdonahue 5y ago

So you can nick collide people, obviously.

nathias 5y ago

The side effect was also great for communities on .net servers that didn't have services like user accounts and channels. Channel ops were battle-won and people who had them were much better at not sucking completely.

Sunspark 5y ago
ChanOps have always been a problem. Anyone who becomes one, is a dictator for life. There is no recourse, the only option is to either be on their good side, or go to a different channel or network. I like IRC as an open technology. I don't like the lack of accountability from the gatekeepers. It is the same problem on online forums like reddit. If the mods do not look upon you with favour, you are banned, even if the rules have not been broken.
- nathias 5y ago
  Yes, chanops make channels into properties of individuals, but without them they are property of the community that uses them.

Ologn 5y ago

> One of the problems of having multiple servers is that netsplits can occur. In the early/mid 1990s, the IRC servers in Australia would split from the IRC servers in the US all of the time (sometimes Europe would break from the US as well). The Internet connection between the US and Australia was slower and flakier back then. It made lots of sense for Australians to be on Australian IRC servers and Americans to be on US IRC servers, and to all be talking together when the link was working (the majority of the time) and to not be when the link broke (fairly regularly). The CAP theorem says something has to go in those cases, and the thing that went was consistency between US and Australian (or European) messages sent to a channel - the messages from the other side of the split would be dropped during the split. I don't remember many technical netsplits on Freenode or Libera in recent years, so it is less of a thing now. IRC servers were always federated, so there was the original split of Anet and EFnet, and the Undernet split, then the EFnet/IRCnet split which revolved around those US/Europe/Australia issues. More recently there was the Freenode/Libera split. IRC's model always worked for me.

jchw 5y ago

Well, from my historical reading of it, initially, IRC was a federated network of servers that were essentially one network, the way email is one network: there was no shared administration or anything. Anyone could run a server and jump into the network. Due to abuse, servers began restricting who they peered with, and it fractured into multiple networks. So really, I suspect it was designed to be distributed and federated, and it just became what it is by accident.

Ekaros 5y ago
Many other services also used to be like this. Think of Usenet aka. news. It is effective model when you think of Internet as network of networks. When there was real difference between connecting to your local area network, metropolitan area network or even wide area network. Actually we have come quite far from those days and full speed point-to-point links between most points is somewhat realistic.
unilynx 5y ago
It was never open to attach your server to a network, unlike email. A server connection was way too powerful for that. You needed an existing server admin to allow your server to connect.
- ghancock 5y ago
  I wasn’t there but I have seen multiple histories say that there were servers that accepted connections from anyone (most famously eris.berkeley.edu but not only that one). For example, https://about.psyc.eu/IRC https://about.psyc.eu/IRC
albertgoeswoof 5y ago
What would be the abuse issues from open peering? How were we able to solve them for email, but not IRC?
- nickelpro 5y ago
  We didn't, email spam exists to this day. The solution has been to ban entire swaths of domains and even IP ranges by chucking all mail from them into spam folders

jbverschoor 5y ago

Because many of the early protocols, including IP, we’re designed with network failures in mind.

throwthere 5y ago

I don’t know if the numbers are realistic here. First and most importantly, messages are only sent to clients in the same chatroom, not sever wide. Second, 10% of users are only very rarely going to send messages at once. By rare you can probably substitute never. This, this is simple very small text messages where seconds of lag don’t really matter— why would it be hard to manage tens of thousands of concurrent connections?whatsapp crushed millions of connections on single server back in 2012— https://web.archive.org/web/20140501234954/https://blog.whatsapp.com/196/1-million-is-so-2011 https://web.archive.org/web/20140501234954/https://blog.what...

throwaway20371 5y ago

Why are Linux distributions hosted on multiple mirror servers that they don't own? 1) money 2) availability 3) trust 4) security 1) If you don't have a lot of money, you take the servers you can get. Donated mirrors means you don't have to pay the bandwidth or hosting bills. 2) If you have multiple servers, it's less likely that one server going down will tank your project. When GitHub, AWS, or even Level3 has an outage, Linux distros keep on chugging like nothing happened. Traditional server maintenance is also easier when everyone can just switch to a different server. 3) Maintainers can use their PGP keys to create signed packages and downloads. Their public keys are distributed on mirrors, as well as embedded in the downloads they've signed. Once downloaded by users, the distribution can verify its own integrity. But how does the user know they started with the real maintainers' public key? The public key is distributed on a hundred geographically-distributed servers all owned by different people; the user can check them all. So other than compromising a maintainer's key, it's logistically impossible to compromise end-user security. (this one is more Linux-specific than IRC-specific) 4) If you only have one server & it gets compromised, it can be hard to tell. By comparing its operating state to the other servers, you can sometimes more quickly identify the compromise. And if you do find a compromise, you can remove the compromised server quickly, close the hole on the other servers, and start regenerating keys. It's an eventuality every large project should be prepared for, and IRC servers do get compromised. Linux mirrors don't matter in this regard, but the build servers etc do matter. IRC comes from the same time and place, and has some (but not all) of the same considerations.

H8crilA 5y ago

Just remember that "netsplits" exist in every distributed system, be it a chat app or a database. It's just the CAP theorem. IRC has chosen to sacrifice C (consistency). The only thing that changed in the modern times is that the P (partitions) are extremely rare in modern high octane cloud infrastructures. Also, modern solutions often decide to sacrifice A (availability), by returning an error saying "we're aware of the problem and we're working on a solution". This is what happened quite recently when Google authentication went out and half of the internet went dark, while under the hood they had a simple out-of-quota situation on one of the replicas of their core authentication systems. The system was programmed to sacrifice A (availability) and reject all authentication requests.

throwaway20371 5y ago
> "netsplits" exist in every distributed system, be it a chat app or a database, it's just the CAP theorem Well let's not get carried away. Network partitions happen everywhere, but everything is not about the CAP theorem. CAP theorem is a very specific model that a lot of apps (even ACID databases) don't conform with. Comparing IRC to CAP theorem is like comparing it to ACID and saying, "IRC decided to sacrifice transaction integrity". IRC didn't explicitly sacrifice the C in CAP, they designed a simple server protocol. They could have added a bunch of weirdness to hide splits from users, but it would have been unnecessarily complicated and not contributed significantly to the user experience.
- H8crilA 5y ago
  I'm sorry but I don't think you realise how simple and fundamental the CAP theorem is. It's almost a tautology. And yes it applies fully. The most basic case is if there's absolutely no method of exchanging information from point A to point B. Then agents at A and B will not be able to communicate. That's it. Any system built to facilitate information exhange will either have to deliver incomplete information (C) or will have to refuse to operate (A). Now then, as I said, nowadays it's extremely unlikely that there's truly no connection between any two major Internet hubs (though it can happen, hello BGP). It still happens in specific systems that do not work on any method of information transfer but rather on specific methods of information transfer. The IRC example requires specific servers to be up, not just a functioning IP routing between the end clients. If some server is not up then (at least temporarily) from IRC's point of view there's no way to deliver information from A to B. The Google auth outage example requires (among most likely many other things) disk space availability on specific servers for information exchange to happen.

sdfs6645d 5y ago

https://www.reddit.com/live/17n32347h2coa https://www.reddit.com/live/17n32347h2coa https://www.reddit.com/live/17n32eybvox8j https://www.reddit.com/live/17n32eybvox8j https://www.reddit.com/live/17n33zpj7ujrq https://www.reddit.com/live/17n33zpj7ujrq https://www.reddit.com/live/17n34xytz9su9 https://www.reddit.com/live/17n34xytz9su9 https://www.reddit.com/live/17n35ablgxldg https://www.reddit.com/live/17n35ablgxldg https://www.reddit.com/live/17n35grmkeeqo https://www.reddit.com/live/17n35grmkeeqo https://www.reddit.com/live/17n35walz24op https://www.reddit.com/live/17n35walz24op https://www.reddit.com/live/17n364wbm5m7n https://www.reddit.com/live/17n364wbm5m7n https://www.reddit.com/live/17n36fxxw5f06 https://www.reddit.com/live/17n36fxxw5f06 https://www.reddit.com/live/17n3bisi2bly2 https://www.reddit.com/live/17n3bisi2bly2 https://www.reddit.com/live/17n3bx98vndyl https://www.reddit.com/live/17n3bx98vndyl https://www.reddit.com/live/17n3c3kq8vgi1 https://www.reddit.com/live/17n3c3kq8vgi1 https://www.reddit.com/live/17n3cbuq2sawu https://www.reddit.com/live/17n3cbuq2sawu https://www.reddit.com/live/17n3clc0kf3qk https://www.reddit.com/live/17n3clc0kf3qk https://www.reddit.com/live/17n3crh9t9hba https://www.reddit.com/live/17n3crh9t9hba https://www.reddit.com/live/17n3d4tgfroo8 https://www.reddit.com/live/17n3d4tgfroo8 https://www.reddit.com/live/17n3dcxpjb6mz https://www.reddit.com/live/17n3dcxpjb6mz https://www.reddit.com/live/17n3dj906t21a https://www.reddit.com/live/17n3dj906t21a https://www.reddit.com/live/17n3drj836arb https://www.reddit.com/live/17n3drj836arb https://www.reddit.com/r/nflChiefsvBrownstvs/ https://www.reddit.com/r/nflChiefsvBrownstvs/ https://www.reddit.com/r/nflChiefsvBrowns/ https://www.reddit.com/r/nflChiefsvBrowns/ https://www.reddit.com/live/17n3g9kcuru7t https://www.reddit.com/live/17n3g9kcuru7t https://www.reddit.com/live/17n3gg2056pzw https://www.reddit.com/live/17n3gg2056pzw https://www.reddit.com/live/17n3grv476zr0 https://www.reddit.com/live/17n3grv476zr0 https://www.reddit.com/live/17n3h0p5701cy https://www.reddit.com/live/17n3h0p5701cy https://www.reddit.com/live/17n3iascrq39o https://www.reddit.com/live/17n3iascrq39o https://www.reddit.com/live/17n3i4nkj2vwg https://www.reddit.com/live/17n3i4nkj2vwg https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk5.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk4.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk3.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk2.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk1.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk0.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/videos-browns-vs-chiefs-liv-tpk.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv3.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv2.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv1.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv0.html https://links.uky.edu/sites/default/files/webform/marketing-... https://links.uky.edu/sites/default/files/webform/marketing-requests/video-djk-vs-mdv.html https://links.uky.edu/sites/default/files/webform/marketing-...