4 ms·

I never really understood how usenet works. There are usenet servers, and you post to usenet via these, right? Given there are O(100s) of servers (at least), I

by sargun 3y ago

I never really understood how usenet works. There are usenet servers, and you post to usenet via these, right? Given there are O(100s) of servers (at least), I assume that the protocol between these servers is not full mesh, and there's some kind of small set of servers that participate in a mesh, and then followers that synchronize from them. How does this synchronization protocol work? How does one join the mesh? I've read the wikipedia article, and it kinda talks to the topology and such, but not in great detail.

ogurechny 3y ago
There is no “synchronization” because there is no defined state for servers to reach. Protocol simply allows to exchange messages in formatted text files between two nodes. What happens to them next is not defined (and may vary greatly). A server might decide to exchange messages in groups A, B, C… with peers X, Y, Z. Choice of peers depends on their capacity, working hours, geographic location, network position, personal connections, access to dedicated lines… Choice of groups depends on popularity, space requirements, personal preferences, user requests… Some of them allow messages to spread step-by-step to all or almost all of the network, those are global groups. Some of them are only shared by a couple of servers or exist on a single server, those are local. Nothing prevents you from making a “boobble.shmoobble.goobble” group and stating that it's the central most important point of all Usenet. You only need to make all the rest believe it. Then there are additional complexities and tools to deal with batch transfers, figuring out which messages have not yet been seen by a peer since the previous interchange (you don't want X to send lots of updates to Z, then Y to take the same long time sending mostly the same updates to Z), choice of best next hop (routing, manual load balancing), and so on. Of course, when synchronization is needed — say, for group moderator to be able to delete spam message from many servers holding that group instead of each admin doing that on each server independently, and to prevent reintroduction and further spread of the message, — it results in hairy ball of hacks on top of original architecture, and ad-hoc external trust channels, like with signatures. Understandably, Usenet appeared in a hierarchical and controlled professional environment (users were bound by formal and informal rules), then it was re-purposed for free for all operation.
tptacek 3y ago
It's a manually configured topology. An ordinary server would get a feed from somewhere else (probably their upstream ISP). A transit server would take dozens of feeds. The operation of each of those links is the same, though: `IHAVE <articleid>` and `SENDME <articleid>`.
csixty4 3y ago
NNTP descends from UUCP, the Unix to Unix Copy Protocol. Used to be that some businesses and universities configured their Unix machines to call other machines on their modems overnight (the rates were cheaper) and synchronize files. Think of it like a very slow, ancient version of rsync. You would post on your school's Usenet server, which would then sync with one or more servers overnight. And eventually (hopefully) your message would spread across the whole network. This could take days. Some of the big timesharing services (think Telenet and Tymnet, eventually even AOL) out there ran their own servers, which a lot of smaller services dialed into. How do you join the mesh? You met someone at Usenix or a similar conference and said "I'm sitting on a whopping 800mb of storage and a T1 line at my university. I'm tired of waiting a week to get the new comp.lang.c hotness. Think I could hook up to your machine for Usenet?" and they'd be like "Cool! Yeah, here's the dialup number. Try to call after 11pm so my boss doesn't know about it." NNTP took this concept of syncing files and made it Internet native and specific to Usenet. But the architecture remained largely the same.
- denton-scratch 3y ago
  > Used to be that some businesses and universities configured their Unix machines to call other machines on their modems overnight I used to work in London, for Olivetti, which was partnered with AT&T. Olivetti's office was in Finsbury, just north of The City. AT&T were south of the river, near Vauxhall. I believe our Usenet feed (and our internet email) was couriered-over from Vauxhall on mag tape. In those days, most ordinary businesses didn't have fixed-line internet access, and maintaining a Usenet feed over dial-up was expensive, even text-only.
floren 3y ago
These days, you join the Usenet by posting in news.peering and saying "I want to peer with somebody", or just emailing the Eternal September folks: http://www.eternal-september.org/index.php?showpage=peering http://www.eternal-september.org/index.php?showpage=peering You configure both configure your servers to exchange messages and then... they exchange messages :) It's good practice to have a couple peers, in case one goes offline or doesn't carry the groups you're interested in -- or filters things you might want. I run NoCeM on-spool and have stricter-than-normal CleanFeed rules, which translated from Usenet admin speak means my server drops a lot of spam and garbage posts, and if you want those you'll have to get them from another peer.

by sargun 3y ago

ogurechny 3y ago

There is no “synchronization” because there is no defined state for servers to reach. Protocol simply allows to exchange messages in formatted text files between two nodes. What happens to them next is not defined (and may vary greatly). A server might decide to exchange messages in groups A, B, C… with peers X, Y, Z. Choice of peers depends on their capacity, working hours, geographic location, network position, personal connections, access to dedicated lines… Choice of groups depends on popularity, space requirements, personal preferences, user requests… Some of them allow messages to spread step-by-step to all or almost all of the network, those are global groups. Some of them are only shared by a couple of servers or exist on a single server, those are local. Nothing prevents you from making a “boobble.shmoobble.goobble” group and stating that it's the central most important point of all Usenet. You only need to make all the rest believe it. Then there are additional complexities and tools to deal with batch transfers, figuring out which messages have not yet been seen by a peer since the previous interchange (you don't want X to send lots of updates to Z, then Y to take the same long time sending mostly the same updates to Z), choice of best next hop (routing, manual load balancing), and so on. Of course, when synchronization is needed — say, for group moderator to be able to delete spam message from many servers holding that group instead of each admin doing that on each server independently, and to prevent reintroduction and further spread of the message, — it results in hairy ball of hacks on top of original architecture, and ad-hoc external trust channels, like with signatures. Understandably, Usenet appeared in a hierarchical and controlled professional environment (users were bound by formal and informal rules), then it was re-purposed for free for all operation.

tptacek 3y ago

It's a manually configured topology. An ordinary server would get a feed from somewhere else (probably their upstream ISP). A transit server would take dozens of feeds. The operation of each of those links is the same, though: `IHAVE <articleid>` and `SENDME <articleid>`.

csixty4 3y ago

NNTP descends from UUCP, the Unix to Unix Copy Protocol. Used to be that some businesses and universities configured their Unix machines to call other machines on their modems overnight (the rates were cheaper) and synchronize files. Think of it like a very slow, ancient version of rsync. You would post on your school's Usenet server, which would then sync with one or more servers overnight. And eventually (hopefully) your message would spread across the whole network. This could take days. Some of the big timesharing services (think Telenet and Tymnet, eventually even AOL) out there ran their own servers, which a lot of smaller services dialed into. How do you join the mesh? You met someone at Usenix or a similar conference and said "I'm sitting on a whopping 800mb of storage and a T1 line at my university. I'm tired of waiting a week to get the new comp.lang.c hotness. Think I could hook up to your machine for Usenet?" and they'd be like "Cool! Yeah, here's the dialup number. Try to call after 11pm so my boss doesn't know about it." NNTP took this concept of syncing files and made it Internet native and specific to Usenet. But the architecture remained largely the same.

denton-scratch 3y ago
> Used to be that some businesses and universities configured their Unix machines to call other machines on their modems overnight I used to work in London, for Olivetti, which was partnered with AT&T. Olivetti's office was in Finsbury, just north of The City. AT&T were south of the river, near Vauxhall. I believe our Usenet feed (and our internet email) was couriered-over from Vauxhall on mag tape. In those days, most ordinary businesses didn't have fixed-line internet access, and maintaining a Usenet feed over dial-up was expensive, even text-only.

floren 3y ago

These days, you join the Usenet by posting in news.peering and saying "I want to peer with somebody", or just emailing the Eternal September folks: http://www.eternal-september.org/index.php?showpage=peering http://www.eternal-september.org/index.php?showpage=peering You configure both configure your servers to exchange messages and then... they exchange messages :) It's good practice to have a couple peers, in case one goes offline or doesn't carry the groups you're interested in -- or filters things you might want. I run NoCeM on-spool and have stricter-than-normal CleanFeed rules, which translated from Usenet admin speak means my server drops a lot of spam and garbage posts, and if you want those you'll have to get them from another peer.