10 ms·

Old usenet-head here (on it regularly from 1991, first met it 1986) ... First problem: there's no identity authentication mechanism in NNTP. So spam is a probl

by cstross 11y ago

Old usenet-head here (on it regularly from 1991, first met it 1986) ... First problem: there's no identity authentication mechanism in NNTP. So spam is a problem, forged moderation headers are a problem, general abuse is a problem. (A modern syndicated forum system with OAuth or some successor model would be a lot easier to ride herd on.) Second problem: storage demands expand faster than the user base. Because it's a flood-fill store-and-forward system, each server node tries to replicate the entire feed. Consequently news admins tended to put a short expiry on posts in binary groups so they'd be deleted fairly promptly ... but if you do that, the lusers can't find what they're looking for so they ask their friends to repost the bloody things, ad nauseam. Third problem: etiquette. Yeah, yeah, I am coming over all elitist here, but the original usenet mindset was exactly that. These days we're used to being overrun by everyone who can use a point-and-drool interface on their phone to look at Facebook, but back in September 1992 it was a real shock to the system when usenet was suddenly gatewayed onto AOL, I can tell you. Previously usenet more or less got along because the users were university staff and students (who could be held accountable to some extent) and computer industry folks. Thereafter, well, a lot of the worse aspects of 4chan and Reddit were pioneered on usenet. (Want to know why folks hero-worshipped Larry Wall before he wrote Perl? Because he wrote this thing called rn(1). Which had killfiles.) Anyway, a side-effect of this was that when web browsers began to show up, the response was to double-down on the high-powered CURSES-based or pure command-line clients rather than to try and figure out how to put an easy-to-use interface on top of a news spool. Upshot: usenet clients remained rooted in the early 1990s at best. These days much of the functionality of usenet (minus the binaries) is provided by Reddit. Usenet itself turned into a half-assed space-hogging brain dead file sharing network. And we know what ISPs think of space-hogging half-assed stuff that doesn't make them money and risks getting them sued.

pyre 11y ago
> (Want to know why folks hero-worshipped Larry Wall before he wrote Perl? Because he wrote this thing called rn(1). Which had killfiles.) Perl came into being in 1987, which is before September 1992, so his creation of rn must have had traction/worship before the "Eternal September." I'd hardly say that said event was the reason that people worshipped him for rn, especially since Perl was out for ~5 years at that point.
- dsr_ 11y ago
  We needed killfiles well before September began. (Go read the net.legends FAQ.) And Perl wasn't a great language before 4, in 1993, and didn't attain anywhere near peak popularity until 5.
  - dfan 11y ago
    Just to get the chronology straight, Perl 4 was released in 1991; the first Perl 5 was released in 1994.
- GFK_of_xmaspast 11y ago
  (a) the eternal September was 1993 and (b) there were plenty of bozos around well before then.
davidgerard 11y ago
I wrote a piece thinking about why nobody ever did write a good web interface for NNTP: http://reddragdiva.dreamwidth.org/566555.html http://reddragdiva.dreamwidth.org/566555.html my thesis: the atom of NNTP is the message, the atom of web forums is the thread. (Commenters on that are mostly old BOFHnetters you may still recognise.)
- baudehlo 11y ago
  It's more than that. The design of NNTP is much like POP3 - you download entire threads and messages all at once. It doesn't work all that well on the web compared with say IMAP.
  - davidgerard 11y ago
    Yeah, there's a number of reasons. But I do think that's why there's been no good web front end ever.
- RyJones 11y ago
  NNTP should replace RSS and comments to posts would be newsgroup posts.
  - asjo 11y ago
    gwene.org is half way there (RSS/Atom to NNTP, read-only).
- codinghorror 11y ago
  As alluded to later in the replies here, no discussion website ever tried to be fully decentralized like NNTP was from inception. It is a super hard problem, and as the declining popularity of NNTP itself illustrates (and the tendency toward mega-centralization on the web like Facebook, too) maybe it was solving the wrong problem for today.
  - cstross 11y ago
    I think the modern centralization tendency is a bad thing. If I was going to redesign usenet today ...? 1. You want non-repudiability in transactions: a given user owns their posts or cancel messages and can be linked to them. So, use a blockchain-based solution for authentication. (Not bitcoin; a separate non-financialized system.) 2. Instead of local spools consisting of a directory tree and sync'd via flood-fill, implement it as a virtual filesystem, peer-to-peer discovery, locally requested contents are cached, so high-traffic newsgroups will be cached more widely and therefore be faster to load. 3. Anti-spam: a big problem is preventing spammers sprouting new sock-puppet identities to get around already-imposed blocks. Hence non-repudiability. I'm not sure how to go about enforcing limits on sock-puppeting: this is a hard problem. But if the authentication mechanism is blockchain based we might be able to link it to an underlying funding system and thereby impose a monetary cost on posting -- even $0.01 per message should be enough to put a drag-brake on spamming. This was always the weakness of usenet: spammers externalized all their costs.
wtbob 11y ago
> First problem: there's no identity authentication mechanism in NNTP. Yup. Which is weird, since every message is sent by someone at a host; it should have been possible to simply use signatures to prove which site generated a message—and punish sites which didn't police their users. But crypto was hard (and illegal to export, once upon a time). > Because it's a flood-fill store-and-forward system, each server node tries to replicate the entire feed. The upshot of this is that reading news was fast. So fast that folks these days can't believe what a user-friendly experience reading from a local news feed can be. Imagine reading a website where pages come up in milliseconds—that's Usenet on a local feed. > Consequently news admins tended to put a short expiry on posts in binary groups Frankly, binaries were Usenet's downfall. Had those been eliminated, then I hope that Usenet would be a lot healthier today. I couldn't get numbers on the size of a daily text feed nowadays, but I imagine it's pretty manageable. > Upshot: usenet clients remained rooted in the early 1990s at best. Back in the 90s I used a pretty nice Mac GUI client. But really, it doesn't get better than gnus… Anyway, Usenet's not dead—it's still alive, people are still posting and some groups are doing pretty well. They world could use a new Usenet, with the lessons learned from the first one: site-to-site; non-commercial; anti-spam measures built in.
- pavlov 11y ago
  Imagine reading a website where pages come up in milliseconds—that's Usenet on a local feed. Now that you mention it, I think the web has never reached the speed of Usenet ages ago. I propose the following law: "The size of web pages must expand so that, regardless of CPU and bandwidth improvements, a page will load slower than a Usenet post on a 9.6kbps modem in 1993."
  - nullc 11y ago
    Dunno if you've tried a gopher client recently; but it's shocking how much faster than the web it is... (though still holds nothing on a local mailbox or news spool).
justwannasing 11y ago
I especially agree with your third point. Any kid getting his first Windows machine was now a computer expert and ran to the (then) only knowledgeable place to get more info, and that was usenet. That turned usenet into reddit without any controls at all. Reddit, at least, does some minimal clean up.
tptacek 11y ago
Yes. Binaries killed Usenet. I ran a Freenix-competitive Usenet server for a popular ISP in the mid-1990s (by way of bona fides: we were competitive because I hacked a history lookup cache into INN, a concept we apparently co-invented alongside Netcom). Usenet was by far our most expensive and most time-consuming infrastructure. The reason for that was binaries. The amount of storage we were required to keep online for binaries was staggering. We ended up buying those ridiculous chrome-plated NetApp fileservers to handle the load. The hardware was expensive, but more expensive was the admin overhead: things went wrong with the INN filesystems regularly, and there was nothing you could do to recover from them quickly; simple filesystem errors that were really just an fsck away from repair could mean 4-6 hours of downtime. Which, by the way, tended to happen at night. File-sharers sprayed multiple copies of huge files across several newsgroups, in little chunks. If any of those chunks went missing, our users screamed bloody murder. ISPs that tried to host no-binaries Usenet became the target of PR campaigns. Hosting discussion groups on Usenet was cheap and easy. Running a competitive full-feed server, on the other hand, required nearly full-time attention from an admin that could do light filesystem hacking. The result was a death-spiral: as Usenet got more expensive to host, fewer ISPs hosted it; many outsourced to other providers. They could easily have hosted just the discussion groups! Usenet could have been kept alive and decentralized, and maybe even evolved alongside the web. Instead, software and pornography pirates coerced the network into a few centralized providers, who eventually decided not to waste huge amounts of money hosting infrastructure for those kinds of users. Not that I'm bitter.
- Tomte 11y ago
  At least in Germany the death had other causes. And I suspect, these causes were also main causes in the US. Here, almost no servers carried binaries to begin with. Usenet was seen as a text-only medium, because binaries were so huge and mostly illegal. And it thrived for a time. There were certain niches that I still miss. Legal discussions were fantastic, with many lawyers, prosecutors and judges participating. What killed it was that there simply wasn't fresh blood. No mindshare, no articles in computer magazines about Usenet, strictly (and sometimes viciously) defended netiquette. People liked web forums, and I can see why. Threaded discussions seem tedious. No special software. I even found out that my Usenet-influenced quoting style, answering point by point and paragraph by paragraph is seen by some people who were never socialized in that atmosphere as "hostile". "You're trying to attack every one of my sentences". Pure culture clash.

by cstross 11y ago

pyre 11y ago

> (Want to know why folks hero-worshipped Larry Wall before he wrote Perl? Because he wrote this thing called rn(1). Which had killfiles.) Perl came into being in 1987, which is before September 1992, so his creation of rn must have had traction/worship before the "Eternal September." I'd hardly say that said event was the reason that people worshipped him for rn, especially since Perl was out for ~5 years at that point.

dsr_ 11y ago
We needed killfiles well before September began. (Go read the net.legends FAQ.) And Perl wasn't a great language before 4, in 1993, and didn't attain anywhere near peak popularity until 5.
- dfan 11y ago
  Just to get the chronology straight, Perl 4 was released in 1991; the first Perl 5 was released in 1994.
GFK_of_xmaspast 11y ago
(a) the eternal September was 1993 and (b) there were plenty of bozos around well before then.

davidgerard 11y ago

I wrote a piece thinking about why nobody ever did write a good web interface for NNTP: http://reddragdiva.dreamwidth.org/566555.html http://reddragdiva.dreamwidth.org/566555.html my thesis: the atom of NNTP is the message, the atom of web forums is the thread. (Commenters on that are mostly old BOFHnetters you may still recognise.)

baudehlo 11y ago
It's more than that. The design of NNTP is much like POP3 - you download entire threads and messages all at once. It doesn't work all that well on the web compared with say IMAP.
- davidgerard 11y ago
  Yeah, there's a number of reasons. But I do think that's why there's been no good web front end ever.
RyJones 11y ago
NNTP should replace RSS and comments to posts would be newsgroup posts.
- asjo 11y ago
  gwene.org is half way there (RSS/Atom to NNTP, read-only).
codinghorror 11y ago
As alluded to later in the replies here, no discussion website ever tried to be fully decentralized like NNTP was from inception. It is a super hard problem, and as the declining popularity of NNTP itself illustrates (and the tendency toward mega-centralization on the web like Facebook, too) maybe it was solving the wrong problem for today.
- cstross 11y ago
  I think the modern centralization tendency is a bad thing. If I was going to redesign usenet today ...? 1. You want non-repudiability in transactions: a given user owns their posts or cancel messages and can be linked to them. So, use a blockchain-based solution for authentication. (Not bitcoin; a separate non-financialized system.) 2. Instead of local spools consisting of a directory tree and sync'd via flood-fill, implement it as a virtual filesystem, peer-to-peer discovery, locally requested contents are cached, so high-traffic newsgroups will be cached more widely and therefore be faster to load. 3. Anti-spam: a big problem is preventing spammers sprouting new sock-puppet identities to get around already-imposed blocks. Hence non-repudiability. I'm not sure how to go about enforcing limits on sock-puppeting: this is a hard problem. But if the authentication mechanism is blockchain based we might be able to link it to an underlying funding system and thereby impose a monetary cost on posting -- even $0.01 per message should be enough to put a drag-brake on spamming. This was always the weakness of usenet: spammers externalized all their costs.

wtbob 11y ago

> First problem: there's no identity authentication mechanism in NNTP. Yup. Which is weird, since every message is sent by someone at a host; it should have been possible to simply use signatures to prove which site generated a message—and punish sites which didn't police their users. But crypto was hard (and illegal to export, once upon a time). > Because it's a flood-fill store-and-forward system, each server node tries to replicate the entire feed. The upshot of this is that reading news was fast. So fast that folks these days can't believe what a user-friendly experience reading from a local news feed can be. Imagine reading a website where pages come up in milliseconds—that's Usenet on a local feed. > Consequently news admins tended to put a short expiry on posts in binary groups Frankly, binaries were Usenet's downfall. Had those been eliminated, then I hope that Usenet would be a lot healthier today. I couldn't get numbers on the size of a daily text feed nowadays, but I imagine it's pretty manageable. > Upshot: usenet clients remained rooted in the early 1990s at best. Back in the 90s I used a pretty nice Mac GUI client. But really, it doesn't get better than gnus… Anyway, Usenet's not dead—it's still alive, people are still posting and some groups are doing pretty well. They world could use a new Usenet, with the lessons learned from the first one: site-to-site; non-commercial; anti-spam measures built in.

pavlov 11y ago
Imagine reading a website where pages come up in milliseconds—that's Usenet on a local feed. Now that you mention it, I think the web has never reached the speed of Usenet ages ago. I propose the following law: "The size of web pages must expand so that, regardless of CPU and bandwidth improvements, a page will load slower than a Usenet post on a 9.6kbps modem in 1993."
- nullc 11y ago
  Dunno if you've tried a gopher client recently; but it's shocking how much faster than the web it is... (though still holds nothing on a local mailbox or news spool).

justwannasing 11y ago

I especially agree with your third point. Any kid getting his first Windows machine was now a computer expert and ran to the (then) only knowledgeable place to get more info, and that was usenet. That turned usenet into reddit without any controls at all. Reddit, at least, does some minimal clean up.

tptacek 11y ago

Yes. Binaries killed Usenet. I ran a Freenix-competitive Usenet server for a popular ISP in the mid-1990s (by way of bona fides: we were competitive because I hacked a history lookup cache into INN, a concept we apparently co-invented alongside Netcom). Usenet was by far our most expensive and most time-consuming infrastructure. The reason for that was binaries. The amount of storage we were required to keep online for binaries was staggering. We ended up buying those ridiculous chrome-plated NetApp fileservers to handle the load. The hardware was expensive, but more expensive was the admin overhead: things went wrong with the INN filesystems regularly, and there was nothing you could do to recover from them quickly; simple filesystem errors that were really just an fsck away from repair could mean 4-6 hours of downtime. Which, by the way, tended to happen at night. File-sharers sprayed multiple copies of huge files across several newsgroups, in little chunks. If any of those chunks went missing, our users screamed bloody murder. ISPs that tried to host no-binaries Usenet became the target of PR campaigns. Hosting discussion groups on Usenet was cheap and easy. Running a competitive full-feed server, on the other hand, required nearly full-time attention from an admin that could do light filesystem hacking. The result was a death-spiral: as Usenet got more expensive to host, fewer ISPs hosted it; many outsourced to other providers. They could easily have hosted just the discussion groups! Usenet could have been kept alive and decentralized, and maybe even evolved alongside the web. Instead, software and pornography pirates coerced the network into a few centralized providers, who eventually decided not to waste huge amounts of money hosting infrastructure for those kinds of users. Not that I'm bitter.

Tomte 11y ago
At least in Germany the death had other causes. And I suspect, these causes were also main causes in the US. Here, almost no servers carried binaries to begin with. Usenet was seen as a text-only medium, because binaries were so huge and mostly illegal. And it thrived for a time. There were certain niches that I still miss. Legal discussions were fantastic, with many lawyers, prosecutors and judges participating. What killed it was that there simply wasn't fresh blood. No mindshare, no articles in computer magazines about Usenet, strictly (and sometimes viciously) defended netiquette. People liked web forums, and I can see why. Threaded discussions seem tedious. No special software. I even found out that my Usenet-influenced quoting style, answering point by point and paragraph by paragraph is seen by some people who were never socialized in that atmosphere as "hostile". "You're trying to attack every one of my sentences". Pure culture clash.