8 ms·

Daily Usenet Feed Size Hits 300TB

70 points by xhrpost 2y ago

surteen 2y ago
Gosh that's a lot of discussion!
- tombert 2y ago
  Yeah, it actually requires a lot of work on the posters' end because they choose to do all their discussions in the form of videos apparently.
  - Scoundreller 2y ago
    A lot of very artistic form of discussion
yieldcrv 2y ago
where's the action nowadays?
- smartdude123 2y ago
  [flagged]
eesmith 2y ago
So, binaries, warez, images, and the like? A backchannel to download papers from sci-hub? Or are people using Usenet as a way to send encrypted messages in a way that makes traffic analysis more difficult? (If 50,000 people download everything to a group, and post encrypted or steganographic message to that group, then it's easier than seeing that X sent an email blob to Y.) Or, Usenet as the new numbers station?
- Scoundreller 2y ago
  I always encrypt my “hi mom” messages as multigigabyte .rar files to blend in.
garciasn 2y ago
Based on this table there's been a 1100% increase over the last 7 years, with a 60% increase between 2020 and 2021. I'd be interested to see WHY this is the case. Is it attributable to a larger share of data that cannot be compressed vs more compressible data (e.g., Warez/Movies)? It just seems highly unlikely this is driven by a growing user base; but, without more details other than this data table, I am at a loss for the reasons why.
- joecool1029 2y ago
  It is a growing user base. Streaming services were convenient, now they are not (ads, increased cost, reduced content availability, fragmentation). > It just seems highly unlikely this is driven by a growing user base; If you are old enough, there was a time when everyone pirated stuff due to the alternative being rather expensive or unavailable (physical media). Then a golden age of streaming services that were cheap and had high availability basically killed torrenting for the general public. Now people are returning to piracy as the streaming services got worse for reasons I stated above.
- mapontosevenths 2y ago
  One of the most popular types of content is now remuxed (not reencoded or recompressed) 4k blu-ray rips that can be anywhere from 50-150Gb per disc. In the DVD/Blu-ray era movies were often reencoded to lower bitrates the way streaming services now do and had tiny original sources. Those files were orders of magnitude smaller (600Mb-1.5Gb typically). The people who still care enough to pirate in the era of ubiquitous streaming are often doing it explicitly to get the best possible quality, because no streaming service now offers anything of comparable fidelity for any price.
  - alickz 2y ago
    some (most?) streaming services don't even allow you to manually set the quality anymore gotta hope they decide your connection is capable of 4k (and they decide they want to spare you the bandwidth)
crazygringo 2y ago
Serious question: why does anybody use Usenet for pirating rather than torrents? It seems so fundamentally ill-suited to the task. And if the answer has something to do with privacy or warnings from your ISP, it seems like VPNs would be the answer. What am I missing?
- TheSecondMouse 2y ago
  Because it will max out your connection speed, got a 5Gb connection? no problem, it will instantly start downloading at full tilt.
- ocharles 2y ago
  Honestly there's way more there, and you get consistent solid speeds. Find a provider with a lot of retention and you can find almost all mainstream media regardless of it's age. (Public) torrents tend to track what's popular and quickly fade. The masses seem to favour low size encodes too, so if you're looking for more quality (and again, public trackers) you're usually much more out of luck.
- Ekaros 2y ago
  Download speeds, no ratios, very low risk of any action by right holders. Would be things I would consider. Also setting VPN right is not entirely untrivial, you could get appearance it being in use, but traffic actual passing through other ways.
  - lifty 2y ago
    Why are the risks of action by right holders low?
    - theshrike79 2y ago
      At least at one point news servers were in a kind of a limbo rights-wise. They were kinda like ISPs providing a service and thus weren't liable. Just like I can send you a copyrighted image via email and the email server owner isn't liable for the material on their drive. Also it's hard to track compared to the ease of monitoring public torrent swarms. With torrents you can just enter the swarm, log all IP addresses and maybe download a bit that it's actually infringing material. With news the stuff is ascii jumble spread in posts with weird names and you need to download it all to figure out if it's what it says. And just looking for material from the 300TB backlog is a huge pain.
Ekaros 2y ago
Wow, is this all AI generated content? AI bots just discussing between each other constantly? Is there really even that much media produced everyday? Or media that does get uploaded?
- nerdix 2y ago
  You can post binaries on Usenet. And a lot of the stuff is dupes (the same content being re-ripped in different formats and posted by different groups). Any particular piece of content might be uploaded 50 times or more with different resolutions, codecs, sound formats, languages, etc.
  - joecool1029 2y ago
    That's part of it, but also most every binary post has additional redundancy added in the form of par2 parity files. This is so if some of the articles don't make it to the news server or some get DMCA'd, it's possible to calculate and repair downloads. Also, due to the same claims, posters just re-upload the same thing again.
freeqaz 2y ago
So 300TB * 365 = ~110PB for a mirror to have 1 year of retention. That's pretty insane lol. How many mirrors are there that can actually manage that much storage? If you're using 20TB disks that's ~5500 disks per year with zero redundancy. Double or triple that for a bare minimum... not counting the load of actually serving that data to everybody. How is this economical for anybody at this point? Or are these Usenet mirrors all massive businesses that can support running hundreds of PBs or storage and I'm just naive?
- crazygringo 2y ago
  I'm also curious -- how come you hear big stories about governments going after torrent sites, which don't even host the pirated content. But I've never heard big stories about the government going after big Usenet providers. Do they? If not, why not? Or does it just not make the news? Or am I just not paying attention?
  - fckgw 2y ago
    Usenet files get taken down all the time. Rights holders send notices to the Usenet providers and they remove enough of the binaries where they can't be repaired by the PAR files. Indexers are separate entities. Those are the ones who catalog and host the .nzb files. You see those go offline far more often but Usenet is still pretty small compared to torrents so they don't get reported as much.
  - joecool1029 2y ago
    > But I've never heard big stories about the government going after big Usenet providers. The most litigious government has a 'safe harbor' provision most of the major Usenet backbones comply with: https://www.copyright.gov/512/ https://www.copyright.gov/512/ And they do take down a substantial amount of content every day, so while everyone knows piracy is common there, what more are they supposed to do? At this point the main effective thing governments can do to make it annoying to use Usenet for piracy is target the indexers (and they do this, but more always move in to replace the ones that are shut down)
- Cyberdog 2y ago
deleted 2y ago
[deleted]
vessenes 2y ago
rn seems to have had it right: ripped from my youth, the pre-posting warning comes to mind: “This program posts news to thousands of machines throughout the entire civilized world. You message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing.”
joecool1029 2y ago
Feel like it's obligatory to say this for people unfamiliar with Usenet, but there are text groups (think text threads like a mailing list or forum) and binary groups (shows/movies/software/etc). The size growth is due to the latter. I use both types. Text groups almost certainly fell off a cliff earlier this year when Google Groups shut off their spam posting gateway. http://www.eternal-september.org/ http://www.eternal-september.org/ is a good free project for the text groups. I've been a newsdemon customer for years but they suck for text groups (their headers got messed up a few years back after a move from highwinds backbone to usenetexpress backbone).
486sx33 2y ago
And a TON of it is spam and “robots talking to robots”
- eschaton 2y ago
  To a first approximation none of it is, because it’s drowned out by the absurdly massive amounts of piracy. When it comes to text-only groups there’s still some spam but a lot of it is cranks howling into the void about their insane world views rather than unsolicited and off-topic commercial offers, though those do still exist too.
rekabis 2y ago
Wow. Makes me want to break out my archived installer for Microplanet Gravity, assuming it even works on Windows 10 or 11.

70 points by xhrpost 2y ago

surteen 2y ago

Gosh that's a lot of discussion!

tombert 2y ago
Yeah, it actually requires a lot of work on the posters' end because they choose to do all their discussions in the form of videos apparently.
- Scoundreller 2y ago
  A lot of very artistic form of discussion

yieldcrv 2y ago

where's the action nowadays?

smartdude123 2y ago
[flagged]

eesmith 2y ago

So, binaries, warez, images, and the like? A backchannel to download papers from sci-hub? Or are people using Usenet as a way to send encrypted messages in a way that makes traffic analysis more difficult? (If 50,000 people download everything to a group, and post encrypted or steganographic message to that group, then it's easier than seeing that X sent an email blob to Y.) Or, Usenet as the new numbers station?

Scoundreller 2y ago
I always encrypt my “hi mom” messages as multigigabyte .rar files to blend in.

garciasn 2y ago

Based on this table there's been a 1100% increase over the last 7 years, with a 60% increase between 2020 and 2021. I'd be interested to see WHY this is the case. Is it attributable to a larger share of data that cannot be compressed vs more compressible data (e.g., Warez/Movies)? It just seems highly unlikely this is driven by a growing user base; but, without more details other than this data table, I am at a loss for the reasons why.

joecool1029 2y ago
It is a growing user base. Streaming services were convenient, now they are not (ads, increased cost, reduced content availability, fragmentation). > It just seems highly unlikely this is driven by a growing user base; If you are old enough, there was a time when everyone pirated stuff due to the alternative being rather expensive or unavailable (physical media). Then a golden age of streaming services that were cheap and had high availability basically killed torrenting for the general public. Now people are returning to piracy as the streaming services got worse for reasons I stated above.
mapontosevenths 2y ago
One of the most popular types of content is now remuxed (not reencoded or recompressed) 4k blu-ray rips that can be anywhere from 50-150Gb per disc. In the DVD/Blu-ray era movies were often reencoded to lower bitrates the way streaming services now do and had tiny original sources. Those files were orders of magnitude smaller (600Mb-1.5Gb typically). The people who still care enough to pirate in the era of ubiquitous streaming are often doing it explicitly to get the best possible quality, because no streaming service now offers anything of comparable fidelity for any price.
- alickz 2y ago
  some (most?) streaming services don't even allow you to manually set the quality anymore gotta hope they decide your connection is capable of 4k (and they decide they want to spare you the bandwidth)

crazygringo 2y ago

Serious question: why does anybody use Usenet for pirating rather than torrents? It seems so fundamentally ill-suited to the task. And if the answer has something to do with privacy or warnings from your ISP, it seems like VPNs would be the answer. What am I missing?

TheSecondMouse 2y ago
Because it will max out your connection speed, got a 5Gb connection? no problem, it will instantly start downloading at full tilt.
ocharles 2y ago
Honestly there's way more there, and you get consistent solid speeds. Find a provider with a lot of retention and you can find almost all mainstream media regardless of it's age. (Public) torrents tend to track what's popular and quickly fade. The masses seem to favour low size encodes too, so if you're looking for more quality (and again, public trackers) you're usually much more out of luck.
Ekaros 2y ago
Download speeds, no ratios, very low risk of any action by right holders. Would be things I would consider. Also setting VPN right is not entirely untrivial, you could get appearance it being in use, but traffic actual passing through other ways.
- lifty 2y ago
  Why are the risks of action by right holders low?
  - theshrike79 2y ago
    At least at one point news servers were in a kind of a limbo rights-wise. They were kinda like ISPs providing a service and thus weren't liable. Just like I can send you a copyrighted image via email and the email server owner isn't liable for the material on their drive. Also it's hard to track compared to the ease of monitoring public torrent swarms. With torrents you can just enter the swarm, log all IP addresses and maybe download a bit that it's actually infringing material. With news the stuff is ascii jumble spread in posts with weird names and you need to download it all to figure out if it's what it says. And just looking for material from the 300TB backlog is a huge pain.

Ekaros 2y ago

Wow, is this all AI generated content? AI bots just discussing between each other constantly? Is there really even that much media produced everyday? Or media that does get uploaded?

nerdix 2y ago
You can post binaries on Usenet. And a lot of the stuff is dupes (the same content being re-ripped in different formats and posted by different groups). Any particular piece of content might be uploaded 50 times or more with different resolutions, codecs, sound formats, languages, etc.
- joecool1029 2y ago
  That's part of it, but also most every binary post has additional redundancy added in the form of par2 parity files. This is so if some of the articles don't make it to the news server or some get DMCA'd, it's possible to calculate and repair downloads. Also, due to the same claims, posters just re-upload the same thing again.

freeqaz 2y ago

So 300TB * 365 = ~110PB for a mirror to have 1 year of retention. That's pretty insane lol. How many mirrors are there that can actually manage that much storage? If you're using 20TB disks that's ~5500 disks per year with zero redundancy. Double or triple that for a bare minimum... not counting the load of actually serving that data to everybody. How is this economical for anybody at this point? Or are these Usenet mirrors all massive businesses that can support running hundreds of PBs or storage and I'm just naive?

crazygringo 2y ago
I'm also curious -- how come you hear big stories about governments going after torrent sites, which don't even host the pirated content. But I've never heard big stories about the government going after big Usenet providers. Do they? If not, why not? Or does it just not make the news? Or am I just not paying attention?
- fckgw 2y ago
  Usenet files get taken down all the time. Rights holders send notices to the Usenet providers and they remove enough of the binaries where they can't be repaired by the PAR files. Indexers are separate entities. Those are the ones who catalog and host the .nzb files. You see those go offline far more often but Usenet is still pretty small compared to torrents so they don't get reported as much.
- joecool1029 2y ago
  > But I've never heard big stories about the government going after big Usenet providers. The most litigious government has a 'safe harbor' provision most of the major Usenet backbones comply with: https://www.copyright.gov/512/ https://www.copyright.gov/512/ And they do take down a substantial amount of content every day, so while everyone knows piracy is common there, what more are they supposed to do? At this point the main effective thing governments can do to make it annoying to use Usenet for piracy is target the indexers (and they do this, but more always move in to replace the ones that are shut down)
Cyberdog 2y ago

deleted 2y ago

[deleted]

vessenes 2y ago

rn seems to have had it right: ripped from my youth, the pre-posting warning comes to mind: “This program posts news to thousands of machines throughout the entire civilized world. You message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing.”

joecool1029 2y ago

Feel like it's obligatory to say this for people unfamiliar with Usenet, but there are text groups (think text threads like a mailing list or forum) and binary groups (shows/movies/software/etc). The size growth is due to the latter. I use both types. Text groups almost certainly fell off a cliff earlier this year when Google Groups shut off their spam posting gateway. http://www.eternal-september.org/ http://www.eternal-september.org/ is a good free project for the text groups. I've been a newsdemon customer for years but they suck for text groups (their headers got messed up a few years back after a move from highwinds backbone to usenetexpress backbone).

486sx33 2y ago

And a TON of it is spam and “robots talking to robots”

eschaton 2y ago
To a first approximation none of it is, because it’s drowned out by the absurdly massive amounts of piracy. When it comes to text-only groups there’s still some spam but a lot of it is cranks howling into the void about their insane world views rather than unsolicited and off-topic commercial offers, though those do still exist too.

rekabis 2y ago

Wow. Makes me want to break out my archived installer for Microplanet Gravity, assuming it even works on Windows 10 or 11.