5 ms·

> If somebody else wants to take over the concept, I can FedEx you a disk containing the archive (as an NNTP spool). Lights candles to summon Jason Scott to in

by anc84 10y ago

> If somebody else wants to take over the concept, I can FedEx you a disk containing the archive (as an NNTP spool). Lights candles to summon Jason Scott to ingest it into the Internet Archive

naveen99 10y ago
or upload it to google big query or a self pay amazon s3 bucket... public datasets get free storage iirc. would be a nice complement to the reddit / hn datasets.
- forgotpwtomain 10y ago
  > public datasets get free storage iirc I was not aware of this. Do you have a link? edit: self clarifying my own question. Not sure what the specifics for hosting an AWS Public data set are though. Big Query >A public dataset is any dataset that is stored in BigQuery and made available to the general public. This page lists a special group of public datasets that Google BigQuery hosts for you to access and integrate into your applications. Google pays for the storage of these data sets and provides public access to the data via BigQuery. You pay only for the queries that you perform on the data (the first 1 TB per month is free, subject to query pricing details). [0] https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/ AWS > Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications. AWS is hosting the public data sets at no charge for the community, and like all AWS services, users pay only for the compute and storage they use for their own applications. Learn more about Public Data Sets on AWS and visit the Public Data Sets forum. [1] https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/
- jks 10y ago
  In https://web.archive.org/web/20160708085949/http://gmane.org/about.php https://web.archive.org/web/20160708085949/http://gmane.org/... (in the section titled "A Statement") Lars wrote he wouldn't "use any of the contents of the Gmane news spool for spamming, for advertisement, for sending mass mail, for gathering profiling information, or anything of that kind, or willingly allow anybody else to do the same". So I assume he wouldn't be fine with just making it all publicly downloadable but would want his successor to provide a similar guarantee.
- djsumdog 10y ago
  I've worked on research projects that turned into a startup. We migrated all our data from free University boxes to cloud storage the startup paid for. The professor I was working with told us about how he lost all his data once with a similar startup attempt. Shortly after I was let go from the startup I pulled all the data via the web service I wrote, luckily before they shut it down. TL;DR I'd trust an entity like the Internet Archive way over commercial entities, even those that offer free storage for open projects. Sure the Internet Archive could up and go away, but I still trust their motives and their commitment to not let that happen.
  - derefr 10y ago
    Plus, it's not like the Internet Archive couldn't store your archive as a public dataset in AWS to reduce their own costs, if they were to decide AWS fit their criteria as a reliable public archive host. The difference would be that if AWS itself wanted to stop providing public archive hosting, they'd just up and do that. Whereas if the Internet Archive was hosting a public archive on AWS, and AWS cut off support for public archives, the Internet Archive would then work to migrate the dataset off AWS. Non-profits have incentives to do (whatever their charter says they do.) Corporations just have an incentive to make money. If you want a certain "preference function" to carry on into the future without self-modifying into something unrecognizable, you probably want to instantiate that preference-function in the form of a non-profit.
- kbaker 10y ago
  Or torrent it! This is the best use case for a torrent... spread the load around, no extra cost or having to find someone to flip a 'public dataset = free' bit.
catchmrbharath 10y ago
> Lights candles to summon Jason Scott to ingest it into the Internet Archive And he has responded. https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/comment-page-1/#comment-13501 https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/com...
e12e 10y ago
I wonder how much traffic (DDOS aside) the web bit needs. Also wonder if it would be feasible to just plug the NNTP spool into the d-lang forums[1], and have it "just work" ... ? But no matter what, the best possible solution would probably be to move the archive to the Internet Archive, and have that same archive continue to ingest email->nntp and provide access to it. I get that people fear spammers etc, but at this point, I don't see a way to avoid it if you want a usable web archive. [1] https://github.com/CyberShadow/DFeed https://github.com/CyberShadow/DFeed
yeukhon 10y ago
Archival is only part of the problem. What about the future? I don't mind help out and donate money to build out the infrastructure. But I need a team...

by anc84 10y ago

> If somebody else wants to take over the concept, I can FedEx you a disk containing the archive (as an NNTP spool). Lights candles to summon Jason Scott to ingest it into the Internet Archive

naveen99 10y ago

or upload it to google big query or a self pay amazon s3 bucket... public datasets get free storage iirc. would be a nice complement to the reddit / hn datasets.

forgotpwtomain 10y ago
> public datasets get free storage iirc I was not aware of this. Do you have a link? edit: self clarifying my own question. Not sure what the specifics for hosting an AWS Public data set are though. Big Query >A public dataset is any dataset that is stored in BigQuery and made available to the general public. This page lists a special group of public datasets that Google BigQuery hosts for you to access and integrate into your applications. Google pays for the storage of these data sets and provides public access to the data via BigQuery. You pay only for the queries that you perform on the data (the first 1 TB per month is free, subject to query pricing details). [0] https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/ AWS > Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications. AWS is hosting the public data sets at no charge for the community, and like all AWS services, users pay only for the compute and storage they use for their own applications. Learn more about Public Data Sets on AWS and visit the Public Data Sets forum. [1] https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/
jks 10y ago
In https://web.archive.org/web/20160708085949/http://gmane.org/about.php https://web.archive.org/web/20160708085949/http://gmane.org/... (in the section titled "A Statement") Lars wrote he wouldn't "use any of the contents of the Gmane news spool for spamming, for advertisement, for sending mass mail, for gathering profiling information, or anything of that kind, or willingly allow anybody else to do the same". So I assume he wouldn't be fine with just making it all publicly downloadable but would want his successor to provide a similar guarantee.
djsumdog 10y ago
I've worked on research projects that turned into a startup. We migrated all our data from free University boxes to cloud storage the startup paid for. The professor I was working with told us about how he lost all his data once with a similar startup attempt. Shortly after I was let go from the startup I pulled all the data via the web service I wrote, luckily before they shut it down. TL;DR I'd trust an entity like the Internet Archive way over commercial entities, even those that offer free storage for open projects. Sure the Internet Archive could up and go away, but I still trust their motives and their commitment to not let that happen.
- derefr 10y ago
  Plus, it's not like the Internet Archive couldn't store your archive as a public dataset in AWS to reduce their own costs, if they were to decide AWS fit their criteria as a reliable public archive host. The difference would be that if AWS itself wanted to stop providing public archive hosting, they'd just up and do that. Whereas if the Internet Archive was hosting a public archive on AWS, and AWS cut off support for public archives, the Internet Archive would then work to migrate the dataset off AWS. Non-profits have incentives to do (whatever their charter says they do.) Corporations just have an incentive to make money. If you want a certain "preference function" to carry on into the future without self-modifying into something unrecognizable, you probably want to instantiate that preference-function in the form of a non-profit.
kbaker 10y ago
Or torrent it! This is the best use case for a torrent... spread the load around, no extra cost or having to find someone to flip a 'public dataset = free' bit.

catchmrbharath 10y ago

> Lights candles to summon Jason Scott to ingest it into the Internet Archive And he has responded. https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/comment-page-1/#comment-13501 https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/com...

e12e 10y ago

I wonder how much traffic (DDOS aside) the web bit needs. Also wonder if it would be feasible to just plug the NNTP spool into the d-lang forums[1], and have it "just work" ... ? But no matter what, the best possible solution would probably be to move the archive to the Internet Archive, and have that same archive continue to ingest email->nntp and provide access to it. I get that people fear spammers etc, but at this point, I don't see a way to avoid it if you want a usable web archive. [1] https://github.com/CyberShadow/DFeed https://github.com/CyberShadow/DFeed

yeukhon 10y ago

Archival is only part of the problem. What about the future? I don't mind help out and donate money to build out the infrastructure. But I need a team...