4 ms·

The EFF just recently wrote an article with instructions on how to persevere & archive your own tweets on the Wayback Machine, but it involves exporting your ow

by pogue 3y ago

The EFF just recently wrote an article with instructions on how to persevere & archive your own tweets on the Wayback Machine, but it involves exporting your own backup and uploading it to them. Since the API is completely cut off from Twitter, there is no official way to backup other people's accounts. But archive.today uses scraping and all sorts of tricky methods to bypass paywalls. I honestly don't understand why Nitter can't just stay logged out and rotate IPs. Although I'm sure that gets pricey when other people are accessing it constantly. https://www.eff.org/deeplinks/2024/01/save-your-twitter-account https://www.eff.org/deeplinks/2024/01/save-your-twitter-acco...

toomuchtodo 3y ago
If the scraping model is impaired due to aggressive countermeasures, end game are browser extensions that scrape as users view the site and ship scraped data back to a processor, similar to recap the law (uses an extension to scrape the PACER legal database and ship digital artifacts to the Internet Archive). Care will need to be taken around potentially sensitive data that could be shipped if users are logged in. https://free.law/recap https://free.law/recap
- pogue 3y ago
  Oh, that's a very cool project! How successful has it been? If it wasn't for Sci-hub that would be a great idea for the scientific publishing world as well.
  - toomuchtodo 3y ago
    Very successful. Millions of court filings extracted and indexed, and a crucial component in driving down PACER costs. https://news.ycombinator.com/item?id=24086570 https://news.ycombinator.com/item?id=24086570 > I'm the director of Free Law Project. For the case mentioned in the article we actually did a full expert testimony figuring out roughly how much per page it'd cost to run PACER using AWS GovCloud and a handful of other assumptions. It was...half a ten thousandth of a penny per page https://www.courtlistener.com/docket/4214664/52/15/national-veterans-legal-services-program-v-united-states/ https://www.courtlistener.com/docket/4214664/52/15/national-... https://news.ycombinator.com/item?id=24085158 https://news.ycombinator.com/item?id=24085158 > Government’s PACER Fees Are Too High, Federal Circuit Says https://news.bloomberglaw.com/white-collar-and-criminal-law/federal-circuit-affirms-limits-on-how-government-uses-pacer-fees https://news.bloomberglaw.com/white-collar-and-criminal-law/...
- phone8675309 3y ago
  This model also works well for deep web content archiving. There was a gaming message board where someone wrote a browser extension that would back up all topics someone visited in the background while they were reading them. It became important for archiving as much content from those forums as possible as the forum was in the process of shutting down.

by pogue 3y ago

toomuchtodo 3y ago

If the scraping model is impaired due to aggressive countermeasures, end game are browser extensions that scrape as users view the site and ship scraped data back to a processor, similar to recap the law (uses an extension to scrape the PACER legal database and ship digital artifacts to the Internet Archive). Care will need to be taken around potentially sensitive data that could be shipped if users are logged in. https://free.law/recap https://free.law/recap

pogue 3y ago
Oh, that's a very cool project! How successful has it been? If it wasn't for Sci-hub that would be a great idea for the scientific publishing world as well.
- toomuchtodo 3y ago
  Very successful. Millions of court filings extracted and indexed, and a crucial component in driving down PACER costs. https://news.ycombinator.com/item?id=24086570 https://news.ycombinator.com/item?id=24086570 > I'm the director of Free Law Project. For the case mentioned in the article we actually did a full expert testimony figuring out roughly how much per page it'd cost to run PACER using AWS GovCloud and a handful of other assumptions. It was...half a ten thousandth of a penny per page https://www.courtlistener.com/docket/4214664/52/15/national-veterans-legal-services-program-v-united-states/ https://www.courtlistener.com/docket/4214664/52/15/national-... https://news.ycombinator.com/item?id=24085158 https://news.ycombinator.com/item?id=24085158 > Government’s PACER Fees Are Too High, Federal Circuit Says https://news.bloomberglaw.com/white-collar-and-criminal-law/federal-circuit-affirms-limits-on-how-government-uses-pacer-fees https://news.bloomberglaw.com/white-collar-and-criminal-law/...
phone8675309 3y ago
This model also works well for deep web content archiving. There was a gaming message board where someone wrote a browser extension that would back up all topics someone visited in the background while they were reading them. It became important for archiving as much content from those forums as possible as the forum was in the process of shutting down.