6 ms·

disclaimer: I'm a Member of Archive Team who's helping coordinate the joining of Yahoo Groups in preparation for archival. Yahoo's banning of a large amount of

disclaimer: I'm a Member of Archive Team who's helping coordinate the joining of Yahoo Groups in preparation for archival. Yahoo's banning of a large amount of the accounts we were using is a huge setback for us. In total we lost over access to over 55,000 Yahoo Groups, many of these will now not be archived and will be lost when Yahoo deletes everything on December 14. Particularly disastrous was the loss of access to all of the 30,000 Fandom (fanfic / fanart / etc..) groups that were requested to be archived by members of the fandom community. We're back to square one now, and it is looking increasingly likely that we're only going to be able to re-join (and therefore archive) a small percentage of these groups before December 14. (And now for the inevitable, shameless plug...) We could really use some help! If you've got an hour or so, we could really use people to come and complete CAPTCHAs for us. (A CAPTCHA is needed to join every group). Instructions at: https://github.com/davidferguson/yahoogroups-joiner https://github.com/davidferguson/yahoogroups-joiner

mehhh 7y ago
Have you considered using NordVPN for CAPTCHA bypass? They are a shady company, but their network of residential VPNs is impressive.
scarejunba 7y ago
I imagine you guys already know this but considering we’re up against the timeline, I’d use the captcha solving service (easy to google yourself) and Luminati to distribute the IP addresses while swallowing my ethical qualms.
- StavrosK 7y ago
  I would donate my IP/bandwidth to archive.org if I could run a scraper easily.
  - ignoranceprior 7y ago
    You can install ArchiveTeam Warrior: https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior https://www.archiveteam.org/index.php?title=ArchiveTeam_Warr...
    - squarefoot 7y ago
      Thanks! I never heard of that before; just like project SETI though for archival purposes. What are the hardware requirements of that VM? I'm attempting to import it on my NAS4Free home NAS Virtualbox service which is the only machine I keep up 24/7 atm, but it takes forever to import. The hardware is very limited however (Atom D410 + a bit over 1GB RAM available), so I'm not sure it would succeed, but so far it loads forever, no errors given. I'd like to run it for this project to start contributing quickly albeit with limited hw before the deadline, then find better iron in the future.
      - icebraining 7y ago
        I don't find it processor or memory heavy, it's mostly doing a lot of IO (network and disk).
      - kalleboo 7y ago
        I'm running it on a Synology NAS (Celeron J3455), and the docker manager UI claims it's using 180 MB RAM and less than 1% CPU (and I just confirmed it's currently working on archiving Yahoo! Groups)
      - MandieD 7y ago
        I’m running the Docker image on the smallest Hetzner VMs, with 5 concurrent groups and 40 shared rsync threads per container, and 12 containers per server. Start one container, do docker top on it to make sure it’s pulling, then start the others one by one, taking a few seconds between each to avoid overwhelming the CPU. I’ve got 6 of those little VMs going, and have rolled up 4GB and 2800 groups worth in 6 hours. After they settle down, they’re more memory than processor intensive. I’ve considered playing with the settings a bit, but thought it was more important to get a bunch of them running on a couple different VMs at different sites. If I were really feeling fancy, I’d write a nice deployment definition for orchestrating this with microk8s...
    - Avamander 7y ago
      Unfortunately it doesn't offer a qemu-compatible image or an image that would work when converted, it's a shame and shooting itself in the foot.
john_moscow 7y ago
Forgive my naivety, but why would blocking of your accounts delete the data you have already backed up? This sounds like you are doing it the wrong WAY, IMO.
- Diagon 7y ago
  They hadn't backed up yet. They had set up accounts with yahoo that they were then planning to use to back up those groups. Backups themselves were starting, but they had to go slowly enough not to bog down yahoo's servers.
- betamaxthetape 7y ago
  Two reasons: (a) If we hit Yahoo with everything we've got, groups would have almost certainly crashed, or at least become unbearably slow. That's not a reasonable thing to do, and would be (IMHO) grounds for Verison banning us. (b) We were still testing / writing the scripts to do the actual archiving. Most of the groups we did save before the banning were from test runs of the archiving script. And sure, given hindsight, I'd do things differently. We've learned, now, and are archiving a groups soon after it is joined.
  - john_moscow 7y ago
    OK, thanks for explaining this. Just my 2 cents then: big companies make decisions like this based on the potential PR win/loss. If ignoring you keeps the PR delta at 0, while allowing to export the data exposes them to even a minimal risk (I dunno, someone's private details buried in), they will ignore, or even actively resist you. Politically, you need to arrange it so that cooperating with you will give Verizon a small PR boost, while ignoring you will be seen negatively by the public. This thread had a good example of interesting data that is worth preserving, so I would try reaching out to news companies (NY Times and whatnot) to see if anyone wants to publish a piece. Phrasing this positively and ensuring enough people see it, would greatly increase the chances of cooperation from Verizon.
- deleted 7y ago
  [deleted]
Diagon 7y ago
While the above post is concerned with Fandom groups, my concern is with groups that started doing early community driven biohacking type research. There are medical tests results and discussions of medical interventions. While that's my focus, I'm sure there's additiona important material. We really need to save this data.
yots 7y ago
FYI: The extension offers many private groups that I can't join without approval and that seems to disrupt the flow of the extensions.
- betamaxthetape 7y ago
  Yeah, sorry about that. The current (as of 2100 UTC) set of groups being sent out to be joined were ones submitted through our nomination form: https://tinyurl.com/savegroups https://tinyurl.com/savegroups I did specify that groups requiring approval to join shouldn't be submitted, but not everyone took notice. (And then there was the several dozen Google Groups URLs that were submitted!)
  - misnome 7y ago
    It seems a weird set of groups. Like, lots of three-to-five person groups roleplaying doctor who, spiderman and things like that. Is this the long tail of what hasn't been archived or is there not even a good way to tell post/member count without loading up through the extensions?
    - sodosopa 7y ago
      Yeah not volunteering for that mate.
      - Diagon 7y ago
        See immediately above in the thread. Instructions were perhaps not clear.
    - betamaxthetape 7y ago
      It's a set of groups that have been specifically requested by the fandom community. Of course, the groups handed out depend on what's been joined, so if / once all the fandom groups are joined, we'll move onto something else. I appreciate this isn't made clear in the instructions, but if you have a desired set of groups in mind, you don't need to use the chrome extension. Just join the groups you want saved and (provided you've sent the account details through the form) they'll be added to the queue to be archived. I did a lot of Amateur Radio (Ham Radio in US) groups that way.
tootie 7y ago
Is there any cited reason for the groups they're blocking?
- deleted 7y ago
  [deleted]
- Diagon 7y ago
  Verizon's response, and the response to the response, are in the article of the OP. They claim they offer a Group Downloads Manager, but it's very broken.
ar-jan 7y ago
Just solved a bunch of captchas, but Chrome crashed a few times during. Due to the addon?
- Diagon 7y ago
  I checked on IRC. One person says they've been using it for hours on chromium without a problem. "I've been using Edge (Chromium) for past few hours, no issues. Could be unrelated, could be related. May help to use a standalone chromium build for this."
- withinrafael 7y ago
  I've been using Edge (Chromium) for past few hours, no issues yet. Plugin could be unrelated to your crashing. May help to use a standalone Chromium build for this https://chromium.woolyss.com/ https://chromium.woolyss.com/
ar-jan 7y ago
btw, maybe Mechanical Turk could help with the captcha part?
- vermilingua 7y ago
  I feel like there must be some protection in place against using mTurk with captcha, or it would have already been abused.
  - CriticalCathed 7y ago
    Mturk's turnaround for this stuff can't be fast enough to work would be my guess. I know jobs I put up there for transcription, despite a generous bonus, were always delayed for at the very least hours.
    - abcpassword 7y ago
      You misunderstand. You keep a live page open and point jobs to the live page. No need to put a captcha image in the mturk job. You can absolutely purchase captcha answers.
- adventskalender 7y ago
  A couple of years ago I saw somebody giving a talk, where they demonstrated a CAPCHA-Solving API, with people from India solving the CAPCHAs for a few cents.
  - edm0nd 7y ago
    That's basically what the DeathByCaptcha server is.
    - adventskalender 7y ago
      Thanks - I just wanted to say such services exist or used to exist, didn't remember the name.
jstanley 7y ago
I tried to do this but upon clicking the purple "Join Group" button Yahoo is giving me an error saying my email address is not linked to a Yahoo account: > Your email address is not linked to a Yahoo ID. To join this group, you need to link your email address to a Yahoo account. When I click "link your email address", it just takes me to a page called "Personal info" which doesn't have any obvious way to link my email address. So I'm not sure how to proceed. EDIT: Solved it. I had initially only "verified" the account with a phone number, but you have to add an email address as well. It's now working. For anyone who, like me, signed up for this and filled in the Google form, but then couldn't find the leaderboard URL after closing the tab, it is https://df58.host.cs.st-andrews.ac.uk/yahoogroups/leaderboard https://df58.host.cs.st-andrews.ac.uk/yahoogroups/leaderboar... It seems to be working through a list in reverse alphabetical order. Watching the progress being made is quite satisfying. When I started it was on groups like "sciencefiction" and now it's moved on to "petzluverz".
- hota_mazi 7y ago
  How long did it take you between adding the email address and being able to join the group? Seeing the same thing now, I added an email address and verified it, but I'm still not allowed to join the group.
  - ranger_lennier 7y ago
    It didn't take long at all for me after verification. Although I have sometimes randomly gotten that error message. Interestingly, sometimes it actually had joined the group anyway. The site has been a little glitchy off and on, but it's working for me right now.
wanderer2323 7y ago
It went pretty good for the first 10-20 or so groups but now I get the multiples of the really annoying captchas (click until none remain) per group... Damnit yahoo...
- wanderer2323 7y ago
  update: just enabling the vpn was enough to 'reset' captcha to the simple level, seems like yahoo does not take into account whether your IP is 'residential'.
  - Diagon 7y ago
    I also noted that for yahoo changing IP, even changing continents, allowed me to use the same cookies as long as I kept my original browser window open.
rkagerer 7y ago
Thanks for fighting the good fight! I assumed I could help by going to a web page and solving a bunch of captchas for you, but when I read those instructions I found there's more involved (forging a Yahoo account, installing an extension) and it turned me off. If captcha's are the bottleneck, maybe some generous soul here could figure out a way to automate the rest and just give me a page I can go solve captchas? Further reducing the friction might help get you some more uptick from the community - more monkeys like me banging at typewriters. Sorry I wasn't more help, and best of luck with your efforts.
Angostura 7y ago
Have you posted this on Reddit anywhere? Possibly /technology? You might even get the admins to make an announcement.
- britmob 7y ago
  It’s been all over r/datahoarder lately, also saw a post on r/YouShouldKnow
citruscomputing 7y ago
Shoutout to https://github.com/dessant/buster https://github.com/dessant/buster by the way! `Buster is a browser extension which helps you to solve difficult captchas by completing reCAPTCHA audio challenges using speech recognition. Challenges are solved by clicking on the extension button at the bottom of the reCAPTCHA widget.`
- davidferguson 7y ago
  That's nice, but it doesn't scale. Google only let you solve a few (5 or so) audio captchas in quick succession before you're banned for a while, so it's no good for us.
  - citruscomputing 7y ago
    It's been working for me instead of clicking on all the little busses or crosswalks, even if it doesn't work at scale. Thought it might help some other users of the extension.
dartdartdart 7y ago
As an aside, is there anyway to recover emails if I didn't sign into Yahoo for a year? I and a lot of others had up to 15 years of sentimental mail exchanged during that period :(
- Diagon 7y ago
  I don't see why not. Point Thunderbird at it or something and then just transfer the mails over to somewhere else if you want that - but this is not about mail. Rather it's about Yahoo Groups, whose archives are about to go away.
Aeolun 7y ago
Hah, this is fun! I've so far stumbled on a fantastic group with Sims 1 houses (pictures, and the actual lots), and a Dream Street fan-club, which of course prompted me to see who the hell they were. I confess I'm doing this mostly to see what people posted on the internet at some point in time :) Edit: All groups have around 1600 members... what causes this...
- giancarlostoro 7y ago
  > Edit: All groups have around 1600 members... what causes this... That's possibly the maximum cap?

mehhh 7y ago

Have you considered using NordVPN for CAPTCHA bypass? They are a shady company, but their network of residential VPNs is impressive.

scarejunba 7y ago

I imagine you guys already know this but considering we’re up against the timeline, I’d use the captcha solving service (easy to google yourself) and Luminati to distribute the IP addresses while swallowing my ethical qualms.

StavrosK 7y ago
I would donate my IP/bandwidth to archive.org if I could run a scraper easily.
- ignoranceprior 7y ago
  You can install ArchiveTeam Warrior: https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior https://www.archiveteam.org/index.php?title=ArchiveTeam_Warr...
  - squarefoot 7y ago
    Thanks! I never heard of that before; just like project SETI though for archival purposes. What are the hardware requirements of that VM? I'm attempting to import it on my NAS4Free home NAS Virtualbox service which is the only machine I keep up 24/7 atm, but it takes forever to import. The hardware is very limited however (Atom D410 + a bit over 1GB RAM available), so I'm not sure it would succeed, but so far it loads forever, no errors given. I'd like to run it for this project to start contributing quickly albeit with limited hw before the deadline, then find better iron in the future.
    - icebraining 7y ago
      I don't find it processor or memory heavy, it's mostly doing a lot of IO (network and disk).
    - kalleboo 7y ago
      I'm running it on a Synology NAS (Celeron J3455), and the docker manager UI claims it's using 180 MB RAM and less than 1% CPU (and I just confirmed it's currently working on archiving Yahoo! Groups)
    - MandieD 7y ago
      I’m running the Docker image on the smallest Hetzner VMs, with 5 concurrent groups and 40 shared rsync threads per container, and 12 containers per server. Start one container, do docker top on it to make sure it’s pulling, then start the others one by one, taking a few seconds between each to avoid overwhelming the CPU. I’ve got 6 of those little VMs going, and have rolled up 4GB and 2800 groups worth in 6 hours. After they settle down, they’re more memory than processor intensive. I’ve considered playing with the settings a bit, but thought it was more important to get a bunch of them running on a couple different VMs at different sites. If I were really feeling fancy, I’d write a nice deployment definition for orchestrating this with microk8s...
  - Avamander 7y ago
    Unfortunately it doesn't offer a qemu-compatible image or an image that would work when converted, it's a shame and shooting itself in the foot.

john_moscow 7y ago

Forgive my naivety, but why would blocking of your accounts delete the data you have already backed up? This sounds like you are doing it the wrong WAY, IMO.

Diagon 7y ago
They hadn't backed up yet. They had set up accounts with yahoo that they were then planning to use to back up those groups. Backups themselves were starting, but they had to go slowly enough not to bog down yahoo's servers.
betamaxthetape 7y ago
Two reasons: (a) If we hit Yahoo with everything we've got, groups would have almost certainly crashed, or at least become unbearably slow. That's not a reasonable thing to do, and would be (IMHO) grounds for Verison banning us. (b) We were still testing / writing the scripts to do the actual archiving. Most of the groups we did save before the banning were from test runs of the archiving script. And sure, given hindsight, I'd do things differently. We've learned, now, and are archiving a groups soon after it is joined.
- john_moscow 7y ago
  OK, thanks for explaining this. Just my 2 cents then: big companies make decisions like this based on the potential PR win/loss. If ignoring you keeps the PR delta at 0, while allowing to export the data exposes them to even a minimal risk (I dunno, someone's private details buried in), they will ignore, or even actively resist you. Politically, you need to arrange it so that cooperating with you will give Verizon a small PR boost, while ignoring you will be seen negatively by the public. This thread had a good example of interesting data that is worth preserving, so I would try reaching out to news companies (NY Times and whatnot) to see if anyone wants to publish a piece. Phrasing this positively and ensuring enough people see it, would greatly increase the chances of cooperation from Verizon.
deleted 7y ago
[deleted]

Diagon 7y ago

While the above post is concerned with Fandom groups, my concern is with groups that started doing early community driven biohacking type research. There are medical tests results and discussions of medical interventions. While that's my focus, I'm sure there's additiona important material. We really need to save this data.

yots 7y ago

FYI: The extension offers many private groups that I can't join without approval and that seems to disrupt the flow of the extensions.

betamaxthetape 7y ago
Yeah, sorry about that. The current (as of 2100 UTC) set of groups being sent out to be joined were ones submitted through our nomination form: https://tinyurl.com/savegroups https://tinyurl.com/savegroups I did specify that groups requiring approval to join shouldn't be submitted, but not everyone took notice. (And then there was the several dozen Google Groups URLs that were submitted!)
- misnome 7y ago
  It seems a weird set of groups. Like, lots of three-to-five person groups roleplaying doctor who, spiderman and things like that. Is this the long tail of what hasn't been archived or is there not even a good way to tell post/member count without loading up through the extensions?
  - sodosopa 7y ago
    Yeah not volunteering for that mate.
    - Diagon 7y ago
      See immediately above in the thread. Instructions were perhaps not clear.
  - betamaxthetape 7y ago
    It's a set of groups that have been specifically requested by the fandom community. Of course, the groups handed out depend on what's been joined, so if / once all the fandom groups are joined, we'll move onto something else. I appreciate this isn't made clear in the instructions, but if you have a desired set of groups in mind, you don't need to use the chrome extension. Just join the groups you want saved and (provided you've sent the account details through the form) they'll be added to the queue to be archived. I did a lot of Amateur Radio (Ham Radio in US) groups that way.

tootie 7y ago

Is there any cited reason for the groups they're blocking?

deleted 7y ago
[deleted]
Diagon 7y ago
Verizon's response, and the response to the response, are in the article of the OP. They claim they offer a Group Downloads Manager, but it's very broken.

ar-jan 7y ago

Just solved a bunch of captchas, but Chrome crashed a few times during. Due to the addon?

Diagon 7y ago
I checked on IRC. One person says they've been using it for hours on chromium without a problem. "I've been using Edge (Chromium) for past few hours, no issues. Could be unrelated, could be related. May help to use a standalone chromium build for this."
withinrafael 7y ago
I've been using Edge (Chromium) for past few hours, no issues yet. Plugin could be unrelated to your crashing. May help to use a standalone Chromium build for this https://chromium.woolyss.com/ https://chromium.woolyss.com/

ar-jan 7y ago

btw, maybe Mechanical Turk could help with the captcha part?

vermilingua 7y ago
I feel like there must be some protection in place against using mTurk with captcha, or it would have already been abused.
- CriticalCathed 7y ago
  Mturk's turnaround for this stuff can't be fast enough to work would be my guess. I know jobs I put up there for transcription, despite a generous bonus, were always delayed for at the very least hours.
  - abcpassword 7y ago
    You misunderstand. You keep a live page open and point jobs to the live page. No need to put a captcha image in the mturk job. You can absolutely purchase captcha answers.
adventskalender 7y ago
A couple of years ago I saw somebody giving a talk, where they demonstrated a CAPCHA-Solving API, with people from India solving the CAPCHAs for a few cents.
- edm0nd 7y ago
  That's basically what the DeathByCaptcha server is.
  - adventskalender 7y ago
    Thanks - I just wanted to say such services exist or used to exist, didn't remember the name.

jstanley 7y ago

I tried to do this but upon clicking the purple "Join Group" button Yahoo is giving me an error saying my email address is not linked to a Yahoo account: > Your email address is not linked to a Yahoo ID. To join this group, you need to link your email address to a Yahoo account. When I click "link your email address", it just takes me to a page called "Personal info" which doesn't have any obvious way to link my email address. So I'm not sure how to proceed. EDIT: Solved it. I had initially only "verified" the account with a phone number, but you have to add an email address as well. It's now working. For anyone who, like me, signed up for this and filled in the Google form, but then couldn't find the leaderboard URL after closing the tab, it is https://df58.host.cs.st-andrews.ac.uk/yahoogroups/leaderboard https://df58.host.cs.st-andrews.ac.uk/yahoogroups/leaderboar... It seems to be working through a list in reverse alphabetical order. Watching the progress being made is quite satisfying. When I started it was on groups like "sciencefiction" and now it's moved on to "petzluverz".

hota_mazi 7y ago
How long did it take you between adding the email address and being able to join the group? Seeing the same thing now, I added an email address and verified it, but I'm still not allowed to join the group.
- ranger_lennier 7y ago
  It didn't take long at all for me after verification. Although I have sometimes randomly gotten that error message. Interestingly, sometimes it actually had joined the group anyway. The site has been a little glitchy off and on, but it's working for me right now.

wanderer2323 7y ago

It went pretty good for the first 10-20 or so groups but now I get the multiples of the really annoying captchas (click until none remain) per group... Damnit yahoo...

wanderer2323 7y ago
update: just enabling the vpn was enough to 'reset' captcha to the simple level, seems like yahoo does not take into account whether your IP is 'residential'.
- Diagon 7y ago
  I also noted that for yahoo changing IP, even changing continents, allowed me to use the same cookies as long as I kept my original browser window open.

rkagerer 7y ago

Thanks for fighting the good fight! I assumed I could help by going to a web page and solving a bunch of captchas for you, but when I read those instructions I found there's more involved (forging a Yahoo account, installing an extension) and it turned me off. If captcha's are the bottleneck, maybe some generous soul here could figure out a way to automate the rest and just give me a page I can go solve captchas? Further reducing the friction might help get you some more uptick from the community - more monkeys like me banging at typewriters. Sorry I wasn't more help, and best of luck with your efforts.

Angostura 7y ago

Have you posted this on Reddit anywhere? Possibly /technology? You might even get the admins to make an announcement.

britmob 7y ago
It’s been all over r/datahoarder lately, also saw a post on r/YouShouldKnow

citruscomputing 7y ago

Shoutout to https://github.com/dessant/buster https://github.com/dessant/buster by the way! `Buster is a browser extension which helps you to solve difficult captchas by completing reCAPTCHA audio challenges using speech recognition. Challenges are solved by clicking on the extension button at the bottom of the reCAPTCHA widget.`

davidferguson 7y ago
That's nice, but it doesn't scale. Google only let you solve a few (5 or so) audio captchas in quick succession before you're banned for a while, so it's no good for us.
- citruscomputing 7y ago
  It's been working for me instead of clicking on all the little busses or crosswalks, even if it doesn't work at scale. Thought it might help some other users of the extension.

dartdartdart 7y ago

As an aside, is there anyway to recover emails if I didn't sign into Yahoo for a year? I and a lot of others had up to 15 years of sentimental mail exchanged during that period :(

Diagon 7y ago
I don't see why not. Point Thunderbird at it or something and then just transfer the mails over to somewhere else if you want that - but this is not about mail. Rather it's about Yahoo Groups, whose archives are about to go away.

Aeolun 7y ago

Hah, this is fun! I've so far stumbled on a fantastic group with Sims 1 houses (pictures, and the actual lots), and a Dream Street fan-club, which of course prompted me to see who the hell they were. I confess I'm doing this mostly to see what people posted on the internet at some point in time :) Edit: All groups have around 1600 members... what causes this...

giancarlostoro 7y ago
> Edit: All groups have around 1600 members... what causes this... That's possibly the maximum cap?