5 ms·

They were downloading at 50 Gbps for a while https://twitter.com/donk_enby/status/1348497204940595201 https://twitter.com/donk_enby/status/1348497204940595201

They were downloading at 50 Gbps for a while https://twitter.com/donk_enby/status/1348497204940595201 https://twitter.com/donk_enby/status/1348497204940595201 https://twitter.com/donk_enby/status/1348440720504401921 https://twitter.com/donk_enby/status/1348440720504401921 Also, auth provider (twilio?) removed Parler as a client so for a short while it was possible to create accounts without a phone number (2FA). https://twitter.com/donk_enby/status/1348298836930867204 https://twitter.com/donk_enby/status/1348298836930867204 edit: okta, free trial, thanks: https://news.ycombinator.com/item?id=25774943 https://news.ycombinator.com/item?id=25774943

alborzb 6y ago
Just a note here, it wasn't Twilio. It was a free trial of Okta that they were using for their entire userbase. https://twitter.com/okta/status/1348191370528256002?ref_src=twsrc%5Etfw https://twitter.com/okta/status/1348191370528256002?ref_src=...
- chefkoch 6y ago
  How cheap can you be to run your whole site on a trial and fail open if it doesn't work anymore?
  - skrtskrt 6y ago
    I read that they chose to fail open when Okta dropped them in order to stay online for awhile longer.
secondcoming 6y ago
She is probably liable for that data egress bill. Also, how does some randomer have ten of terabytes of disk lying around?
- Nextgrid 6y ago
  Maybe the data compresses well and/or a lot of it is redundant (so instead of storing it raw you store it in a database and use relationships to link related pieces of data)?
  - johnchristopher 6y ago
    That's not how you usually measure a database size.
- asiando 6y ago
  Given the speed I’m guessing it was towards S3 so there’s plenty of terabytes.
  - secondcoming 6y ago
    Yes probably. Wherever it is, that provider is in an interesting legal position by holding this data. Also, I assume there is content from EU citizens in there, and so GDPR violations galore.
    - roywashere 6y ago
      So people post it on Parlor themselves, it is publicly available, but it would be illegal to download for me? That does not make much sense.
      - secondcoming 6y ago
        My understanding is that not all of it was public posts. As an EU citizen, I can request that a company deletes my data. Unless this data dump is being treated as a crime scene or something, then the holders of this data will need some way to comply with these requests. Also, now AWS (or whatever cloud provider it is) is holding content that contains racist and/or illegal content. Are they not effectively now just another Parler?
        sedev 6y ago
        No, they're not. Holding a static, private archive is not the same thing as hosting a live, public site.
        secondcoming 6y ago
        How so? Aren't people freely perusing all this data, extracting GPS metadata and reading both public and private conversations?
        ilovefood 6y ago
- gerikson 6y ago
  Wasn't this an ArchiveTeam Warrior project? https://archiveteam.org/index.php?title=ArchiveTeam_Warrior https://archiveteam.org/index.php?title=ArchiveTeam_Warrior If so it's distributed among many volunteers. But the data still has to end up somewhere... Archive.org?
  - philshem 6y ago
    the data is being processed by archive team https://www.archiveteam.org/index.php?title=Parler https://www.archiveteam.org/index.php?title=Parler Grab status: https://tracker.archiveteam.org/parler/ https://tracker.archiveteam.org/parler/ and will be hosted by archive.org
- ClumsyPilot 6y ago
  Have you ever been to r/Datahoarder or r/Homelab? 10TB fits on one desktop drive, it's completely pedestrian
  - secondcoming 6y ago
    I have not. I have one 4TB HD in my machine and I've never come close to filling it!
    - ghaff 6y ago
      It's really a matter of multimedia: video, raw format photos, music, ripped movies. That's the vast bulk of the probably 8TB or so I have.
    - dijit 6y ago
      Many people have <1TiB on their machines and are content with it. Others, like me, have home NAS's which have 10-20TiB, and they're usually close to full. Fewer people, but a non-zero amount, hoard data, they have 20-100TiB or even more, full on homelabs. If you're a person collecting datadumps or running rainbow tables, you probably have such a unit. It's not even that expensive really, you can get a 24TiB Pegasus Thunderbolt raid array for <1,000 USD
    - mkr-hn 6y ago
      The raw files off my D5600 are roughly 30MB each. If I took 100 photos a day, I would need +1TB every year to keep all of them. The battery is good for about 600 shots. Someone who does photography professionally can easily blow right past that on one job before accounting for their second shooter's files.
- DanBC 6y ago
  > some randomer They're not a randomer, they're a person interested in data dumps.
  - ersii 6y ago
    donk_enby wants to be called she/her according to Twitter, unless you're speaking of someone else
- blaser-waffle 6y ago
  I'm not a huge data fiend but I've got maybe ~3-4 2TB drives in my house: a Synology NAS + a spare drive. Including old desktops and a couple of random external HDDs, I could probably hit 8-10TB easily. And it's not like it's hard to get more. If I hit a montherlode and I need keep it, it's a 20 minute drive to Target / Best Buy / Walmart for a drive or three. Not as cheap as bulk orders off of Newegg but cost-effective enough to store these dumps.

alborzb 6y ago

Just a note here, it wasn't Twilio. It was a free trial of Okta that they were using for their entire userbase. https://twitter.com/okta/status/1348191370528256002?ref_src=twsrc%5Etfw https://twitter.com/okta/status/1348191370528256002?ref_src=...

chefkoch 6y ago
How cheap can you be to run your whole site on a trial and fail open if it doesn't work anymore?
- skrtskrt 6y ago
  I read that they chose to fail open when Okta dropped them in order to stay online for awhile longer.

secondcoming 6y ago

She is probably liable for that data egress bill. Also, how does some randomer have ten of terabytes of disk lying around?

Nextgrid 6y ago
Maybe the data compresses well and/or a lot of it is redundant (so instead of storing it raw you store it in a database and use relationships to link related pieces of data)?
- johnchristopher 6y ago
  That's not how you usually measure a database size.
asiando 6y ago
Given the speed I’m guessing it was towards S3 so there’s plenty of terabytes.
- secondcoming 6y ago
  Yes probably. Wherever it is, that provider is in an interesting legal position by holding this data. Also, I assume there is content from EU citizens in there, and so GDPR violations galore.
  - roywashere 6y ago
    So people post it on Parlor themselves, it is publicly available, but it would be illegal to download for me? That does not make much sense.
    - secondcoming 6y ago
      My understanding is that not all of it was public posts. As an EU citizen, I can request that a company deletes my data. Unless this data dump is being treated as a crime scene or something, then the holders of this data will need some way to comply with these requests. Also, now AWS (or whatever cloud provider it is) is holding content that contains racist and/or illegal content. Are they not effectively now just another Parler?
      - sedev 6y ago
        No, they're not. Holding a static, private archive is not the same thing as hosting a live, public site.
        secondcoming 6y ago
        How so? Aren't people freely perusing all this data, extracting GPS metadata and reading both public and private conversations?
        ilovefood 6y ago
gerikson 6y ago
Wasn't this an ArchiveTeam Warrior project? https://archiveteam.org/index.php?title=ArchiveTeam_Warrior https://archiveteam.org/index.php?title=ArchiveTeam_Warrior If so it's distributed among many volunteers. But the data still has to end up somewhere... Archive.org?
- philshem 6y ago
  the data is being processed by archive team https://www.archiveteam.org/index.php?title=Parler https://www.archiveteam.org/index.php?title=Parler Grab status: https://tracker.archiveteam.org/parler/ https://tracker.archiveteam.org/parler/ and will be hosted by archive.org
ClumsyPilot 6y ago
Have you ever been to r/Datahoarder or r/Homelab? 10TB fits on one desktop drive, it's completely pedestrian
- secondcoming 6y ago
  I have not. I have one 4TB HD in my machine and I've never come close to filling it!
  - ghaff 6y ago
    It's really a matter of multimedia: video, raw format photos, music, ripped movies. That's the vast bulk of the probably 8TB or so I have.
  - dijit 6y ago
    Many people have <1TiB on their machines and are content with it. Others, like me, have home NAS's which have 10-20TiB, and they're usually close to full. Fewer people, but a non-zero amount, hoard data, they have 20-100TiB or even more, full on homelabs. If you're a person collecting datadumps or running rainbow tables, you probably have such a unit. It's not even that expensive really, you can get a 24TiB Pegasus Thunderbolt raid array for <1,000 USD
  - mkr-hn 6y ago
    The raw files off my D5600 are roughly 30MB each. If I took 100 photos a day, I would need +1TB every year to keep all of them. The battery is good for about 600 shots. Someone who does photography professionally can easily blow right past that on one job before accounting for their second shooter's files.
DanBC 6y ago
> some randomer They're not a randomer, they're a person interested in data dumps.
- ersii 6y ago
  donk_enby wants to be called she/her according to Twitter, unless you're speaking of someone else
blaser-waffle 6y ago
I'm not a huge data fiend but I've got maybe ~3-4 2TB drives in my house: a Synology NAS + a spare drive. Including old desktops and a couple of random external HDDs, I could probably hit 8-10TB easily. And it's not like it's hard to get more. If I hit a montherlode and I need keep it, it's a 20 minute drive to Target / Best Buy / Walmart for a drive or three. Not as cheap as bulk orders off of Newegg but cost-effective enough to store these dumps.