5 ms·

I'm surprised IA relies on outside, potentially ephemeral services at all. I thought all of their material was self-hosted. Had that actually been true, it seem

I'm surprised IA relies on outside, potentially ephemeral services at all. I thought all of their material was self-hosted. Had that actually been true, it seems like IA wouldn't have needed to delete its account over an issue of whether their usage fit the intended use case for Flickr.

rolph 5y ago
this is why archives should be independently duplicated, to head off a massacre. what isnt clear to me is the survival of the content, was there a data migration or was it all just nerfed?
- johtso 5y ago
  All the images still exist on the internet archive, just not in a format that can be explored. What's been lost is the efforts of all the users to separate the interesting images from the chaff.. and organise them into collections. For such a large corpus that's a really valuable thing. Also links people had to those images are just dead. And all the images users had previously favourited have just disappeared from their collections.
  - rolph 5y ago
    what a task it would be, to reconstruct the relations and cobble up a db from user data.
- ineedasername 5y ago
  >this is why archives should be independently duplicated It's an interesting challenge with IA because they have > 100 petabytes of content, many thousands of $ in hard drives alone. Too much for random individuals to decide they want to host their own clone & become a mirror. So, we're be looking at large institutions or companies to do this philanthropically. Or, here's the tech challenge, some distributed mechanism where participants all host a portion, with enough redundancy that a large % could go offline and still keep integrity.
johtso 5y ago
The problem is, the internet archive is great at storing data, not not the best at making that data discoverable, viewable. A partnership like this makes quite a bit of sense.
- ineedasername 5y ago
  It seems like there'd be a lot of volunteers to help with building their own. On that scale, I'm not sure if/which FOSS option might be suited. But the sheer technical challenge of scaling a system to > 100 petabytes along with a compelling mission would attract a lot of talent. Heck, if they'd just worked with Flickr long enough to extract the metadata created surrounding these images so that all that effort wouldn't be lost then I bet there would be ample volunteers just to reimplement a way of utilizing it. To discard all that work... I can only.hope that no actual data is lost and they will be able to do something with it in the future.
  - exolymph 5y ago
    Uncoordinated volunteers are worse than useless, sadly.
    - ineedasername 5y ago
      Why would it be uncoordinated? Volunteers work for FOSS after all, and if the work for IA was also under a FOSS license I don't see why it couldn't work. And IA is high profile enough to attract really talented people from the FOSS community that already understand how to work under that paradigm.
  - db48x 5y ago
    You can already do this without building a huge storage system or needing any sort of permission from the Internet Archive. Anyone can upload material to IA, and anything on IA can be embedded into other webpages. You can dream up the ideal user interface and build it without needing any permission or cooperation from IA.
    - 5y ago
stjohnswarts 5y ago
We don't know this was the doing of Flickr though, I mean it seems like there could have been an intermediary step where IA could download all the comments and such? I mean nothing was stated at all, and that's a bit scary that the IA would just not say anything and allow all that to be deleted nonchalantly.
- ineedasername 5y ago
  That's what I'm hoping. Although I doubt that IA, on it's own, would suddenly go to Flickr & say "Hey our use case doesn't really fit what you intended, let's figure out how to go our separate ways." My guess is that the conversation was initiated by Flickr. Whether IA agreed with them from the beginning or was forced to "agree" probably doesn't matter, especially because I can see the merit of the argument that Flickr may not be an appropriate platform. I'm just hoping things were save in a way that they'll be able to implement a replacement at some point.

rolph 5y ago

this is why archives should be independently duplicated, to head off a massacre. what isnt clear to me is the survival of the content, was there a data migration or was it all just nerfed?

johtso 5y ago
All the images still exist on the internet archive, just not in a format that can be explored. What's been lost is the efforts of all the users to separate the interesting images from the chaff.. and organise them into collections. For such a large corpus that's a really valuable thing. Also links people had to those images are just dead. And all the images users had previously favourited have just disappeared from their collections.
- rolph 5y ago
  what a task it would be, to reconstruct the relations and cobble up a db from user data.
ineedasername 5y ago
>this is why archives should be independently duplicated It's an interesting challenge with IA because they have > 100 petabytes of content, many thousands of $ in hard drives alone. Too much for random individuals to decide they want to host their own clone & become a mirror. So, we're be looking at large institutions or companies to do this philanthropically. Or, here's the tech challenge, some distributed mechanism where participants all host a portion, with enough redundancy that a large % could go offline and still keep integrity.

johtso 5y ago

The problem is, the internet archive is great at storing data, not not the best at making that data discoverable, viewable. A partnership like this makes quite a bit of sense.

ineedasername 5y ago
It seems like there'd be a lot of volunteers to help with building their own. On that scale, I'm not sure if/which FOSS option might be suited. But the sheer technical challenge of scaling a system to > 100 petabytes along with a compelling mission would attract a lot of talent. Heck, if they'd just worked with Flickr long enough to extract the metadata created surrounding these images so that all that effort wouldn't be lost then I bet there would be ample volunteers just to reimplement a way of utilizing it. To discard all that work... I can only.hope that no actual data is lost and they will be able to do something with it in the future.
- exolymph 5y ago
  Uncoordinated volunteers are worse than useless, sadly.
  - ineedasername 5y ago
    Why would it be uncoordinated? Volunteers work for FOSS after all, and if the work for IA was also under a FOSS license I don't see why it couldn't work. And IA is high profile enough to attract really talented people from the FOSS community that already understand how to work under that paradigm.
- db48x 5y ago
  You can already do this without building a huge storage system or needing any sort of permission from the Internet Archive. Anyone can upload material to IA, and anything on IA can be embedded into other webpages. You can dream up the ideal user interface and build it without needing any permission or cooperation from IA.
  - 5y ago

stjohnswarts 5y ago

We don't know this was the doing of Flickr though, I mean it seems like there could have been an intermediary step where IA could download all the comments and such? I mean nothing was stated at all, and that's a bit scary that the IA would just not say anything and allow all that to be deleted nonchalantly.

ineedasername 5y ago
That's what I'm hoping. Although I doubt that IA, on it's own, would suddenly go to Flickr & say "Hey our use case doesn't really fit what you intended, let's figure out how to go our separate ways." My guess is that the conversation was initiated by Flickr. Whether IA agreed with them from the beginning or was forced to "agree" probably doesn't matter, especially because I can see the merit of the argument that Flickr may not be an appropriate platform. I'm just hoping things were save in a way that they'll be able to implement a replacement at some point.