6 ms·

Torrent?

toomuchtodo 7y ago
I asked for a data dump via email, waiting to hear back. Ripping it all out is going to cost someone a pretty penny since it's hosted in AWS, better to ask someone to fill and ship some SATA drives. Then off to the Internet Archive!
- zorm 7y ago
  I saw a companion AWS blog on this: https://aws.amazon.com/blogs/publicsector/smithsonian-3-million-images-smithsonian-open-access-initiative/ https://aws.amazon.com/blogs/publicsector/smithsonian-3-mill... Looks like its part of the public dataset program so you can probably just ask for the bucket name and get full free access to everything.
  - toomuchtodo 7y ago
    > just ask for the bucket name and get full free access to everything Jed or jeffbarr, what say ye?
    - toomuchtodo 7y ago
      Disregard! Pete’s comment is sufficient for bucket enumeration.
  - Reelin 7y ago
    This is weird. The Smithsonian page [1] says "Data hosting provided by AWS Public Dataset Program", and there's the Amazon blog post you linked, but the data set seems to be missing from the registry Amazon publishes. [2] I guess no one submitted a pull request yet? [3] [1] https://www.si.edu/OpenAccess https://www.si.edu/OpenAccess [2] https://registry.opendata.aws https://registry.opendata.aws [3] https://github.com/awslabs/open-data-registry https://github.com/awslabs/open-data-registry
  - peatmoss 7y ago
    I can confirm that this dataset is indeed part of the AWS Public Dataset Program. We were finalizing some details, but this dataset is now listed in the Registry of Open Data: https://registry.opendata.aws/smithsonian-open-access/ https://registry.opendata.aws/smithsonian-open-access/ Enjoy! I’m personally very excited about this dataset, and couldn’t be more impressed with the people, mission, and, well, everything at the Smithsonian. Source: I work on the AWS Open Data team / had a small role in this (normal lawyerspeak caveats apply that my views and opinions are my own)
    - toomuchtodo 7y ago
      Thank you!!
- ChrisArchitect 7y ago
  How much data do you think it encompasses? Approaching/surpassing Petabytes?
  - toomuchtodo 7y ago
    Couldn't even hazard a guess, I can send a Backblaze-like chassis if needed, but if there's no cost to pull from the AWS public S3 bucket, I'll just spin up some VMs and enumerate everything into items in the Internet Archive. Trying to find the balance between time, cost, and inconvenience for all parties involved.
    - ChrisArchitect 7y ago
      still wondering/trying to get a sense of how much data has been accumulated in the open data digitization process. Gleaning that it might be multiple petabytes based on the open data sessions that were going on last few years but dunno...
      - ChrisArchitect 7y ago
        Some of the ppl interested in the data and a lead at AWS got back to me on it: aws query on the open dataset comes back with: Total Objects: 4649789 Total Size: 312.5 TiB!

Torrent?

I asked for a data dump via email, waiting to hear back. Ripping it all out is going to cost someone a pretty penny since it's hosted in AWS, better to ask someone to fill and ship some SATA drives. Then off to the Internet Archive!

zorm 7y ago
I saw a companion AWS blog on this: https://aws.amazon.com/blogs/publicsector/smithsonian-3-million-images-smithsonian-open-access-initiative/ https://aws.amazon.com/blogs/publicsector/smithsonian-3-mill... Looks like its part of the public dataset program so you can probably just ask for the bucket name and get full free access to everything.
- toomuchtodo 7y ago
  > just ask for the bucket name and get full free access to everything Jed or jeffbarr, what say ye?
  - toomuchtodo 7y ago
    Disregard! Pete’s comment is sufficient for bucket enumeration.
- Reelin 7y ago
  This is weird. The Smithsonian page [1] says "Data hosting provided by AWS Public Dataset Program", and there's the Amazon blog post you linked, but the data set seems to be missing from the registry Amazon publishes. [2] I guess no one submitted a pull request yet? [3] [1] https://www.si.edu/OpenAccess https://www.si.edu/OpenAccess [2] https://registry.opendata.aws https://registry.opendata.aws [3] https://github.com/awslabs/open-data-registry https://github.com/awslabs/open-data-registry
- peatmoss 7y ago
  I can confirm that this dataset is indeed part of the AWS Public Dataset Program. We were finalizing some details, but this dataset is now listed in the Registry of Open Data: https://registry.opendata.aws/smithsonian-open-access/ https://registry.opendata.aws/smithsonian-open-access/ Enjoy! I’m personally very excited about this dataset, and couldn’t be more impressed with the people, mission, and, well, everything at the Smithsonian. Source: I work on the AWS Open Data team / had a small role in this (normal lawyerspeak caveats apply that my views and opinions are my own)
  - toomuchtodo 7y ago
    Thank you!!
ChrisArchitect 7y ago
How much data do you think it encompasses? Approaching/surpassing Petabytes?
- toomuchtodo 7y ago
  Couldn't even hazard a guess, I can send a Backblaze-like chassis if needed, but if there's no cost to pull from the AWS public S3 bucket, I'll just spin up some VMs and enumerate everything into items in the Internet Archive. Trying to find the balance between time, cost, and inconvenience for all parties involved.
  - ChrisArchitect 7y ago
    still wondering/trying to get a sense of how much data has been accumulated in the open data digitization process. Gleaning that it might be multiple petabytes based on the open data sessions that were going on last few years but dunno...
    - ChrisArchitect 7y ago
      Some of the ppl interested in the data and a lead at AWS got back to me on it: aws query on the open dataset comes back with: Total Objects: 4649789 Total Size: 312.5 TiB!