4 ms·

Maybe they should switch to Github. https://github.com/features/packages https://github.com/features/packages

by hvs 6y ago

toomuchtodo 6y ago
Or store the containers in the Internet Archive alongside the paper. They’re just tarballs. Lots of options as long as you're comfortable with object storage.
- captn3m0 6y ago
  quay is another alternative.
- brutos 6y ago
  This still means that tools published in the last few years until now might just be gone soon. The people who uploaded the images might have graduated or moved on and none will be there to save the work.
  - icebraining 6y ago
    Sounds like a job for the Archive Team, as long as there's some way to identify the images worth saving.
    - maxfan8 6y ago
      Yep, just mentioned it to the Archive Team IRC. We're probably going to selectively archive particular Docker images, although that's a lot of manual labor. If you have any ideas wrt to selecting important images, that'd be great.
      - thebouv 6y ago
        Rough idea: maintain an Awesome List of images worth saving, take submissions from public, use that list to automate what to pull?
        maxfan8 6y ago
        Yeah, good idea — I’m not in these fields so it’s difficult for me to judge. Also, it sounds like we should be prioritizing niche images that only a handful of papers use rather than images that people rely upon regularly.
        cosmie 6y ago
        Couldn't you bootstrap a list by searching/parsing the Archive dataset itself? Searching for A) "docker pull" commands and parsing the text that comes after it based on the command's syntax[1] to extract instructional references to images such as "docker pull ubuntu:latest, and B) Searching for links/text beginning with "https://hub.docker.com/_/" https://hub.docker.com/_/" to identify informational references to image base pages such as (https://hub.docker.com/_/ubuntu https://hub.docker.com/_/ubuntu) [1] https://docs.docker.com/engine/reference/commandline/pull/ https://docs.docker.com/engine/reference/commandline/pull/
        maxfan8 6y ago
        Good idea! The base images are probably not in danger of being deleted though. The other issue is that (to my knowledge) the amount of papers on IA isn't terribly impressive. I think maybe indexing and going through SciHub will be better since some of these fields slap paywalls in front of their papers. However, that's a pretty large task as well. The other thing is that papers rarely say "to reproduce my work do . . .". Usually the best we've got is a link to a GitHub repo (if that). I'm not sure how effective that strategy will be since it's guaranteed to be an under-count of the docker images we'd need to archive. Perhaps in conjunction with archiving all images that fall under particular search queries, we'd get the best of both worlds. I've you've got ideas, feel free to hop onto efnet (#archiveteam and #archiveteam-bs) (also on hackint) to share your thoughts.
      - contravariant 6y ago
        Since images tend to be based on each other I wonder if someone's analyzed the corresponding dependency graph yet. In theory you should get quite far if you isolate the most commonly used base images.
        CameronNemo 6y ago
        Are those not the images that are basically guaranteed to stay in Dockerhub?
        toomuchtodo 6y ago
        “Guaranteed” is a strong word.
vegannet 6y ago
GitHub storage for docker images is very expensive relative to free: I don’t think it’s a viable solution in this case.
lstamour 6y ago
Publishing containers to GitHub might be free but you have to login to GitHub to download the containers from free accounts, significantly hampering end-user usability compared to Docker Hub, particularly if 2FA authentication is enabled on a GitHub account. As mentioned elsewhere Quay.io might be another alternative.
- qppo 6y ago
  You don't need to register an SSH key to download a public repo I thought
  - timdorr 6y ago
    Not an SSH key, but you do need an access token: > You need an access token to publish, install, and delete packages in GitHub Packages. https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages#authenticating-to-github-packages https://docs.github.com/en/packages/using-github-packages-wi...
    - qppo 6y ago
      ...but not to download. You can clone a repo and download release artifacts without a PAT. That's only necessary for interacting with the API for actions that need authentication, which would be anything involving mutating a repository.
      - akerl_ 6y ago
        Using the GitHub Docker Registry requires auth, even just for downloads. https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages https://docs.github.com/en/packages/using-github-packages-wi... GitHub Packages is different from GitHub Releases (and their artifacts) or cloning repos.
      - mynameisvlad 6y ago
        > You need an access token to publish, install, and delete packages in GitHub Packages. Yes, you do.

Maybe they should switch to Github. https://github.com/features/packages https://github.com/features/packages

by hvs 6y ago

Maybe they should switch to Github. https://github.com/features/packages https://github.com/features/packages

Or store the containers in the Internet Archive alongside the paper. They’re just tarballs. Lots of options as long as you're comfortable with object storage.

captn3m0 6y ago
quay is another alternative.
brutos 6y ago
This still means that tools published in the last few years until now might just be gone soon. The people who uploaded the images might have graduated or moved on and none will be there to save the work.
- icebraining 6y ago
  Sounds like a job for the Archive Team, as long as there's some way to identify the images worth saving.
  - maxfan8 6y ago
    Yep, just mentioned it to the Archive Team IRC. We're probably going to selectively archive particular Docker images, although that's a lot of manual labor. If you have any ideas wrt to selecting important images, that'd be great.
    - thebouv 6y ago
      Rough idea: maintain an Awesome List of images worth saving, take submissions from public, use that list to automate what to pull?
      - maxfan8 6y ago
        Yeah, good idea — I’m not in these fields so it’s difficult for me to judge. Also, it sounds like we should be prioritizing niche images that only a handful of papers use rather than images that people rely upon regularly.
        cosmie 6y ago
        Couldn't you bootstrap a list by searching/parsing the Archive dataset itself? Searching for A) "docker pull" commands and parsing the text that comes after it based on the command's syntax[1] to extract instructional references to images such as "docker pull ubuntu:latest, and B) Searching for links/text beginning with "https://hub.docker.com/_/" https://hub.docker.com/_/" to identify informational references to image base pages such as (https://hub.docker.com/_/ubuntu https://hub.docker.com/_/ubuntu) [1] https://docs.docker.com/engine/reference/commandline/pull/ https://docs.docker.com/engine/reference/commandline/pull/
        maxfan8 6y ago
        Good idea! The base images are probably not in danger of being deleted though. The other issue is that (to my knowledge) the amount of papers on IA isn't terribly impressive. I think maybe indexing and going through SciHub will be better since some of these fields slap paywalls in front of their papers. However, that's a pretty large task as well. The other thing is that papers rarely say "to reproduce my work do . . .". Usually the best we've got is a link to a GitHub repo (if that). I'm not sure how effective that strategy will be since it's guaranteed to be an under-count of the docker images we'd need to archive. Perhaps in conjunction with archiving all images that fall under particular search queries, we'd get the best of both worlds. I've you've got ideas, feel free to hop onto efnet (#archiveteam and #archiveteam-bs) (also on hackint) to share your thoughts.
    - contravariant 6y ago
      Since images tend to be based on each other I wonder if someone's analyzed the corresponding dependency graph yet. In theory you should get quite far if you isolate the most commonly used base images.
      - CameronNemo 6y ago
        Are those not the images that are basically guaranteed to stay in Dockerhub?
        toomuchtodo 6y ago
        “Guaranteed” is a strong word.

vegannet 6y ago

GitHub storage for docker images is very expensive relative to free: I don’t think it’s a viable solution in this case.

lstamour 6y ago

Publishing containers to GitHub might be free but you have to login to GitHub to download the containers from free accounts, significantly hampering end-user usability compared to Docker Hub, particularly if 2FA authentication is enabled on a GitHub account. As mentioned elsewhere Quay.io might be another alternative.

qppo 6y ago
You don't need to register an SSH key to download a public repo I thought
- timdorr 6y ago
  Not an SSH key, but you do need an access token: > You need an access token to publish, install, and delete packages in GitHub Packages. https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages#authenticating-to-github-packages https://docs.github.com/en/packages/using-github-packages-wi...
  - qppo 6y ago
    ...but not to download. You can clone a repo and download release artifacts without a PAT. That's only necessary for interacting with the API for actions that need authentication, which would be anything involving mutating a repository.
    - akerl_ 6y ago
      Using the GitHub Docker Registry requires auth, even just for downloads. https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages https://docs.github.com/en/packages/using-github-packages-wi... GitHub Packages is different from GitHub Releases (and their artifacts) or cloning repos.
    - mynameisvlad 6y ago
      > You need an access token to publish, install, and delete packages in GitHub Packages. Yes, you do.