4 ms·

Slightly tangential question, but this is yet another piece of work published on Distill. I keep a library of the papers I've read in Zotero. However, when it c

by Reelin 8y ago

Slightly tangential question, but this is yet another piece of work published on Distill. I keep a library of the papers I've read in Zotero. However, when it comes to Distill I've run into a problem - I can't figure out a robust method of archival. Regular journal articles are published as PDF files, which make a decent electronic analog of physical paper. In the case of Distill, there's the usual DOI, volume, and issue, but the page itself isn't static. I did find a seemingly undocumented feature referenced in a GitHub issue - you can append "index.archive.html" to the url to get a supposedly standalone html archive of the page. However, when I tried this with a number of recent publications there were missing images and broken formatting all over the place, seemingly due to the authors having linked in external resources (primarily for a number of the figures). Does anyone know how to do this or have any ideas?

phreeza 8y ago
Haven't tried it on distil, but maybe this works? https://github.com/jgm/standalone-html https://github.com/jgm/standalone-html
- Reelin 8y ago
  Looks interesting, I'll try it out when I get a chance. I suspect it functions in a similar manner to Zotero's built in archiver though, so I'm not optimistic about what the results will look like.
RustingSword 8y ago
I use Zotero with [Zotero Connector](https://www.zotero.org/download/connectors https://www.zotero.org/download/connectors) plugin for Firefox. When saving a webpage to Zotero, it will also save a snapshot of that webpage (a configurable option in Zotero preferences). I tried it with latest two Distill posts, and didn't see any missing images. Actually for those interactive images, there will be a static version and an interactive version of that image in the snapshot, which is acceptable for me. Another problem more annoying is that those inline citations will show up even if you don't hover over them, thus covering underlying texts.
- Reelin 8y ago
  Forgot to mention originally, but I did try Zotero's snapshot feature. It seems to work better for some papers than others - the activation atlases one (https://distill.pub/2019/activation-atlas/ https://distill.pub/2019/activation-atlas/) is particularly bad. As you note, the hoverable citations and notes are also a problem. It's not that I have any pressing need for offline copies at this precise instant, but they have proven useful on a number of occasions.
clickok 8y ago
It depends how deep you want to go into the web archiving rabbit-hole, but using something like `archivenow`[0] would probably be the local optimum in terms of convenience, robustness, and permanence. It's "a tool to push web resources into web archives", which basically entails making requests of archiving services (like the Wayback machine, for example). Using `archivenow --all <url>`, you can back up <url> to a whole bunch of different services, and get links to snapshots of that page at the time of access. This is mostly sufficient, so long as at least one service is trustworthy and doesn't shut down unexpectedly. One issue that might come up is that the archives might miss content if the user has to interact with the page a bunch in order to reveal it, but that's unlikely to be an issue for the sorts of things an academic would be archiving. As another poster mentioned, some bibliography managers have this capability baked in, or can acquire it with a plugin. However, if you want more control, you might consider using something like `pywb`[1] to make your own web archive, and then host it yourself (e.g., on S3 or something). For more robustness, you could also publish those WARC files as torrents, or on IPFS, or if you're really cool, as part of the transaction messages on some blockchain (provided that you can stomach the transaction fees and don't anticipate having a large audience in China). The above will probably suffice so long as the Internet continues to function as a going concern. To accommodate concerns about what happens in the case of total societal collapse, I've been backing up the papers that I cite by converting to binary[2] and preserving the result via aluminum punchcards, which I then mail to Sam Altman since he's got a way better game plan for the end times than I do[3]. I'm not sure if aluminum is optimal for this purpose though, both for material science reasons and because I've been informed that I am under investigation for my role in some sort of aluminum smuggling ring. If that fails, my last backup plan entails constructing an allegorical representation of the papers in question (which is surprisingly easy-- the Hero's Journey from Joseph Campbell[4]'s exegesis is basically about differential equations anyways), which I will then craft into a compelling mythology and teach to the local youths. Hopefully this will ensure that the citations are preserved via oral tradition even in the wake of cataclysmic change. Figuring out the BibTeX for this archival format will be pretty tricky, though. But yeah, unless you have some extra aluminum lying around, `archivenow` or whatever's built-in to your reference manager should do the trick. --- 0. github.com/oduwsdl/archivenow 1. github.com/webrecorder/pywb 2. After stripping most of the markup removing images using `pandoc`, as aluminum is expensive. 3. https://www.bloomberg.com/features/2018-rich-new-zealand-doomsday-preppers/ https://www.bloomberg.com/features/2018-rich-new-zealand-doo... 4. https://en.wikipedia.org/wiki/Joseph_Campbell https://en.wikipedia.org/wiki/Joseph_Campbell
ludwigschubert 8y ago
Hi! I help maintain Distill's infrastructure. We'd love to find a robust solution for archival, too, but we haven't been able to think of a satisfying automatic way of providing static fallbacks to interactive diagrams. (That's why index.archive.html isn't a public feature ^_^) If you have an idea of what we could do on our side with a fixed amount of work, please let me know! Unfortunately we can't require our third-party authors to be experts in graceful degradation. I'm sure better tooling could help, but currently have no specific inspiration. As a fallback for your specific use case—we do actually provide a tiny bit of print styles in our CSS. Try your browser's PDF conversion/print feature; I find it often produces acceptable results. (Minus proper alignment to pages, of course. :/)
- Reelin 8y ago
  Really glad to hear you guys are aware of this! I do love the expressiveness and ease of understanding that interactive diagrams offer, but find the prospect of a mutable scientific literature to be a somewhat horrifying one. I understand and completely agree that academic authors can not reasonably be required to become experts in web design. --- From my perspective, the main issue seems to be broken formatting and media (alignment, hoverable citations and notes, missing or partial images, etc). The recent activation atlases paper is a particularly good example of the problem (https://distill.pub/2019/activation-atlas https://distill.pub/2019/activation-atlas). I don't have time to tally up all of the differences right now, but briefly just a few big ones so we have something concrete to refer to here: * The index.archive.html version shows "Loading..." in place of all the images in the first figure (and most of the other figures as well). * The Zotero web page snapshot version actually does have the images for the first figure, but that figure still has other issues. Many of the other figures still just show the "Loading..." text though. * In both cases, text is flowing around the figures in strange and unreadable ways. * The bibliography for the index.archive.html version is completely broken. * In both archive cases, uMatrix is showing third party scripts running on the page. I've allowed them, but obviously loading things from remote locations means it isn't really standalone anymore. --- > we haven't been able to think of a satisfying automatic way of providing static fallbacks to interactive diagrams. Why does it have to be static? Why worry about graceful degradation at all? I guess I just find myself wondering why there would be two versions of a scientific paper (ie "real" and "archive") in the first place? I don't see any reason why scientific work needs to be published in a static medium, but it does seem like it should always be standalone and immutable. To that end, why not require submissions themselves to be fully standalone entities with absolutely no external dependencies (ie review them without internet access) and then just publish those? Then instead of pdf papers, you end up with interactive (but standalone) html papers that are viewable in any standards compliant web browser, scripts and all. Barring such a drastic solution, I don't have any clever ideas from a technical standpoint; I'm not a web design expert nor am I familiar with what tooling you allow authors to make use of in their papers. A fairly blunt approach might just be to require authors to provide static images as standins for each figure. It could literally just be a static image of what the interactive figure initially looks like before you do anything with it. Then you'd just have the existing formatting issues to sort out, but presumably those are fairly straightforward. An aside: Figures don't seem to be numbered in the activation atlases paper I referenced above. It might be nice to add a requirement to clearly number and label figures, tables, and other such insets - as far as I know, all the major academic publications do this. If you're wondering why, at minimum it makes it much easier to discuss the paper when you have a clear label such as "fig 3a" or "fig 4 panel 2" or whatever to refer to.

by Reelin 8y ago

phreeza 8y ago

Haven't tried it on distil, but maybe this works? https://github.com/jgm/standalone-html https://github.com/jgm/standalone-html

Reelin 8y ago
Looks interesting, I'll try it out when I get a chance. I suspect it functions in a similar manner to Zotero's built in archiver though, so I'm not optimistic about what the results will look like.

RustingSword 8y ago

I use Zotero with [Zotero Connector](https://www.zotero.org/download/connectors https://www.zotero.org/download/connectors) plugin for Firefox. When saving a webpage to Zotero, it will also save a snapshot of that webpage (a configurable option in Zotero preferences). I tried it with latest two Distill posts, and didn't see any missing images. Actually for those interactive images, there will be a static version and an interactive version of that image in the snapshot, which is acceptable for me. Another problem more annoying is that those inline citations will show up even if you don't hover over them, thus covering underlying texts.

Reelin 8y ago
Forgot to mention originally, but I did try Zotero's snapshot feature. It seems to work better for some papers than others - the activation atlases one (https://distill.pub/2019/activation-atlas/ https://distill.pub/2019/activation-atlas/) is particularly bad. As you note, the hoverable citations and notes are also a problem. It's not that I have any pressing need for offline copies at this precise instant, but they have proven useful on a number of occasions.

clickok 8y ago

It depends how deep you want to go into the web archiving rabbit-hole, but using something like `archivenow`[0] would probably be the local optimum in terms of convenience, robustness, and permanence. It's "a tool to push web resources into web archives", which basically entails making requests of archiving services (like the Wayback machine, for example). Using `archivenow --all <url>`, you can back up <url> to a whole bunch of different services, and get links to snapshots of that page at the time of access. This is mostly sufficient, so long as at least one service is trustworthy and doesn't shut down unexpectedly. One issue that might come up is that the archives might miss content if the user has to interact with the page a bunch in order to reveal it, but that's unlikely to be an issue for the sorts of things an academic would be archiving. As another poster mentioned, some bibliography managers have this capability baked in, or can acquire it with a plugin. However, if you want more control, you might consider using something like `pywb`[1] to make your own web archive, and then host it yourself (e.g., on S3 or something). For more robustness, you could also publish those WARC files as torrents, or on IPFS, or if you're really cool, as part of the transaction messages on some blockchain (provided that you can stomach the transaction fees and don't anticipate having a large audience in China). The above will probably suffice so long as the Internet continues to function as a going concern. To accommodate concerns about what happens in the case of total societal collapse, I've been backing up the papers that I cite by converting to binary[2] and preserving the result via aluminum punchcards, which I then mail to Sam Altman since he's got a way better game plan for the end times than I do[3]. I'm not sure if aluminum is optimal for this purpose though, both for material science reasons and because I've been informed that I am under investigation for my role in some sort of aluminum smuggling ring. If that fails, my last backup plan entails constructing an allegorical representation of the papers in question (which is surprisingly easy-- the Hero's Journey from Joseph Campbell[4]'s exegesis is basically about differential equations anyways), which I will then craft into a compelling mythology and teach to the local youths. Hopefully this will ensure that the citations are preserved via oral tradition even in the wake of cataclysmic change. Figuring out the BibTeX for this archival format will be pretty tricky, though. But yeah, unless you have some extra aluminum lying around, `archivenow` or whatever's built-in to your reference manager should do the trick. --- 0. github.com/oduwsdl/archivenow 1. github.com/webrecorder/pywb 2. After stripping most of the markup removing images using `pandoc`, as aluminum is expensive. 3. https://www.bloomberg.com/features/2018-rich-new-zealand-doomsday-preppers/ https://www.bloomberg.com/features/2018-rich-new-zealand-doo... 4. https://en.wikipedia.org/wiki/Joseph_Campbell https://en.wikipedia.org/wiki/Joseph_Campbell

ludwigschubert 8y ago

Hi! I help maintain Distill's infrastructure. We'd love to find a robust solution for archival, too, but we haven't been able to think of a satisfying automatic way of providing static fallbacks to interactive diagrams. (That's why index.archive.html isn't a public feature ^_^) If you have an idea of what we could do on our side with a fixed amount of work, please let me know! Unfortunately we can't require our third-party authors to be experts in graceful degradation. I'm sure better tooling could help, but currently have no specific inspiration. As a fallback for your specific use case—we do actually provide a tiny bit of print styles in our CSS. Try your browser's PDF conversion/print feature; I find it often produces acceptable results. (Minus proper alignment to pages, of course. :/)

Reelin 8y ago
Really glad to hear you guys are aware of this! I do love the expressiveness and ease of understanding that interactive diagrams offer, but find the prospect of a mutable scientific literature to be a somewhat horrifying one. I understand and completely agree that academic authors can not reasonably be required to become experts in web design. --- From my perspective, the main issue seems to be broken formatting and media (alignment, hoverable citations and notes, missing or partial images, etc). The recent activation atlases paper is a particularly good example of the problem (https://distill.pub/2019/activation-atlas https://distill.pub/2019/activation-atlas). I don't have time to tally up all of the differences right now, but briefly just a few big ones so we have something concrete to refer to here: * The index.archive.html version shows "Loading..." in place of all the images in the first figure (and most of the other figures as well). * The Zotero web page snapshot version actually does have the images for the first figure, but that figure still has other issues. Many of the other figures still just show the "Loading..." text though. * In both cases, text is flowing around the figures in strange and unreadable ways. * The bibliography for the index.archive.html version is completely broken. * In both archive cases, uMatrix is showing third party scripts running on the page. I've allowed them, but obviously loading things from remote locations means it isn't really standalone anymore. --- > we haven't been able to think of a satisfying automatic way of providing static fallbacks to interactive diagrams. Why does it have to be static? Why worry about graceful degradation at all? I guess I just find myself wondering why there would be two versions of a scientific paper (ie "real" and "archive") in the first place? I don't see any reason why scientific work needs to be published in a static medium, but it does seem like it should always be standalone and immutable. To that end, why not require submissions themselves to be fully standalone entities with absolutely no external dependencies (ie review them without internet access) and then just publish those? Then instead of pdf papers, you end up with interactive (but standalone) html papers that are viewable in any standards compliant web browser, scripts and all. Barring such a drastic solution, I don't have any clever ideas from a technical standpoint; I'm not a web design expert nor am I familiar with what tooling you allow authors to make use of in their papers. A fairly blunt approach might just be to require authors to provide static images as standins for each figure. It could literally just be a static image of what the interactive figure initially looks like before you do anything with it. Then you'd just have the existing formatting issues to sort out, but presumably those are fairly straightforward. An aside: Figures don't seem to be numbered in the activation atlases paper I referenced above. It might be nice to add a requirement to clearly number and label figures, tables, and other such insets - as far as I know, all the major academic publications do this. If you're wondering why, at minimum it makes it much easier to discuss the paper when you have a clear label such as "fig 3a" or "fig 4 panel 2" or whatever to refer to.