4 ms·

I posted a link to the other comment, but here it is for you as well: https://juuso.dev/blogPosts/nix-as-a-static-site-generator/nix-as-a-static-site-generator.

by Jhsto 3y ago

I posted a link to the other comment, but here it is for you as well: https://juuso.dev/blogPosts/nix-as-a-static-site-generator/nix-as-a-static-site-generator.html https://juuso.dev/blogPosts/nix-as-a-static-site-generator/n...

__MatrixMan__ 3y ago
I admire how nimble you are. I aspire to write blog posts at the drop of a hat like this, but I rarely do. I also like the use of flake inputs for content. It reminds me of a world that I've been imagining where the conclusions in scientific papers are generated as a flake outputs (an adjacent output would be the human readable thing, a PDF or whatever). In this world, you can just run `nix flake update && nix build`, and if a paper that you cite published an update which invalidates your conclusion, you know right away because their output is your input, so your build fails. We think about repeatable builds being for executable binaries, but they could equally well be for conclusions and assumptions. Perhaps nix is too big of a hammer for the job, but it seems like the best shot we have at achieving this without also constraining the scientist re: tooling. I realize that you don't want to be storing mountains of data in the nix store, but it would work just as well if the output in question is an IPFS CID, to be resolved during the build instead. The publisher can then be in charge of keeping that CID pinned and of notifying scientists when they're "build" starts failing.
- Jhsto 3y ago
  > I admire how nimble you are. I aspire to write blog posts at the drop of a hat like this, but I rarely do. Thanks! I took up blogging more often as of recent, and for me, having a manageable system is a large part of that. The last thing I want to happen on a Sunday evening is breaking some page of my website. That being said, I hope to one day make the workflow easier. > It reminds me of a world that I've been imagining where the conclusions in scientific papers are generated as a flake outputs (and adjacent output would be the human readable thing, a PDF or whatever). I happen to be a reviewer for software artifacts in a scientific journal, and I often use Nix here. Not that many projects do use it, but if I'm able to reproduce it with Nix, then I know the author has not missed any implicit dependencies. I like to imagine it's also useful for the authors as a feedback, whether they use Nix or not. > I realize that you don't want to be storing mountains of data in the nix store, but it would work just as well if the output in question is an IPFS CID, to be resolved during the build instead. I maintain separate build serves of my own using Nix integrations and the Nix cache is quite large already (so called remote builders) sitting at around 500GB. I host these at Hetzner. I have also thought adding IPFS integration for my website, but haven't got around to it.
  - __MatrixMan__ 3y ago
    > I happen to be a reviewer for software artifacts in a scientific journal That's very cool. I have a question for you. I'm taking a bioinformatics class, despite not having the chemistry prerequisites. I'm getting a crash course in biochem, and the rest of the class benefits from having an expert in what-kind-of-quotes-to-use. I've been thinking: would it be helpful if the care and maintenance of these compute environments wasn't left to each scientist but was instead aggregated (perhaps per-class or per-university)? We're setting these chemists up with conda in Ubuntu in WSL in a terminal whose startup command activates the conda environment. Not exactly a recipe for reproducibility after they get a new laptop. What if certain compute-heavy classes published flakes which the students could... a) use while taking the class so we stop wasting time on troubleshooting ssl deps via conda b) reference in publications after the fact. They could say: > Here's a Jupyter notebook, download it and run it in the UCCS biochem environment like so: `nix run github:UCCS/CHEM4573?rev=16afd67`, its output lets us make the following conclusions... I know it would be helpful for the students in the class. Do you think it would be helpful to them later on when they were publishing things? I'm thinking about packaging the dependencies for this class, giving it to the teacher, and pitching it to the university: > Set up a technical fellowes program. Waive tuition for us nerds and in exchange we'll support your students and faculty through the maintenance of these environments. I don't mind paying tuition so much, but I'd like to do something to get a bit more cross pollination going between scientists in need of tech support and techies in need of something meaningful to work on. Am I dreaming here, or would it solve some problems? Do you think I have a shot at convincing anybody?
    - Jhsto 3y ago
      Couple of thoughts: > I've been thinking: would it be helpful if the care and maintenance of these compute environments wasn't left to each scientist but was instead aggregated (perhaps per-class or per-university)? This is definitely something that Nix can abstract quite well. In my company we have [an infrastructure of computers](https://github.com/ponkila/homestaking-infra https://github.com/ponkila/homestaking-infra) that we manage with NixOS. We have gone over the system such that `cd` into a directory "conjures" the environment using devenv or direnv. We don't do anything too fancy yet, but we have a project commencing next month in which we start to also manage routers this way. We speculate that this will help us to do things such as follows: register new node, and it gets automatically imported by the router which establishes DNS entries and SSH keys for each user. The idea is that we could have different "views" of the infrastructure depending on the user which the router could control. For administrators, we have a separate UI created with React that pulls NixOS configuration declarations from a git repository (note: these don't have to be public) and shows how the nodes connect with each other. The UI is still under construction, but imagine this but now with more nodes: https://imgur.com/a/obBfRk0 https://imgur.com/a/obBfRk0. We have this set up at https://homestakeros.com https://homestakeros.com. Depending on a project you are working on, you could then have a subset of the infrastructure be shown to the user and have things such as SSH aliases and other niceties set up on `cd` in. When you `cd` out, then your view is destroyed. We have quite overengineered this approach -- we run the nodes from RAM. NixOS has the idea of "delete your darlings" which is having a temporary rootfs. We have gone the extra mile that we don't even store the OS on the computer, the computers boot via PXE and load the latest image from the router (though any HTTP server will do, I boot some from CloudFlare). We do this because it also forces the administrators to document changes that they do -- there is nothing worse than starting to call up people when theres is downtime and try to figure your way back up from what the mutations are. PXE booting establishes a working initial state for each node -- you just reboot the computer, and you are guaranteed to get into a working initial state. I'm personally big on this -- all my servers and even my laptop works like this. We upgrade servers by using kexec -- the NixOS configurations produce self-contained kexec scripts and ISO images for hypervisors (some stakeholders insist on running on Proxmox). I've suggested some kernel changes to NixOS which would allow boostrapping arbitrary size initial ramdisks, because otherwise you are limited to 2GB file size. > We're setting these chemists up with conda in Ubuntu in WSL in a terminal whose startup command activates the conda environment. Not exactly setting them up for reproducibility if they ever move to a different laptop. Python in specific is a PITA to setup with Nix, dream2nix etc., might help but it's definitely the hardest environment to set up of all languages I've tried -- even GPGPU environments are easier. Oftentimes, the only problem is not the packaging, but also the infrastructure used. For that, you could also publish the NixOS configurations and maybe distribute the kexec or ISO images. A notable thing is that devenv also allows creation of containers from the devShell environment, which may further help your case. Researchers could reference docker images instead of insisting on everyone to use Nix. In any case, I put some emails on my HN profile so we can also take the discussion off platform -- we are looking for test users for the holistic approach using PXE, and we are currently funded until Q3 next year.
theK 3y ago
Reading through your link I caught myself thinking if I would put up with all those boilerplate nix steps just to add a new page to the site. Don't get me wrong, I get that you gain big amounts of flexibility out of it the way you do it but if we think about the tasksat hand, adding a page to a predefined blog, it seems a bit involved.
- Jhsto 3y ago
  A fair comment, I do not disagree. I do plan to one day do an `ls` command on the root Nix file so that the manual update to the root flake for both the inputs and the RSS feed would be redundant.

by Jhsto 3y ago

__MatrixMan__ 3y ago

I admire how nimble you are. I aspire to write blog posts at the drop of a hat like this, but I rarely do. I also like the use of flake inputs for content. It reminds me of a world that I've been imagining where the conclusions in scientific papers are generated as a flake outputs (an adjacent output would be the human readable thing, a PDF or whatever). In this world, you can just run `nix flake update && nix build`, and if a paper that you cite published an update which invalidates your conclusion, you know right away because their output is your input, so your build fails. We think about repeatable builds being for executable binaries, but they could equally well be for conclusions and assumptions. Perhaps nix is too big of a hammer for the job, but it seems like the best shot we have at achieving this without also constraining the scientist re: tooling. I realize that you don't want to be storing mountains of data in the nix store, but it would work just as well if the output in question is an IPFS CID, to be resolved during the build instead. The publisher can then be in charge of keeping that CID pinned and of notifying scientists when they're "build" starts failing.

Jhsto 3y ago
> I admire how nimble you are. I aspire to write blog posts at the drop of a hat like this, but I rarely do. Thanks! I took up blogging more often as of recent, and for me, having a manageable system is a large part of that. The last thing I want to happen on a Sunday evening is breaking some page of my website. That being said, I hope to one day make the workflow easier. > It reminds me of a world that I've been imagining where the conclusions in scientific papers are generated as a flake outputs (and adjacent output would be the human readable thing, a PDF or whatever). I happen to be a reviewer for software artifacts in a scientific journal, and I often use Nix here. Not that many projects do use it, but if I'm able to reproduce it with Nix, then I know the author has not missed any implicit dependencies. I like to imagine it's also useful for the authors as a feedback, whether they use Nix or not. > I realize that you don't want to be storing mountains of data in the nix store, but it would work just as well if the output in question is an IPFS CID, to be resolved during the build instead. I maintain separate build serves of my own using Nix integrations and the Nix cache is quite large already (so called remote builders) sitting at around 500GB. I host these at Hetzner. I have also thought adding IPFS integration for my website, but haven't got around to it.
- __MatrixMan__ 3y ago
  > I happen to be a reviewer for software artifacts in a scientific journal That's very cool. I have a question for you. I'm taking a bioinformatics class, despite not having the chemistry prerequisites. I'm getting a crash course in biochem, and the rest of the class benefits from having an expert in what-kind-of-quotes-to-use. I've been thinking: would it be helpful if the care and maintenance of these compute environments wasn't left to each scientist but was instead aggregated (perhaps per-class or per-university)? We're setting these chemists up with conda in Ubuntu in WSL in a terminal whose startup command activates the conda environment. Not exactly a recipe for reproducibility after they get a new laptop. What if certain compute-heavy classes published flakes which the students could... a) use while taking the class so we stop wasting time on troubleshooting ssl deps via conda b) reference in publications after the fact. They could say: > Here's a Jupyter notebook, download it and run it in the UCCS biochem environment like so: `nix run github:UCCS/CHEM4573?rev=16afd67`, its output lets us make the following conclusions... I know it would be helpful for the students in the class. Do you think it would be helpful to them later on when they were publishing things? I'm thinking about packaging the dependencies for this class, giving it to the teacher, and pitching it to the university: > Set up a technical fellowes program. Waive tuition for us nerds and in exchange we'll support your students and faculty through the maintenance of these environments. I don't mind paying tuition so much, but I'd like to do something to get a bit more cross pollination going between scientists in need of tech support and techies in need of something meaningful to work on. Am I dreaming here, or would it solve some problems? Do you think I have a shot at convincing anybody?
  - Jhsto 3y ago
    Couple of thoughts: > I've been thinking: would it be helpful if the care and maintenance of these compute environments wasn't left to each scientist but was instead aggregated (perhaps per-class or per-university)? This is definitely something that Nix can abstract quite well. In my company we have [an infrastructure of computers](https://github.com/ponkila/homestaking-infra https://github.com/ponkila/homestaking-infra) that we manage with NixOS. We have gone over the system such that `cd` into a directory "conjures" the environment using devenv or direnv. We don't do anything too fancy yet, but we have a project commencing next month in which we start to also manage routers this way. We speculate that this will help us to do things such as follows: register new node, and it gets automatically imported by the router which establishes DNS entries and SSH keys for each user. The idea is that we could have different "views" of the infrastructure depending on the user which the router could control. For administrators, we have a separate UI created with React that pulls NixOS configuration declarations from a git repository (note: these don't have to be public) and shows how the nodes connect with each other. The UI is still under construction, but imagine this but now with more nodes: https://imgur.com/a/obBfRk0 https://imgur.com/a/obBfRk0. We have this set up at https://homestakeros.com https://homestakeros.com. Depending on a project you are working on, you could then have a subset of the infrastructure be shown to the user and have things such as SSH aliases and other niceties set up on `cd` in. When you `cd` out, then your view is destroyed. We have quite overengineered this approach -- we run the nodes from RAM. NixOS has the idea of "delete your darlings" which is having a temporary rootfs. We have gone the extra mile that we don't even store the OS on the computer, the computers boot via PXE and load the latest image from the router (though any HTTP server will do, I boot some from CloudFlare). We do this because it also forces the administrators to document changes that they do -- there is nothing worse than starting to call up people when theres is downtime and try to figure your way back up from what the mutations are. PXE booting establishes a working initial state for each node -- you just reboot the computer, and you are guaranteed to get into a working initial state. I'm personally big on this -- all my servers and even my laptop works like this. We upgrade servers by using kexec -- the NixOS configurations produce self-contained kexec scripts and ISO images for hypervisors (some stakeholders insist on running on Proxmox). I've suggested some kernel changes to NixOS which would allow boostrapping arbitrary size initial ramdisks, because otherwise you are limited to 2GB file size. > We're setting these chemists up with conda in Ubuntu in WSL in a terminal whose startup command activates the conda environment. Not exactly setting them up for reproducibility if they ever move to a different laptop. Python in specific is a PITA to setup with Nix, dream2nix etc., might help but it's definitely the hardest environment to set up of all languages I've tried -- even GPGPU environments are easier. Oftentimes, the only problem is not the packaging, but also the infrastructure used. For that, you could also publish the NixOS configurations and maybe distribute the kexec or ISO images. A notable thing is that devenv also allows creation of containers from the devShell environment, which may further help your case. Researchers could reference docker images instead of insisting on everyone to use Nix. In any case, I put some emails on my HN profile so we can also take the discussion off platform -- we are looking for test users for the holistic approach using PXE, and we are currently funded until Q3 next year.

theK 3y ago

Reading through your link I caught myself thinking if I would put up with all those boilerplate nix steps just to add a new page to the site. Don't get me wrong, I get that you gain big amounts of flexibility out of it the way you do it but if we think about the tasksat hand, adding a page to a predefined blog, it seems a bit involved.

Jhsto 3y ago
A fair comment, I do not disagree. I do plan to one day do an `ls` command on the root Nix file so that the manual update to the root flake for both the inputs and the RSS feed would be redundant.