7 ms·

As I've been playing with Raspi's and Beaglebones and stuff lately, it's been driving me nuts that EVERYTHING I do needs to be apt-gotten off the internet, the

As I've been playing with Raspi's and Beaglebones and stuff lately, it's been driving me nuts that EVERYTHING I do needs to be apt-gotten off the internet, the base image doesn't even include basics like screen/tmux. If those repos are inaccessable for any reason, I have a bunch of hardware that's very hard to do anything useful with. I know there are such things as apt-caches and squid caches and stuff, but I could really use thing that goes through every apt-get I've ever done and the top 50,000 packages on github and stuffs 'em all onto an SD card and shows me how to use them from my commandline. OP mentions this as a future direction for the project, but I think it's one of the most important.

iam-TJ 7y ago
On Debian-based distros there are several tools. squid-deb-proxy can do most of this - it fetches and locally caches lists and packages. Clients install squid-deb-proxy-client which uses multicast-DNS to discover the local proxy. Packages are fetched and cached by the proxy first time they're requested and thereafter served locally (subject to lifetime/space etc. squid configuration). There are ways to pre-populate the cache but you're always going to have situations where package updates have to pull from remote archives. apt-mirror is designed for a similar purpose. It allows creating a sub-set or entire mirror of one or more upstream archives. Clients can then be pointed at the local mirror which could be on the same host or LAN. Generally, 'packages' on github will be source-code only so you'd need a quite complex build server, or at least per-package specific links in your local cache rules for pre-built binaries.
- myself248 7y ago
  Update: In case anyone finds this later, I just ran across this page of someone else doing EXACTLY what I wanted to do, for exactly the same reason: https://blog.thelifeofkenneth.com/2018/01/off-grid-raspbian-repositories.html https://blog.thelifeofkenneth.com/2018/01/off-grid-raspbian-...
ekianjo 7y ago
> it's been driving me nuts that EVERYTHING I do needs to be apt-gotten off the internet, One solution is to do it once only: grab the base image, install all the packages you need, and then make an image of it so you don't start from scratch again next time.
- willis936 7y ago
  This is my plan (in action) for an offline GPS nixie tube clock. I'm not sure about what is the right choice for options. Many are nice-to-have features for people like me, who would be the most likely to want a clock like this (snmp to host drift, jitter, and temperature mrtg logs), but come at the mild cost of security. Will those who receive this bother to connect it to networks? Should I even enable network/GUI for those who want to tinker easily? How should the password be handled? I'm considering two extremes: 1. A raspbian lite headless system with all unnecessary peripherals disabled and no network features. 2. A full installation, GUI and all with HDMI and network enabled so it's easy for people to play with it if they want. I'm leaning towards 2 because I think it would be nice to have a clock that auto updates leap seconds and the tzdata db. If I go for 1, that opens up the possibility of using a raspi zero (non-W) for a real BOM and power savings. As for the password: I'm thinking of having something short and simple other than "raspberry" that the user can/should change. This seems like standard practice with many enterprise systems.
- stjohnswarts 7y ago
  This is what I do, and it works well.
- myself248 7y ago
  But next time I'll be doing something different. That's like saying "the only internet you need is that which is already in your browser history". No. What I want is something closer to Kiwix already on an SD card so even if everything is down, I can read about something I've never been interested in before. But for software.
q3k 7y ago
NixOS solves all of these problems. a) define your entire system state and dependencies in a single, declarative file b) prebake an image based on this file It's all done and ready, available now, no hacking around required.
- dfc 7y ago
  (A) is also possible with Debian. OP could just make a pressed file for debinstaller. (B) is probably accomplished with any OS.
- pjc50 7y ago
  .. but if you discover you need something, you have to go back to (a).
  - q3k 7y ago
    Your prebaked image will contain the definition (a), and you can incrementally continue building upon it, then regenerate yet another image if needed.
- KirinDave 7y ago
  This is precisely what every linux distro does. Nix isn't providing novel functionality for bootable OS images. But a bigger question for the ergonomics of NixOS: Are NixOS and Nix prebuilding for ARM now?
  - q3k 7y ago
    > This is precisely what every linux distro does. Nix isn't providing novel functionality for bootable OS images. None of them provide native, single-source-of-truth declarative configuration that is easy to reason about, pure, and guaranteed to deliver sane results every time (vs. something managed via a classic CM system). Oh, also one that it symmetrical to the way the distribution itself is built and managed. > But a bigger question for the ergonomics of NixOS: Are NixOS and Nix prebuilding for ARM now? Yes. https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi Ports to other ARM devices are also very easy.
mister_hn 7y ago
You can build your own image/rootfs Imagine that you can build your rootfs is a docker container and export it to IMG
pjc50 7y ago
Debian provide a physical artefact containing all the packages that could be used without internet access. I don't know whether this also works for Raspian: https://www.debian.org/CD/ https://www.debian.org/CD/
esotericn 7y ago
FWIW providing you have the kernel and firmware basically everything can be built from source. I've built most of the base repository of Arch Linux on my RPi4 without cross compiling. I'm buying a few more so that I can go 'full Gentoo' and pretty much rebuild the world. Have a recording of a container booting from a rebuilt userland: https://asciinema.org/a/283303 https://asciinema.org/a/283303 It might feel like the ARM packages are somehow special but really only the blobby bits like the RPi firmware is. Everything else is just your bog standard armv7/aarch64 ELF. So yeah, back up a GCC binary or bootstrap, I guess? I can probably email you a tmux binary in a pinch, plus I have a full local mirror of the repo for the apocalypse? :P
pnutjam 7y ago
Can't you rsync down your own repository?
- nextlevelwizard 7y ago
  Why would you even want to do that?
  - pnutjam 7y ago
    to keep a local copy to work from
rubenbe 7y ago
This is why projects like yocto and buildroot exist. They help you maintain a stable software stack over time. And generate a fully contained software image that can be flashed without needing the internet afterwards. (All downloads happen during compilation)
- myself248 7y ago
  Thank you for filling in a gap in my understanding! I'm pretty severely on the noob end of the scale when it comes to software, so I don't know if maintaining a new distro for myself would make sense, since all the tutorials I'm following assume I have Raspbian and all its built-ins available to me. But maybe in the post-apocalypse, some hero with a prebuilt Rasp-yocto will rescue all our useless boards.
AnIdiotOnTheNet 7y ago
Yep, big problem with the repo model that Linux insists on using for everything. Offline is a 5th class citizen because of course everyone lives in SV or a university dorm.
- franga2000 7y ago
  I see the repo model as the closest we've come to a solution. Not only are there mirrors in case the main repo drops, because all software is in one repo it's extreme easy to cache/sync/download. Offline mirrors are even an officially supported use case with many package managers. Have enough space? Mirror the whole thing. Not? Select only the interesting package groups. Missed one? You can probably get it from another machine it's installed on. Want to have an offline copy of software on Windows? Go download all of the installers from their respective sites. Want it to constantly update - that's gonna be about 100 lines of Python code. And better make sure they're the "offline" installers, not those idiotic stubs that just dl the latest version for a server. Also don't forget about all of the 20 different VC++ runtimes that are sometimes not packaged in the installer. I have a feeling it would be even harder on macOS. (all of this is of course ignoring the legal issues of distributing those files - which is generally not an issue on Linux)
  - AnIdiotOnTheNet 7y ago
    > because all software is in one repo All software is most definitely not in one repo. If it were, people wouldn't have to use PPAs or compile from source and deal with all the problems those things cause. > better make sure they're the "offline" installers, not those idiotic stubs that just dl the latest version for a server Oh, you mean the ones that aren't just acting like apt. > I have a feeling it would be even harder on macOS. Depends, but MacOS software that uses Application Bundles should be fine. Even a lot of Windows software works fine if you just copy the installer contents to a directory. Self-contained application directories (or single files) are an old as dirt concept that Linux communities never got behind, preferring instead to use overly complicated schemes like package management that come with a bunch of their own problems. So many, in fact, that people are now often distributing applications in Docker images.
swiley 7y ago
Most of the important stuff is in busybox and gcc (maybe Python.) Cache those and their dependancies and you have most of what you want.
granra 7y ago
You may want to look at debos for pre-baking debian/ubuntu images. https://github.com/go-debos/debos https://github.com/go-debos/debos
qubex 7y ago
Yes, software accessibility is an issue in the most pronounced disaster scenarios. But anybody who looks at a device of such exquisite craftsmanship, which is evidently already set up to function, and frets about downloading software after some apocalyptic scenario has missed at least half the point.
m463 7y ago
Since these are small devices, I think a minimal image is a necessity. What you will use is not what someone else will use. As to caching, I tried setting one up and found out a few things. I originally tried apt-cacher-ng: https://www.unix-ag.uni-kl.de/~bloch/acng/ https://www.unix-ag.uni-kl.de/~bloch/acng/ but had trouble (can't remember exactly what) and muddled through polipo instad. I found out: - it was a lot easier and faster than copying cached packages around - when doing "apt-get update; apt-get upgrade" the cache really sped up multi-machine updates. The first machine was slow, the rest were fast. - By far, the majority of cache traffic was for updates to base stuff. foo-1.2, foo-1.2.1 foo-1.2.2, etc. - second was prerequisites for packages I needed. - the few specialized packages I used were not updated as frequently.
stjohnswarts 7y ago
I don't think I've ever seen the rpi apt repository offline, so I'm not sure why you're so worried. Embedded computers tend towards lean rather than heavy and that is the mindset you're going to encounter.
nextlevelwizard 7y ago
>the base image doesn't even include basics like screen/tmux. Easily arguable that RPi isn't made with such use cases in mind. I'm all in favor of removing unnecessary bloat if only a small percentage of user base is ever going use it, especially when the software is readily available. >If those repos are inaccessable for any reason, I have a bunch of hardware that's very hard to do anything useful with. How so? They are just Linux boxes, you can just download the source code and compile the binaries you need. Pre-built packages are not necessary for functional OS.
- taneq 7y ago
  Download the source code, from the repos?
- SmellyGeekBoy 7y ago
  > How so? They are just Linux boxes, you can just download the source code and compile the binaries you need. Pre-built packages are not necessary for functional OS. What about dependencies? If you have the internet access required to download the source code I'd say you'd be better off just using the repos.
ppf 7y ago
How often are the repos inaccessible? I've come around to the opinion that, unless you have a very specific use case, networking is an essential element of any (Linux) computer system. Once you have that, you can benefit from a world of free software, with updates all handled automatically. Personally I find it almost magical that I can install and update almost any software I need with a short command.

iam-TJ 7y ago

On Debian-based distros there are several tools. squid-deb-proxy can do most of this - it fetches and locally caches lists and packages. Clients install squid-deb-proxy-client which uses multicast-DNS to discover the local proxy. Packages are fetched and cached by the proxy first time they're requested and thereafter served locally (subject to lifetime/space etc. squid configuration). There are ways to pre-populate the cache but you're always going to have situations where package updates have to pull from remote archives. apt-mirror is designed for a similar purpose. It allows creating a sub-set or entire mirror of one or more upstream archives. Clients can then be pointed at the local mirror which could be on the same host or LAN. Generally, 'packages' on github will be source-code only so you'd need a quite complex build server, or at least per-package specific links in your local cache rules for pre-built binaries.

myself248 7y ago
Update: In case anyone finds this later, I just ran across this page of someone else doing EXACTLY what I wanted to do, for exactly the same reason: https://blog.thelifeofkenneth.com/2018/01/off-grid-raspbian-repositories.html https://blog.thelifeofkenneth.com/2018/01/off-grid-raspbian-...

ekianjo 7y ago

> it's been driving me nuts that EVERYTHING I do needs to be apt-gotten off the internet, One solution is to do it once only: grab the base image, install all the packages you need, and then make an image of it so you don't start from scratch again next time.

willis936 7y ago
This is my plan (in action) for an offline GPS nixie tube clock. I'm not sure about what is the right choice for options. Many are nice-to-have features for people like me, who would be the most likely to want a clock like this (snmp to host drift, jitter, and temperature mrtg logs), but come at the mild cost of security. Will those who receive this bother to connect it to networks? Should I even enable network/GUI for those who want to tinker easily? How should the password be handled? I'm considering two extremes: 1. A raspbian lite headless system with all unnecessary peripherals disabled and no network features. 2. A full installation, GUI and all with HDMI and network enabled so it's easy for people to play with it if they want. I'm leaning towards 2 because I think it would be nice to have a clock that auto updates leap seconds and the tzdata db. If I go for 1, that opens up the possibility of using a raspi zero (non-W) for a real BOM and power savings. As for the password: I'm thinking of having something short and simple other than "raspberry" that the user can/should change. This seems like standard practice with many enterprise systems.
stjohnswarts 7y ago
This is what I do, and it works well.
myself248 7y ago
But next time I'll be doing something different. That's like saying "the only internet you need is that which is already in your browser history". No. What I want is something closer to Kiwix already on an SD card so even if everything is down, I can read about something I've never been interested in before. But for software.

q3k 7y ago

NixOS solves all of these problems. a) define your entire system state and dependencies in a single, declarative file b) prebake an image based on this file It's all done and ready, available now, no hacking around required.

dfc 7y ago
(A) is also possible with Debian. OP could just make a pressed file for debinstaller. (B) is probably accomplished with any OS.
pjc50 7y ago
.. but if you discover you need something, you have to go back to (a).
- q3k 7y ago
  Your prebaked image will contain the definition (a), and you can incrementally continue building upon it, then regenerate yet another image if needed.
KirinDave 7y ago
This is precisely what every linux distro does. Nix isn't providing novel functionality for bootable OS images. But a bigger question for the ergonomics of NixOS: Are NixOS and Nix prebuilding for ARM now?
- q3k 7y ago
  > This is precisely what every linux distro does. Nix isn't providing novel functionality for bootable OS images. None of them provide native, single-source-of-truth declarative configuration that is easy to reason about, pure, and guaranteed to deliver sane results every time (vs. something managed via a classic CM system). Oh, also one that it symmetrical to the way the distribution itself is built and managed. > But a bigger question for the ergonomics of NixOS: Are NixOS and Nix prebuilding for ARM now? Yes. https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi Ports to other ARM devices are also very easy.

mister_hn 7y ago

You can build your own image/rootfs Imagine that you can build your rootfs is a docker container and export it to IMG

pjc50 7y ago

Debian provide a physical artefact containing all the packages that could be used without internet access. I don't know whether this also works for Raspian: https://www.debian.org/CD/ https://www.debian.org/CD/

esotericn 7y ago

FWIW providing you have the kernel and firmware basically everything can be built from source. I've built most of the base repository of Arch Linux on my RPi4 without cross compiling. I'm buying a few more so that I can go 'full Gentoo' and pretty much rebuild the world. Have a recording of a container booting from a rebuilt userland: https://asciinema.org/a/283303 https://asciinema.org/a/283303 It might feel like the ARM packages are somehow special but really only the blobby bits like the RPi firmware is. Everything else is just your bog standard armv7/aarch64 ELF. So yeah, back up a GCC binary or bootstrap, I guess? I can probably email you a tmux binary in a pinch, plus I have a full local mirror of the repo for the apocalypse? :P

pnutjam 7y ago

Can't you rsync down your own repository?

nextlevelwizard 7y ago
Why would you even want to do that?
- pnutjam 7y ago
  to keep a local copy to work from

rubenbe 7y ago

This is why projects like yocto and buildroot exist. They help you maintain a stable software stack over time. And generate a fully contained software image that can be flashed without needing the internet afterwards. (All downloads happen during compilation)

myself248 7y ago
Thank you for filling in a gap in my understanding! I'm pretty severely on the noob end of the scale when it comes to software, so I don't know if maintaining a new distro for myself would make sense, since all the tutorials I'm following assume I have Raspbian and all its built-ins available to me. But maybe in the post-apocalypse, some hero with a prebuilt Rasp-yocto will rescue all our useless boards.

AnIdiotOnTheNet 7y ago

Yep, big problem with the repo model that Linux insists on using for everything. Offline is a 5th class citizen because of course everyone lives in SV or a university dorm.

franga2000 7y ago
I see the repo model as the closest we've come to a solution. Not only are there mirrors in case the main repo drops, because all software is in one repo it's extreme easy to cache/sync/download. Offline mirrors are even an officially supported use case with many package managers. Have enough space? Mirror the whole thing. Not? Select only the interesting package groups. Missed one? You can probably get it from another machine it's installed on. Want to have an offline copy of software on Windows? Go download all of the installers from their respective sites. Want it to constantly update - that's gonna be about 100 lines of Python code. And better make sure they're the "offline" installers, not those idiotic stubs that just dl the latest version for a server. Also don't forget about all of the 20 different VC++ runtimes that are sometimes not packaged in the installer. I have a feeling it would be even harder on macOS. (all of this is of course ignoring the legal issues of distributing those files - which is generally not an issue on Linux)
- AnIdiotOnTheNet 7y ago
  > because all software is in one repo All software is most definitely not in one repo. If it were, people wouldn't have to use PPAs or compile from source and deal with all the problems those things cause. > better make sure they're the "offline" installers, not those idiotic stubs that just dl the latest version for a server Oh, you mean the ones that aren't just acting like apt. > I have a feeling it would be even harder on macOS. Depends, but MacOS software that uses Application Bundles should be fine. Even a lot of Windows software works fine if you just copy the installer contents to a directory. Self-contained application directories (or single files) are an old as dirt concept that Linux communities never got behind, preferring instead to use overly complicated schemes like package management that come with a bunch of their own problems. So many, in fact, that people are now often distributing applications in Docker images.

swiley 7y ago

Most of the important stuff is in busybox and gcc (maybe Python.) Cache those and their dependancies and you have most of what you want.

granra 7y ago

You may want to look at debos for pre-baking debian/ubuntu images. https://github.com/go-debos/debos https://github.com/go-debos/debos

qubex 7y ago

Yes, software accessibility is an issue in the most pronounced disaster scenarios. But anybody who looks at a device of such exquisite craftsmanship, which is evidently already set up to function, and frets about downloading software after some apocalyptic scenario has missed at least half the point.

m463 7y ago

Since these are small devices, I think a minimal image is a necessity. What you will use is not what someone else will use. As to caching, I tried setting one up and found out a few things. I originally tried apt-cacher-ng: https://www.unix-ag.uni-kl.de/~bloch/acng/ https://www.unix-ag.uni-kl.de/~bloch/acng/ but had trouble (can't remember exactly what) and muddled through polipo instad. I found out: - it was a lot easier and faster than copying cached packages around - when doing "apt-get update; apt-get upgrade" the cache really sped up multi-machine updates. The first machine was slow, the rest were fast. - By far, the majority of cache traffic was for updates to base stuff. foo-1.2, foo-1.2.1 foo-1.2.2, etc. - second was prerequisites for packages I needed. - the few specialized packages I used were not updated as frequently.

stjohnswarts 7y ago

I don't think I've ever seen the rpi apt repository offline, so I'm not sure why you're so worried. Embedded computers tend towards lean rather than heavy and that is the mindset you're going to encounter.

nextlevelwizard 7y ago

>the base image doesn't even include basics like screen/tmux. Easily arguable that RPi isn't made with such use cases in mind. I'm all in favor of removing unnecessary bloat if only a small percentage of user base is ever going use it, especially when the software is readily available. >If those repos are inaccessable for any reason, I have a bunch of hardware that's very hard to do anything useful with. How so? They are just Linux boxes, you can just download the source code and compile the binaries you need. Pre-built packages are not necessary for functional OS.

taneq 7y ago
Download the source code, from the repos?
SmellyGeekBoy 7y ago
> How so? They are just Linux boxes, you can just download the source code and compile the binaries you need. Pre-built packages are not necessary for functional OS. What about dependencies? If you have the internet access required to download the source code I'd say you'd be better off just using the repos.

ppf 7y ago

How often are the repos inaccessible? I've come around to the opinion that, unless you have a very specific use case, networking is an essential element of any (Linux) computer system. Once you have that, you can benefit from a world of free software, with updates all handled automatically. Personally I find it almost magical that I can install and update almost any software I need with a short command.