4 ms·

Just yesterday I was thinking how badly I want an obscenely fast local datastore for Docker. I am managing a handful of clients right now who are all using Dock

Just yesterday I was thinking how badly I want an obscenely fast local datastore for Docker. I am managing a handful of clients right now who are all using Docker containers for their microservice environments and my local machine runs out of disk daily. What I really want is lightning quick network storage, but I don't think it would be feasible to roll out 10g networking in my current rental home.

devonkim 5y ago
10gbe can be done over RJ45 with a direct cable connection and is something I've done before in a pinch between my personal NAS and a client machine (just be careful about your storage protocol because NFS has its pros and cons just like iSCSI and FCoE, and with macOS as a client you're likely in for a Bad Time like I did).
- vitejose 5y ago
  Out of curiosity, what issues did you run into with NFS on macOS? I'm looking into building a NAS for a network of macOS, Linux, and BSD machines and figured NFS would be my first choice.
  - geerlingguy 5y ago
    I've had weird issues with some applications and NFS on macOS (notably, Final Cut Pro X and Photoshop). And I've also had weird issues with SMB and some apps (notably, Photoshop again!). But at least SMB has been pretty stable for years. NFS support has had some hiccups. A few 10.14 macOS releases broke NFSv3 and NFSv4 for me, but it was fixed in later releases.
  - 411111111111111 5y ago
    I'm not the author nor am I an apple user anymore, though i was at one point. NFS doesn't have a lot of metadata, so you loose a lot of functionality. iPhoto for example could cause major issues if you persisted its library on a nfs, potentially making the library irreparable.
  - devonkim 5y ago
    NFSv3 and NFSv4 clients were fine when local machines over TCP and always-on like with a previous Mac Mini I had but resuming from a laptop was pulling teeth and may or may not have been the reason for some of the crashes I got. iSCSI in theory should be better but I had a lot of problems getting the iSCSI initiator I found to work for myself despite being a pretty simple use case I thought, and I gave up on the project around then as I found macOS simply required more budget for myself to buy an appropriate iSCSI initiator. This was over 5 years ago and things may have changed between then and now.
  - BackBlast 5y ago
    I've had stability problems with nfs on MacOS. I ended up using smb instead. Smb ended up being CPU bound on my NAS due to encryption I haven't taken the time to figure out how to disable. It's close to line speed for me though. It's also reliable.
cbushko 5y ago
Would running something like the docker proxy below on an old laptop help? TLDR: A caching proxy for Docker; allows centralized management of (multiple) registries and their authentication; caches images from any registry. Caches the potentially huge blob/layer requests (for bandwidth/time savings), and optionally caches manifest requests ("pulls") to avoid rate-limiting. https://github.com/rpardini/docker-registry-proxy https://github.com/rpardini/docker-registry-proxy
- whalesalad 5y ago
  At some point you need to build images locally or store images locally to run a local docker-compose cluster. I need a fat block store for that.
  - cbushko 5y ago
    That is painful for sure.
kilroy123 5y ago
Just curious what size is your hard drive? I had this pain with my last macbook. I ALWAYS ran out of disk space. Never again! I went with a 1 TB drive for my M1.
- dijit 5y ago
  I offload docker to another machine and I'm still using about 400GiB of storage on my relatively new MacBook; and I basically never use this machine outside of work (and even then I didn't use it for like 4 months because I preferred linux). it has no local media or anything. I'm shocked by how much disk I've used actually. https://imgur.com/a/iHV4IKn https://imgur.com/a/iHV4IKn
- biztos 5y ago
  I got the 2TB drive so I can run out of even more disk space! ;-)
  - jbverschoor 5y ago
    A larger common image + saner node_modules should save a lot
yawniek 5y ago
why don't you simply use one or many nvme thunderbolt/usb disks and move the caches there?
- ehutch79 5y ago
  Not the op, but if the nas was pre-existing, and already serving other network clients, no reason to add a second one.
- cyberge99 5y ago
  Author here: I actually used a thunderbolt nvme and adapter for a bit. Then, for kicks I got another one and combined them in a raid-0 on macos. Realizing I didn’t want to run nvme’s, in raid-0 over two cables in any sort of production (homelab - I use production loosely here), is what sent me down the Thunderbolt rabbit hole that precipitated this blog post.
  - CharlesW 5y ago
    In case a single-cable solution would work, I've been using an external TB NVMe RAID 0 in this enclosure for several months and it's worked flawlessly: https://www.amazon.com/gp/product/B08S5JPWR6 https://www.amazon.com/gp/product/B08S5JPWR6 (The only drawback is that the power supply is bizarrely-large, and the power connector feels a bit janky. In practice it hasn't been an issue.)
- FullyFunctional 5y ago
  Not OP, but my files are shared with many hosts. Having local files are a PITA to manage coherent state.
pr0nin 5y ago
What about separate Linux box laying around just to run docker? 8GB M1 Mac just don't have enough RAM for docker VM. Looks like local docker server should just work exactly same way as local docker while you at same location.
- whalesalad 5y ago
  Then you get into the problem of low latency directory sync between your box and that host. I’m still trying to crack that nut.

devonkim 5y ago

10gbe can be done over RJ45 with a direct cable connection and is something I've done before in a pinch between my personal NAS and a client machine (just be careful about your storage protocol because NFS has its pros and cons just like iSCSI and FCoE, and with macOS as a client you're likely in for a Bad Time like I did).

vitejose 5y ago
Out of curiosity, what issues did you run into with NFS on macOS? I'm looking into building a NAS for a network of macOS, Linux, and BSD machines and figured NFS would be my first choice.
- geerlingguy 5y ago
  I've had weird issues with some applications and NFS on macOS (notably, Final Cut Pro X and Photoshop). And I've also had weird issues with SMB and some apps (notably, Photoshop again!). But at least SMB has been pretty stable for years. NFS support has had some hiccups. A few 10.14 macOS releases broke NFSv3 and NFSv4 for me, but it was fixed in later releases.
- 411111111111111 5y ago
  I'm not the author nor am I an apple user anymore, though i was at one point. NFS doesn't have a lot of metadata, so you loose a lot of functionality. iPhoto for example could cause major issues if you persisted its library on a nfs, potentially making the library irreparable.
- devonkim 5y ago
  NFSv3 and NFSv4 clients were fine when local machines over TCP and always-on like with a previous Mac Mini I had but resuming from a laptop was pulling teeth and may or may not have been the reason for some of the crashes I got. iSCSI in theory should be better but I had a lot of problems getting the iSCSI initiator I found to work for myself despite being a pretty simple use case I thought, and I gave up on the project around then as I found macOS simply required more budget for myself to buy an appropriate iSCSI initiator. This was over 5 years ago and things may have changed between then and now.
- BackBlast 5y ago
  I've had stability problems with nfs on MacOS. I ended up using smb instead. Smb ended up being CPU bound on my NAS due to encryption I haven't taken the time to figure out how to disable. It's close to line speed for me though. It's also reliable.

cbushko 5y ago

Would running something like the docker proxy below on an old laptop help? TLDR: A caching proxy for Docker; allows centralized management of (multiple) registries and their authentication; caches images from any registry. Caches the potentially huge blob/layer requests (for bandwidth/time savings), and optionally caches manifest requests ("pulls") to avoid rate-limiting. https://github.com/rpardini/docker-registry-proxy https://github.com/rpardini/docker-registry-proxy

whalesalad 5y ago
At some point you need to build images locally or store images locally to run a local docker-compose cluster. I need a fat block store for that.
- cbushko 5y ago
  That is painful for sure.

kilroy123 5y ago

Just curious what size is your hard drive? I had this pain with my last macbook. I ALWAYS ran out of disk space. Never again! I went with a 1 TB drive for my M1.

dijit 5y ago
I offload docker to another machine and I'm still using about 400GiB of storage on my relatively new MacBook; and I basically never use this machine outside of work (and even then I didn't use it for like 4 months because I preferred linux). it has no local media or anything. I'm shocked by how much disk I've used actually. https://imgur.com/a/iHV4IKn https://imgur.com/a/iHV4IKn
biztos 5y ago
I got the 2TB drive so I can run out of even more disk space! ;-)
- jbverschoor 5y ago
  A larger common image + saner node_modules should save a lot

yawniek 5y ago

why don't you simply use one or many nvme thunderbolt/usb disks and move the caches there?

ehutch79 5y ago
Not the op, but if the nas was pre-existing, and already serving other network clients, no reason to add a second one.
cyberge99 5y ago
Author here: I actually used a thunderbolt nvme and adapter for a bit. Then, for kicks I got another one and combined them in a raid-0 on macos. Realizing I didn’t want to run nvme’s, in raid-0 over two cables in any sort of production (homelab - I use production loosely here), is what sent me down the Thunderbolt rabbit hole that precipitated this blog post.
- CharlesW 5y ago
  In case a single-cable solution would work, I've been using an external TB NVMe RAID 0 in this enclosure for several months and it's worked flawlessly: https://www.amazon.com/gp/product/B08S5JPWR6 https://www.amazon.com/gp/product/B08S5JPWR6 (The only drawback is that the power supply is bizarrely-large, and the power connector feels a bit janky. In practice it hasn't been an issue.)
FullyFunctional 5y ago
Not OP, but my files are shared with many hosts. Having local files are a PITA to manage coherent state.

pr0nin 5y ago

What about separate Linux box laying around just to run docker? 8GB M1 Mac just don't have enough RAM for docker VM. Looks like local docker server should just work exactly same way as local docker while you at same location.

whalesalad 5y ago
Then you get into the problem of low latency directory sync between your box and that host. I’m still trying to crack that nut.