21 ms·

Ultra fast Thunderbolt NAS with Apple M1 and Linux

220 points by cyberge99 5y ago

cyberge99 5y ago
The maturity of Thunderbolt hardware and software support should have TB on a short list of high speed local connectivity, high-bandwidth multi-disk access options.
- CharlesW 5y ago
  Especially now that it's USB 4.
  - cyberge99 5y ago
    They're very similar, but there are differences. Most notably, Thunderbolt 4 guarantees a minimum speed[0] for compliance, whereas USB4 has a lower minimum. Also, I believe USB4 was designed for smaller files / blocks and Thunderbolt is designed for more raw output. [0] https://liliputing.com/2020/11/differences-between-thunderbolt-4-usb-4-thunderbolt-3-and-usb-3.html https://liliputing.com/2020/11/differences-between-thunderbo...
    - CharlesW 5y ago
      Thanks for the correction, and for citing such a great reference!
reacharavindh 5y ago
Not to take away anything from the fun of figuring out Thunderbolt on Linux, and such. But, I wonder if a NAS with a 40 Gbps or even 100Gbps NIC hooked up to a Thunderbolt to Ethernet adapter would achieve a better result without extra effort.
- minhazm 5y ago
  The fastest thunderbolt to ethernet adapter I could find is 25 Gbps and it's $1500. 10 Gbps ones are in the $200+ range as well.
  - osmarks 5y ago
    Buy an eGPU enclosure and put a NIC in that?
  - lostlogin 5y ago
    For 10Gbe it’s slightly cheaper if you go to SFP+, but then you need sfp cables. I went this route as they are also cooler. Then I got sick of Thunderbolt finicky config and went for cheap Mellanox PCI cards which are truely excellent and basically zero config. The dongle is QNAP QNA-T310G1S Thunderbolt 3 to 10GbE Adaptor, US$169 on Amazon.
  - reacharavindh 5y ago
    True. I assumed it was a simple adaptor for TB3 to 40Gbps Ethernet or atleast QSFP+ Since TB also advertises 40 Gbps, but looks like expensive niche. At $1500, you might as well spend time to figure out how to get Linux to work better with Thunderbolt card. For other devices in the network that do not impose Thunderbolt though, Melanox 40G NICs are surprisingly cheap.
    - genewitch 5y ago
      How cheap is "surprising" and who sells them? I wouldn't mind upgrading my link to the server barn and my house to an assuredly future proof 40gbit...
      - wmf 5y ago
        People talking about cheap 10G/40G are getting it used on eBay. They have a bad habit of comparing used prices against new prices without disclosing it. For example, if you're talking about new equipment, 40G is obsolete and you should buy 25G/50G instead. But used 40G is super-cheap because it's obsolete.
xiconfjs 5y ago
this blog uses webp as image format without a fallback option. Had to switch to Firefox because Safari wouldn't display the pictures.
- mpweiher 5y ago
  Odd. Displays fine on my Safari 14.1.2. Checked and the images do appear to be web for me as well. Current Big Sur release, not a Beta.
  - definitelyhuman 5y ago
    Safari 14.1.2 on latest Catalina doesn't render
    - michelb 5y ago
      Only safari on big sur has webp support.
      - jeroenhd 5y ago
        That's very strange. Is this a versioning problem (i.e. Catalina not being supported anymore?) or is there some kind of hardware restriction? From what I can find, Catalina should still be supported for more than a year from now, right? Edit: well, that's bizarre. Caniuse.com [0] lists the same restriction, WebP is only available from Big Sur onwards. And then people wonder why nobody likes developing for Safari. [0]: https://www.caniuse.com/?search=webp https://www.caniuse.com/?search=webp
        SahAssar 5y ago
        For media formats the support is often dependent on the OS. It's been that way forever (ask anyone deploying video in early html5 and getting it to work cross-os and cross-platform). I guess it's more expected with videos, but images are (usually) handled the same.
jeffbee 5y ago
Where by "ultra fast" it is meant that it is much slower than a directly-attached SSD that occupies the same port.
- reacharavindh 5y ago
  Local storage and Networked storage always have their places. Local solid state storage almost always gets you better performance because you negate the overhead of network. Unless, you’re talking fancy arrays connected to saturate 100G NICs. Local storage falls apart the moment you need that storage accessible from another machine…
  - jeffbee 5y ago
    This doesn't say anything about how such storage would be accessed by the mac and some other clients at the same time. It literally just runs iperf over tb. As a matter of architecture it leaves the question of whether it would be preferable to directly attach the storage to the Mac and export it to the rest of the network from there.
  - systemvoltage 5y ago
    Dumb question: Can you use Mac's shared folder for another machine to access files?
    - filoleg 5y ago
      Yep, it is possible. Here is the official support page on how to set up a shared folder that would be accessible by a any machine using SMB[0], and here is one for access by a macOS machine[1] (that is a bit simpler for setting up certain advanced config options). 0. https://support.apple.com/guide/mac-help/share-mac-files-with-windows-users-mchlp1657/mac https://support.apple.com/guide/mac-help/share-mac-files-wit... 1. https://support.apple.com/guide/mac-help/set-up-file-sharing-on-mac-mh17131/mac https://support.apple.com/guide/mac-help/set-up-file-sharing...
  - SilverRed 5y ago
    Networked storage also falls apart the moment you need anything better than basic file storage. So many filesystem features do not work networked, most network file mounts do not support any kind of modern security and the ones that do are often super slow. Everything has its place, but I wasted so much time trying to get a nas to work as storage for my home server.
whalesalad 5y ago
Just yesterday I was thinking how badly I want an obscenely fast local datastore for Docker. I am managing a handful of clients right now who are all using Docker containers for their microservice environments and my local machine runs out of disk daily. What I really want is lightning quick network storage, but I don't think it would be feasible to roll out 10g networking in my current rental home.
- devonkim 5y ago
  10gbe can be done over RJ45 with a direct cable connection and is something I've done before in a pinch between my personal NAS and a client machine (just be careful about your storage protocol because NFS has its pros and cons just like iSCSI and FCoE, and with macOS as a client you're likely in for a Bad Time like I did).
  - vitejose 5y ago
    Out of curiosity, what issues did you run into with NFS on macOS? I'm looking into building a NAS for a network of macOS, Linux, and BSD machines and figured NFS would be my first choice.
    - geerlingguy 5y ago
      I've had weird issues with some applications and NFS on macOS (notably, Final Cut Pro X and Photoshop). And I've also had weird issues with SMB and some apps (notably, Photoshop again!). But at least SMB has been pretty stable for years. NFS support has had some hiccups. A few 10.14 macOS releases broke NFSv3 and NFSv4 for me, but it was fixed in later releases.
    - 411111111111111 5y ago
      I'm not the author nor am I an apple user anymore, though i was at one point. NFS doesn't have a lot of metadata, so you loose a lot of functionality. iPhoto for example could cause major issues if you persisted its library on a nfs, potentially making the library irreparable.
    - 5y ago
pronoiac 5y ago
> In this post I discuss how you can upgrade a NAS Server by adding Thunderbolt 3 for lightning fast connectivity at 20 or 40Gbps. This is an interesting angle of attack, though I'll note that 10Gbps Ethernet was added as an option on the M1 Mac Minis.
- MayeulC 5y ago
  here's an idea: use link aggregation yo make it 10 G faster, then. I wonder what's the simplest option. could it be made to work with a virtual interface, like wireguard?
- smaccona 5y ago
  10Gbps was already an option on the Intel Mac Minis. My guess is that Apple envisioned a use case for it to be a file server or similar.
boynamedsue 5y ago
I just wish Apple would offer a native iSCSI initiator.
- CharlesW 5y ago
  It sure would be convenient if they Sherlocked the developers of commercial solutions. The globalSAN initiator's been working well for me, FWIW.
  - mbreese 5y ago
    >globalSAN initiator's been working well for me It doesn't look like it's going to be supported for M1 Macs or is really all that compatible with Big Sur. From their site, it seems that you have to jump through a few hoops to add a kernel extension to get it to work[1]. Sounds like a good opportunity for Apple to roll something into the OS. [1] https://support.studionetworksolutions.com/hc/en-us/articles/360052102272-globalSAN-and-macOS-11-Big-Sur- https://support.studionetworksolutions.com/hc/en-us/articles...
insaneirish 5y ago
There's something weird here: root@nas-04:~ # ifconfig thunderbolt0 10.1.1.2 root@nas-04:~ # ifconfig thunderbolt0 thunderbolt0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.10.2 netmask 255.0.0.0 broadcast 10.255.255.255 The IP on the interface is different than the one that it was theoretically configured with. My suspicion is that perhaps the IP used was changed at some point and this is just an editing slip. > Now that the interface has an IP, we have to add a static host route entry for it to communicate: If both sides are configured with an appropriate netmask on the same subnet, there's no need for a specific host route.
- cyberge99 5y ago
  Author here: You’re correct, it’s an editing error. There’s another where I do ifconfig on the mini, but I pasted from the nas. I’ll correct the post.
FullyFunctional 5y ago
Funny I went down this rathole very shortly after discovering the "Thunderbolt networking" change in the Linux kernel. My hope was the use the Linux box (with 10 GbE) as a bridge/switch to give my Mac cheap 10 GbE access. While point-to-point did hit more or less the expected speed, bridging was horribly slow. Ended up just getting an Sonnet Solo10G SFP+ adapter (which is great).
- 21com 5y ago
  > bridging was horribly slow. Can you quantify that? (I'm wondering if it might be good-enough for my needs.)
  - FullyFunctional 5y ago
    I'm afraid it was 2+ years ago so I don't remember, but it must have been slower than the comparable speed from the gigabit wired connection. FWIW, I was so happy with the Sonnet (not as hot as some alternatives I've used) that I bought two more. However, unless you are using fiber or for other reasons preferring SFP+, going with the grain and getting 10Gbase-T is probably easier. Getting back to the OP, this could be interesting for me if and only if I can get fiber based Thunderbolt-4, but it seems to not yet be available.
    - wmf 5y ago
      I'm pretty sure Thunderbolt 3 cables will work: https://www.amazon.com/optical-thunderbolt-cables-Corning/s?k=optical+thunderbolt+cables+by+Corning https://www.amazon.com/optical-thunderbolt-cables-Corning/s?...
      - FullyFunctional 5y ago
        Thanks. $528 for 30 meter TB3 compared to $35 for 30 meter OM4 and a pair of $10 10G SFP+ modules (and the same OM4 will carry 25GbE just fine). Seems like it's not really cost effective.
Teknoman117 5y ago
This is cool, but my main takeaway is that you can get around needing the thunderbolt enablement header by shorting two pins together and use it on any motherboard? What the hell Intel...
- unicornfinder 5y ago
  If memory serves the main problem is some Thunderbolt cards don't ship with the necessary firmware on them, so would need to be used at least once in a supported motherboard before just used in any system.
  - zhengyi13 5y ago
    That seems even more WTF, and puts me in mind of the old Winmodem setup. Not precisely analogous, but boy it seems weird.
    - genewitch 5y ago
      There's still benefits today to using actual modems instead of glorified sound cards or literal sound cards. I have a pactor modem for ham radio use, and it can do several modes other than pactor, like rtty. Compared to using something like fldigi, if I can hear the other side and they can hear me, the data will 100% go through correctly on the physical pactor modem. There's weird jank even when using the basic RF mail software, which can control a variety of modems, or just use your sound card, too. Every session is longer when using software based modems, for example. I'm aware that the pactor modem itself is probably a software defined device like an FPGA, but it is much better than the software on the PC - and I can use it with an anemic laptop from 2007 with no issues at all, since it just sends text back and forth over a serial port - via USB.
- rsync 5y ago
  "What the hell Intel... " I don't understand any of it ... If I wanted to add a LPT parallel port to my motherboard, I would find an appropriate add-in card with the physical port and plug it into the PCI bus and locate an appropriate driver. Thunderbolt is just another kind of port, right ? If you buy the add-in card to get the physical port and you have a driver to drive the card ... what in the world is "enablement" and why would it be required ?
conradev 5y ago
I was actually looking for something like this. I have one computer that I want to back up to both local hard drives and to the cloud. A NAS is great, because you can back up to it and have the backup stored on local hard drives, but it also has a CPU and it can be responsible for uploading the data to the cloud. This is better to do from a dedicated device, especially if you are uploading large amounts of data with limited bandwidth, which I am. The constraint is that locally backing up to a NAS is slow, even though it is sitting next to my computer. Instead of buying into 10GbE networking, it’d be nice to just use a Thunderbolt cable.
- wil421 5y ago
  Running FreeNAS on a 1GbE network and I wouldn’t call it slow at all. The initial back is much much faster than my 1TB personal media library I sometimes backup to OneDrive. I think I have 3TB of video from Usenet that is downloaded onto a windows box, uses a 10 year old intel as an unpack scratch pad, and then is stored on the NAS box. Sure the initial library transfer is slow but if you can use a wire it’s not that slow. Certainly not Comcast 25Mb upload slow. Agree about 10Gb devices being expensive especially if you’re invested in the UniFi ecosystem. I’m tempted to do a direct 10Gb SFP+ connection using used intel or Mellanox 10Gb NICs but I haven’t pulled the trigger. 10Gb Thunderbolt NICs are too expensive and give off questionable amounts of heat. SFP+ is the way to go right now and gives you options. Fiber, DAC (copper), or Ethernet.
  - Fire-Dragon-DoL 5y ago
    I have a freenas too. Uploading to it gives me 100 mb write speed, which is definitely enough for most of my needs.
    - wil421 5y ago
      Yea I was debating on using SAS drives instead of SATA but cost was too much. Bought WD Red drives instead. Hopefully I won’t have to resilver.
  - lizknope 5y ago
brian_herman 5y ago
Does anyone know if I installed ubuntu on a m1 apple mac would I get pci passthrough to gpu. That would be no right because that isnt possible due to the hardware. I was going to do the ubuntu conversion myself but I am running a developer beta and the commands don't work and if I wait longer it might just be in the next LTS or release of ubuntu by the time Monterey is stable.
lxe 5y ago
> As NVMe disks and interfaces have gotten faster I’ve found myself saturating a 10Gb ethernet network. I love this race between network speeds, drive speeds, and RAM speeds. The classic "net vs disk vs ram vs cache vs register" interview question is much trickier these days.
- SilverRed 5y ago
  I recently moved to somewhere with a real fiber internet connection and was impressed that I could get up to a gigabit speeds but when I tested it I found my wifi network could only handle about 100Mbit and sometimes it was dropping to 50. First time I have ever had wifi actually be the limiting factor. Thankfully my apartment has wired Ethernet to most rooms so my steam downloads are flying along now.
  - genewitch 5y ago
    I think most people will find that they'll get <100mbit over wifi. I had to design, buy, and implement 802.11ac stuff to get close to gigabit. And then wifi6/802.11ax came out and nothing wanted to play with Linux and wifi6. So I got a fiber media converter and put a switch on my desk backhauled over fiber to the "house switch" which is in turn backhauled to the server barn switch via fiber. Doing this prevents lightning from blowing up my wifi access points and NAS boxes, which has happened before, necessitating the need for fast WiFi in the first place.
solarkraft 5y ago
This is cool, I’ve been thinking about something like it. However I’m still slightly disappointed because I expected an article describing an M1 Mac Mini as a NAS, running Linux (this dream probably isn’t technically there just yet, as Thunderbolt [and network?] drivers are so far still missing from the Asahi project).
ballenf 5y ago
I wonder if there will be "reality" tv shows about data hoarders one day. Probably have to wait for data to be easily visualized with VR or AR. In fact, I really love the idea that I could have to walk to a particular room in order to view a particular set of old photos. Wearing some AR glasses, of course. And that I could grant view access to visitors. Would suck if the photos accidentally glitched through the floor and I had to dig down through a foot of concrete to get them back though.
johnwalkr 5y ago
This year I set up 10Gbps networking in my home lab since I am lucky enough to have 10Gbps internet and it didn’t make sense to pay for that but not have it for all my machines. Or at least I tried to set it up. 10G pcie cards are now cheap enough, but routers are not. Thunderbolt adapters are also expensive, bulky and use a ton of power[1]. I’m keeping my eye open for a surplus router and cheaper adapters but in the meantime I am only using a 2 port card in my main machine to serve at 10Gbps to one other machine. I wonder if I could also serve my m1 MacBook using thunderbolt 3. [1] In my work life I also tried to use Ethernet in some embedded projects. It turns out Ethernet is just power hungry in general. Even 100Mbps Ethernet uses about 0.5W per port. There are standards for lower power but I’ve yet to see anyone use them.
- bpye 5y ago
  Have you considered using an x86 machine with dual 10G NICs as your router, and then all you’d need is a 10G switch which may be cheaper than a router. I’ve considered the approach but haven’t undertaken it yet.

220 points by cyberge99 5y ago

The maturity of Thunderbolt hardware and software support should have TB on a short list of high speed local connectivity, high-bandwidth multi-disk access options.

CharlesW 5y ago
Especially now that it's USB 4.
- cyberge99 5y ago
  They're very similar, but there are differences. Most notably, Thunderbolt 4 guarantees a minimum speed[0] for compliance, whereas USB4 has a lower minimum. Also, I believe USB4 was designed for smaller files / blocks and Thunderbolt is designed for more raw output. [0] https://liliputing.com/2020/11/differences-between-thunderbolt-4-usb-4-thunderbolt-3-and-usb-3.html https://liliputing.com/2020/11/differences-between-thunderbo...
  - CharlesW 5y ago
    Thanks for the correction, and for citing such a great reference!

reacharavindh 5y ago

Not to take away anything from the fun of figuring out Thunderbolt on Linux, and such. But, I wonder if a NAS with a 40 Gbps or even 100Gbps NIC hooked up to a Thunderbolt to Ethernet adapter would achieve a better result without extra effort.

minhazm 5y ago
The fastest thunderbolt to ethernet adapter I could find is 25 Gbps and it's $1500. 10 Gbps ones are in the $200+ range as well.
- osmarks 5y ago
  Buy an eGPU enclosure and put a NIC in that?
- lostlogin 5y ago
  For 10Gbe it’s slightly cheaper if you go to SFP+, but then you need sfp cables. I went this route as they are also cooler. Then I got sick of Thunderbolt finicky config and went for cheap Mellanox PCI cards which are truely excellent and basically zero config. The dongle is QNAP QNA-T310G1S Thunderbolt 3 to 10GbE Adaptor, US$169 on Amazon.
- reacharavindh 5y ago
  True. I assumed it was a simple adaptor for TB3 to 40Gbps Ethernet or atleast QSFP+ Since TB also advertises 40 Gbps, but looks like expensive niche. At $1500, you might as well spend time to figure out how to get Linux to work better with Thunderbolt card. For other devices in the network that do not impose Thunderbolt though, Melanox 40G NICs are surprisingly cheap.
  - genewitch 5y ago
    How cheap is "surprising" and who sells them? I wouldn't mind upgrading my link to the server barn and my house to an assuredly future proof 40gbit...
    - wmf 5y ago
      People talking about cheap 10G/40G are getting it used on eBay. They have a bad habit of comparing used prices against new prices without disclosing it. For example, if you're talking about new equipment, 40G is obsolete and you should buy 25G/50G instead. But used 40G is super-cheap because it's obsolete.

xiconfjs 5y ago

this blog uses webp as image format without a fallback option. Had to switch to Firefox because Safari wouldn't display the pictures.

mpweiher 5y ago
Odd. Displays fine on my Safari 14.1.2. Checked and the images do appear to be web for me as well. Current Big Sur release, not a Beta.
- definitelyhuman 5y ago
  Safari 14.1.2 on latest Catalina doesn't render
  - michelb 5y ago
    Only safari on big sur has webp support.
    - jeroenhd 5y ago
      That's very strange. Is this a versioning problem (i.e. Catalina not being supported anymore?) or is there some kind of hardware restriction? From what I can find, Catalina should still be supported for more than a year from now, right? Edit: well, that's bizarre. Caniuse.com [0] lists the same restriction, WebP is only available from Big Sur onwards. And then people wonder why nobody likes developing for Safari. [0]: https://www.caniuse.com/?search=webp https://www.caniuse.com/?search=webp
      - SahAssar 5y ago
        For media formats the support is often dependent on the OS. It's been that way forever (ask anyone deploying video in early html5 and getting it to work cross-os and cross-platform). I guess it's more expected with videos, but images are (usually) handled the same.

jeffbee 5y ago

Where by "ultra fast" it is meant that it is much slower than a directly-attached SSD that occupies the same port.

reacharavindh 5y ago
Local storage and Networked storage always have their places. Local solid state storage almost always gets you better performance because you negate the overhead of network. Unless, you’re talking fancy arrays connected to saturate 100G NICs. Local storage falls apart the moment you need that storage accessible from another machine…
- jeffbee 5y ago
  This doesn't say anything about how such storage would be accessed by the mac and some other clients at the same time. It literally just runs iperf over tb. As a matter of architecture it leaves the question of whether it would be preferable to directly attach the storage to the Mac and export it to the rest of the network from there.
- systemvoltage 5y ago
  Dumb question: Can you use Mac's shared folder for another machine to access files?
  - filoleg 5y ago
    Yep, it is possible. Here is the official support page on how to set up a shared folder that would be accessible by a any machine using SMB[0], and here is one for access by a macOS machine[1] (that is a bit simpler for setting up certain advanced config options). 0. https://support.apple.com/guide/mac-help/share-mac-files-with-windows-users-mchlp1657/mac https://support.apple.com/guide/mac-help/share-mac-files-wit... 1. https://support.apple.com/guide/mac-help/set-up-file-sharing-on-mac-mh17131/mac https://support.apple.com/guide/mac-help/set-up-file-sharing...
- SilverRed 5y ago
  Networked storage also falls apart the moment you need anything better than basic file storage. So many filesystem features do not work networked, most network file mounts do not support any kind of modern security and the ones that do are often super slow. Everything has its place, but I wasted so much time trying to get a nas to work as storage for my home server.

whalesalad 5y ago

Just yesterday I was thinking how badly I want an obscenely fast local datastore for Docker. I am managing a handful of clients right now who are all using Docker containers for their microservice environments and my local machine runs out of disk daily. What I really want is lightning quick network storage, but I don't think it would be feasible to roll out 10g networking in my current rental home.

devonkim 5y ago
10gbe can be done over RJ45 with a direct cable connection and is something I've done before in a pinch between my personal NAS and a client machine (just be careful about your storage protocol because NFS has its pros and cons just like iSCSI and FCoE, and with macOS as a client you're likely in for a Bad Time like I did).
- vitejose 5y ago
  Out of curiosity, what issues did you run into with NFS on macOS? I'm looking into building a NAS for a network of macOS, Linux, and BSD machines and figured NFS would be my first choice.
  - geerlingguy 5y ago
    I've had weird issues with some applications and NFS on macOS (notably, Final Cut Pro X and Photoshop). And I've also had weird issues with SMB and some apps (notably, Photoshop again!). But at least SMB has been pretty stable for years. NFS support has had some hiccups. A few 10.14 macOS releases broke NFSv3 and NFSv4 for me, but it was fixed in later releases.
  - 411111111111111 5y ago
    I'm not the author nor am I an apple user anymore, though i was at one point. NFS doesn't have a lot of metadata, so you loose a lot of functionality. iPhoto for example could cause major issues if you persisted its library on a nfs, potentially making the library irreparable.
  - 5y ago

pronoiac 5y ago

> In this post I discuss how you can upgrade a NAS Server by adding Thunderbolt 3 for lightning fast connectivity at 20 or 40Gbps. This is an interesting angle of attack, though I'll note that 10Gbps Ethernet was added as an option on the M1 Mac Minis.

MayeulC 5y ago
here's an idea: use link aggregation yo make it 10 G faster, then. I wonder what's the simplest option. could it be made to work with a virtual interface, like wireguard?
smaccona 5y ago
10Gbps was already an option on the Intel Mac Minis. My guess is that Apple envisioned a use case for it to be a file server or similar.

boynamedsue 5y ago

I just wish Apple would offer a native iSCSI initiator.

CharlesW 5y ago
It sure would be convenient if they Sherlocked the developers of commercial solutions. The globalSAN initiator's been working well for me, FWIW.
- mbreese 5y ago
  >globalSAN initiator's been working well for me It doesn't look like it's going to be supported for M1 Macs or is really all that compatible with Big Sur. From their site, it seems that you have to jump through a few hoops to add a kernel extension to get it to work[1]. Sounds like a good opportunity for Apple to roll something into the OS. [1] https://support.studionetworksolutions.com/hc/en-us/articles/360052102272-globalSAN-and-macOS-11-Big-Sur- https://support.studionetworksolutions.com/hc/en-us/articles...

insaneirish 5y ago

There's something weird here: root@nas-04:~ # ifconfig thunderbolt0 10.1.1.2 root@nas-04:~ # ifconfig thunderbolt0 thunderbolt0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.10.2 netmask 255.0.0.0 broadcast 10.255.255.255 The IP on the interface is different than the one that it was theoretically configured with. My suspicion is that perhaps the IP used was changed at some point and this is just an editing slip. > Now that the interface has an IP, we have to add a static host route entry for it to communicate: If both sides are configured with an appropriate netmask on the same subnet, there's no need for a specific host route.

cyberge99 5y ago
Author here: You’re correct, it’s an editing error. There’s another where I do ifconfig on the mini, but I pasted from the nas. I’ll correct the post.

FullyFunctional 5y ago

Funny I went down this rathole very shortly after discovering the "Thunderbolt networking" change in the Linux kernel. My hope was the use the Linux box (with 10 GbE) as a bridge/switch to give my Mac cheap 10 GbE access. While point-to-point did hit more or less the expected speed, bridging was horribly slow. Ended up just getting an Sonnet Solo10G SFP+ adapter (which is great).

21com 5y ago
> bridging was horribly slow. Can you quantify that? (I'm wondering if it might be good-enough for my needs.)
- FullyFunctional 5y ago
  I'm afraid it was 2+ years ago so I don't remember, but it must have been slower than the comparable speed from the gigabit wired connection. FWIW, I was so happy with the Sonnet (not as hot as some alternatives I've used) that I bought two more. However, unless you are using fiber or for other reasons preferring SFP+, going with the grain and getting 10Gbase-T is probably easier. Getting back to the OP, this could be interesting for me if and only if I can get fiber based Thunderbolt-4, but it seems to not yet be available.
  - wmf 5y ago
    I'm pretty sure Thunderbolt 3 cables will work: https://www.amazon.com/optical-thunderbolt-cables-Corning/s?k=optical+thunderbolt+cables+by+Corning https://www.amazon.com/optical-thunderbolt-cables-Corning/s?...
    - FullyFunctional 5y ago
      Thanks. $528 for 30 meter TB3 compared to $35 for 30 meter OM4 and a pair of $10 10G SFP+ modules (and the same OM4 will carry 25GbE just fine). Seems like it's not really cost effective.

Teknoman117 5y ago

This is cool, but my main takeaway is that you can get around needing the thunderbolt enablement header by shorting two pins together and use it on any motherboard? What the hell Intel...

unicornfinder 5y ago
If memory serves the main problem is some Thunderbolt cards don't ship with the necessary firmware on them, so would need to be used at least once in a supported motherboard before just used in any system.
- zhengyi13 5y ago
  That seems even more WTF, and puts me in mind of the old Winmodem setup. Not precisely analogous, but boy it seems weird.
  - genewitch 5y ago
    There's still benefits today to using actual modems instead of glorified sound cards or literal sound cards. I have a pactor modem for ham radio use, and it can do several modes other than pactor, like rtty. Compared to using something like fldigi, if I can hear the other side and they can hear me, the data will 100% go through correctly on the physical pactor modem. There's weird jank even when using the basic RF mail software, which can control a variety of modems, or just use your sound card, too. Every session is longer when using software based modems, for example. I'm aware that the pactor modem itself is probably a software defined device like an FPGA, but it is much better than the software on the PC - and I can use it with an anemic laptop from 2007 with no issues at all, since it just sends text back and forth over a serial port - via USB.
rsync 5y ago
"What the hell Intel... " I don't understand any of it ... If I wanted to add a LPT parallel port to my motherboard, I would find an appropriate add-in card with the physical port and plug it into the PCI bus and locate an appropriate driver. Thunderbolt is just another kind of port, right ? If you buy the add-in card to get the physical port and you have a driver to drive the card ... what in the world is "enablement" and why would it be required ?

conradev 5y ago

I was actually looking for something like this. I have one computer that I want to back up to both local hard drives and to the cloud. A NAS is great, because you can back up to it and have the backup stored on local hard drives, but it also has a CPU and it can be responsible for uploading the data to the cloud. This is better to do from a dedicated device, especially if you are uploading large amounts of data with limited bandwidth, which I am. The constraint is that locally backing up to a NAS is slow, even though it is sitting next to my computer. Instead of buying into 10GbE networking, it’d be nice to just use a Thunderbolt cable.

wil421 5y ago
Running FreeNAS on a 1GbE network and I wouldn’t call it slow at all. The initial back is much much faster than my 1TB personal media library I sometimes backup to OneDrive. I think I have 3TB of video from Usenet that is downloaded onto a windows box, uses a 10 year old intel as an unpack scratch pad, and then is stored on the NAS box. Sure the initial library transfer is slow but if you can use a wire it’s not that slow. Certainly not Comcast 25Mb upload slow. Agree about 10Gb devices being expensive especially if you’re invested in the UniFi ecosystem. I’m tempted to do a direct 10Gb SFP+ connection using used intel or Mellanox 10Gb NICs but I haven’t pulled the trigger. 10Gb Thunderbolt NICs are too expensive and give off questionable amounts of heat. SFP+ is the way to go right now and gives you options. Fiber, DAC (copper), or Ethernet.
- Fire-Dragon-DoL 5y ago
  I have a freenas too. Uploading to it gives me 100 mb write speed, which is definitely enough for most of my needs.
  - wil421 5y ago
    Yea I was debating on using SAS drives instead of SATA but cost was too much. Bought WD Red drives instead. Hopefully I won’t have to resilver.
- lizknope 5y ago

brian_herman 5y ago

Does anyone know if I installed ubuntu on a m1 apple mac would I get pci passthrough to gpu. That would be no right because that isnt possible due to the hardware. I was going to do the ubuntu conversion myself but I am running a developer beta and the commands don't work and if I wait longer it might just be in the next LTS or release of ubuntu by the time Monterey is stable.

lxe 5y ago

> As NVMe disks and interfaces have gotten faster I’ve found myself saturating a 10Gb ethernet network. I love this race between network speeds, drive speeds, and RAM speeds. The classic "net vs disk vs ram vs cache vs register" interview question is much trickier these days.

SilverRed 5y ago
I recently moved to somewhere with a real fiber internet connection and was impressed that I could get up to a gigabit speeds but when I tested it I found my wifi network could only handle about 100Mbit and sometimes it was dropping to 50. First time I have ever had wifi actually be the limiting factor. Thankfully my apartment has wired Ethernet to most rooms so my steam downloads are flying along now.
- genewitch 5y ago
  I think most people will find that they'll get <100mbit over wifi. I had to design, buy, and implement 802.11ac stuff to get close to gigabit. And then wifi6/802.11ax came out and nothing wanted to play with Linux and wifi6. So I got a fiber media converter and put a switch on my desk backhauled over fiber to the "house switch" which is in turn backhauled to the server barn switch via fiber. Doing this prevents lightning from blowing up my wifi access points and NAS boxes, which has happened before, necessitating the need for fast WiFi in the first place.

solarkraft 5y ago

This is cool, I’ve been thinking about something like it. However I’m still slightly disappointed because I expected an article describing an M1 Mac Mini as a NAS, running Linux (this dream probably isn’t technically there just yet, as Thunderbolt [and network?] drivers are so far still missing from the Asahi project).

ballenf 5y ago

I wonder if there will be "reality" tv shows about data hoarders one day. Probably have to wait for data to be easily visualized with VR or AR. In fact, I really love the idea that I could have to walk to a particular room in order to view a particular set of old photos. Wearing some AR glasses, of course. And that I could grant view access to visitors. Would suck if the photos accidentally glitched through the floor and I had to dig down through a foot of concrete to get them back though.

johnwalkr 5y ago

This year I set up 10Gbps networking in my home lab since I am lucky enough to have 10Gbps internet and it didn’t make sense to pay for that but not have it for all my machines. Or at least I tried to set it up. 10G pcie cards are now cheap enough, but routers are not. Thunderbolt adapters are also expensive, bulky and use a ton of power[1]. I’m keeping my eye open for a surplus router and cheaper adapters but in the meantime I am only using a 2 port card in my main machine to serve at 10Gbps to one other machine. I wonder if I could also serve my m1 MacBook using thunderbolt 3. [1] In my work life I also tried to use Ethernet in some embedded projects. It turns out Ethernet is just power hungry in general. Even 100Mbps Ethernet uses about 0.5W per port. There are standards for lower power but I’ve yet to see anyone use them.

bpye 5y ago
Have you considered using an x86 machine with dual 10G NICs as your router, and then all you’d need is a 10G switch which may be cheaper than a router. I’ve considered the approach but haven’t undertaken it yet.