4 ms·
Wholeheartedly agree. After my cloud storage costs exploded (mostly S3 egress over to Hetzner/OVH), I noticed that renting a 1GBit/s fiber connection to my offi
by fxtentacle 5y ago
Wholeheartedly agree. After my cloud storage costs exploded (mostly S3 egress over to Hetzner/OVH), I noticed that renting a 1GBit/s fiber connection to my office is actually quite affordable at $80 per month (in northern Germany).
Of course, it's not globally distributed and there is no fail-safe, but for the work we're doing, that is no issue. If we employees are offline, it doesn't matter if our tools are offline, too. And now that all AI storage is local anyway, building a GPU compute node is easy. I'm still waiting for 3090 prices to drop further, though, in contrast to the article. But I also went with Ryzon 5950 and Linux. I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit. My workstation now has 1010MB/s 4k random write on the network filesystem. (We used SAMBA 3.1 and CIFS mounts)
I also grabbed the python/ubuntu package lists off Google Colab and created my own Docker to imitate it, and now data processing and AI training is fast (I always get the good GPU, no luck involved) and dirt cheap. Originally the idea was to run it on OVH, but I'm now also running it locally.
https://github.com/fxtentacle/ovh-colab-sagemaker-compatibility-mode/blob/master/Dockerfile https://github.com/fxtentacle/ovh-colab-sagemaker-compatibil...
- GordonS 5y ago> I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit. Wow, that is surprising, maybe it's time I started upgrading my LAN...
- fxtentacle 5y agoI use Intel 82599ES SFP+ and a TL-SX3008F router. But let me warn you: Things are affordable, but NOT consumer-friendly. I needed to study the 500 page PDF manual and do basic link configuration through telnet via USB before I could connect to the router via Ethernet and use its web-GUI to finish the setup.
- sofixa 5y agoHow's the routing performance on that switch? I'm really struggling finding a device to fit my needs ( routing, 6-8 ports, Ethernet, at least two 2.5/5/10G).
- fxtentacle 5y ago8x 10G SFP+ with a 160GB/s full duplex backplane. So far, it has passed all of my testing with flying colors.
- sofixa 5y agoIsn't that only for switching? Do you use it to route, cross-vlan or similar?
- v-yadli 5y agoActually if you have cat5 or cat6 base-T setup, you can get second hand dual port 10gbe pcie cards for 30 dollars each(!) 10GbE baseT switches are more expensive than fiber equivalent though.
- fxtentacle 5y agoIn my case, cat6 cables and the added price of a 10G RJ-45 switch would have been more expensive than a 10G SPF+ switch and some twinax & fiber cables. Amazon has finished SPF-to-SPF assemblies for €10=$12. And for TP-Link, RJ-45 is like 1.5x the price of SPF+ equipment. Also, RJ-45 has a fixed minimum latency, due to it needing to support backwards-compatibility with 1G 100M etc. That means you need much larger send/transmit buffers to saturate the link as ping goes up. Especially if you need multiple hops, fiber is just faster. In my tests, 0.2ms vs 3ms roundtrip time for 10G SPF vs. 10G RJ-45.
- hansel_der 5y ago> Especially if you need multiple hops, fiber is just faster. In my tests, 0.2ms vs 3ms roundtrip time for 10G SPF vs. 10G RJ-45 ime if your unloaded latency on copper ethernet exceeds 0.2ms per hop there is some form of powersaving involved
- fxtentacle 5y agoIf I remember correctly, the IEEE 802.3an line coding overhead is around 2.6ms for each time you switch between SPF+ and RJ-45. That's in line with my measurements. SPF+ switch to OM3 fiber to SPF+ PCIe card => 0.2ms SPF+ switch to RJ-45 cable to RJ-45 PCIe card => around 3ms
- hansel_der 5y agopretty wild, considering my phone averages at 2.2ms ROUND trip time when pinging the router.
- v-yadli 5y ago