4 ms·
The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t suppo
by trebligdivad 1y ago
The networking stuff seems....odd.
'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.'
Where does the switch choice come into whether you DHCP? Wth would you want public IPs.
- giancarlostoro 1y ago> Wth would you want public IPs. So anyone can download 30 PB of data with ease of course.
- buzer 1y ago> Wth would you want public IPs. Possibly to avoid needing NAT (or VPN) gateway that can handle 100Gbps.
- bombcar 1y agoI don't know what they're doing, but Mikrotik can perhaps route that → https://mikrotik.com/product/ccr2216_1g_12xs_2xq#fndtn-testresults https://mikrotik.com/product/ccr2216_1g_12xs_2xq#fndtn-testr... and is about the cost of their used thing. And I think this would be a banger for IPv6 if they really "need" public IPs.
- dustywusty 1y agoExactly what I came in to say, CCR2216 can do this for < $2k, and does it well.
- xp84 1y agoNo DHCP doesn't mean public IPs nor impact the need for NAT, it just means the hosts have to be explicitly configured with IP addresses, default gateways if they need egress, and DNS. Those IPs you end up assigning manually could be private ones or routable ones. If private, authorized traffic could be bridged onto the network by anything, such as a random computer with 2 NICs, one of which is connected eventually to the Internet and one of which is on the local network. If public, a firewall can control access just as well as using NAT can.
- buzer 1y agoI know, I was specifically answering the question of "why the hell would you want public IPs". I don't know why their network setup wouldn't support DHCP, that's extremely common especially in "enterprise" switches via DHCP forwarding.
- xp84 1y agoOk then yes I agree with you. That was weird
- pclmulqdq 1y agoThey didn't seem to want to use a router. Purpose-built 100 Gbps routers are a bit expensive, but you can also turn a computer into one.
- flumpcakes 1y agoMany switches are L3 capable, making them in effect a router. Considering their internet lines appear to be hooked up to their 100 Gbps switch, I'd guess this is one of the L3 ones.
- mystifyingpoi 1y agoIt really feels like they wanted 30 PB of storage accessible over HTTP and literally nothing else. No redundancy, no NAT, dead simple nginx config + some code to track where to find which file on the filesystem. I like that.
- matt-p 1y agoThis was not written by a network person, quite clearly. Hopefully it's just a misunderstanding, otherwise they do need someone with literally any clue about networks.
- g413n 1y agoyeah misunderstanding we'll update the post-- separately it's true that we aren't network specialists and the network wrangling was prob disproportionately hard for us/ shouldn't have taken so long.
- trebligdivad 1y agoI assume your actual training is being done somewhere else? Did you try getting colocation space in the same datacentre as somewhere with the compute - it would have reduced your internet costs even further.
- g413n 1y agoyeah the cost calculus is very different for gpus, it absolutely makes sense for us to be using cloud there. also hardly any datacenters can support the power density, esp in downtown sf
- trebligdivad 1y agoYeh; one other thing - you list a separate management network as an optional - it's not optional! Under no circumstance must you expose the managemnt IPs of switches or the servers to the internet; they are, on average, about as secure as a drunk politician. Use a separate management net, make sure it's only securly accessed.
- Symbiote 1y agoI understood that it's optional because they can walk down the road to the data center instead. They mention plugging monitors in several times. I think I've only done that once in the last couple of years, when a firmware upgrade failed and reset the management interface IP.
- XorNot 1y agoI mean generally above a certain size of deployment DHCP is much more trouble then it's worth. DHCP is really only worth it when your hosts are truly dynamic (i.e. not controlled by you). Otherwise it's a lot easier to handle IP allocation as part of the asset lifecycle process. Heck even my house IoT network is all static IPs because at the small scale it's much more robust to not depend on my home router for address assignment - replacing a smart bulb is a big enough event, so DHCP is solely for bootstrapping in that case. At the enterprise level unpacking a server and recording the asset IDs etc is the time to assign IP addresses.
- Symbiote 1y agoI have static, public IPs across 80 or so servers. It gets set approximately once when the server's automated Ubuntu installation runs, and I never think about it. > Where does the switch choice come into whether you DHCP? Perhaps from home routers which include I've. > Wth would you want public IPs. Why wouldn't you? They have a firewall.