5 ms·

Creating an autonomous system for fun and profit (2017)

87 points by bsilvereagle 3y ago

phirephly 3y ago
This article is also essentially available as a podcast. https://oxide.computer/podcasts/on-the-metal/kenneth-finnegan https://oxide.computer/podcasts/on-the-metal/kenneth-finnega...
dang 3y ago
Discussed at the time: Creating an Autonomous System for Fun and Profit - https://news.ycombinator.com/item?id=15727115 https://news.ycombinator.com/item?id=15727115 - Nov 2017 (16 comments) (p.s. reposts are fine after a year or so; links to past threads are just to satisfy extra-curious readers)
cantaloupe 3y ago
The article mentions that the Cisco router used is limited to a million addresses, which would be exceeded in “2-3 years.” Looks like the author got at least double the life out of the router, because the internet is just approaching one million BGP entries now! https://bgp.potaroo.net/bgprpts/rva-index.html https://bgp.potaroo.net/bgprpts/rva-index.html
- phirephly 3y ago
  I only got a few years out of it. I'm running an Arista 7280SR-48C8 now. The problem is that the million tcam entries are split between IPv4 and IPv6, so I really ran out of space.
  - cantaloupe 3y ago
    Bummer! Didn’t realize the graph was not for not IPv4 and IPv6. Have you done anything fun with the AS or had an opportunity to say “Luckily, I do have an AS!” in a time of need?
    - phirephly 3y ago
      I started an Internet Exchange Point adjacent to it, use it to host mirror.fcix.net, and got a second ASN to build the anycast ns-global.zone service
      - rlaager 3y ago
        Is the map on https://ns-global.zone/ https://ns-global.zone/ actually the MicroMirror map? There seem to be a lot more dots on the map than in the traffic graph. If that map is accurate, I’m curious where the Minneapolis-ish node is (network wise). I’m not seeing 23.128.97.0/24 via MICE.
      - phirephly 3y ago
        The map is a latency heat map. I should probably replace it with a POP map
davisr 3y ago
Don't use Cisco equipment; they put (quite sloppy) backdoors in their products. Absolutely zero trust with them. Snowden: The NSA planted backdoors in Cisco products --- https://www.infoworld.com/article/2608141/snowden--the-nsa-planted-backdoors-in-cisco-products.html https://www.infoworld.com/article/2608141/snowden--the-nsa-p... Backdoors Keep Appearing In Cisco's Routers --- https://www.tomshardware.com/news/cisco-backdoor-hardcoded-accounts-software,37480.html https://www.tomshardware.com/news/cisco-backdoor-hardcoded-a... Sinister secret backdoor found in networking gear perfect for government espionage: The Chinese are – oh no, wait, it's Cisco again --- https://www.theregister.com/2019/05/02/cisco_vulnerabilities/ https://www.theregister.com/2019/05/02/cisco_vulnerabilities...
- sneak 3y ago
  The idea that the network is trustworthy is outdated. Assume all network connections, including those through your own equipment, are under surveillance. You shouldn't be trusting your network equipment to be not-backdoored in the first place. (Never mind that the router on the other end of the cable isn't yours and is untrusted also, whether Cisco or not.) CPUs have AES key schedule accelerator instructions for a reason now. Encrypt and authenticate your traffic.
  - davisr 3y ago
    I respectfully disagree. Are you suggesting everyone just rolls over and forgets about making decisions to the best of their ability to ensure security? The switches on the outside of your network do not matter. The traffic doesn't matter. If you lose trust in your own switching equipment then it's all over. Management network? Compromised. Segmented traffic? Compromised. IPMI/BMC interfaces? Compromised. Anybody else's malformed traffic could breach your defense, and breach the very sanctity of the traffic your network is spitting out. It doesn't matter if your computer encrypts its traffic because a breached switch can just silence it. A company selling switches/routers/firewalls should _not_ have these liabilities, and as these liabilities are known, nobody should buy Cisco equipment, ever. Buy the equipment that you know is the safest. Don't just give up and roll over!
hamandcheese 3y ago
> and its biggest downsides are it's size and power, which are both not that big of issues since I've got a whole 44U rack for just a few servers and I don't get billed for my power usage. I was surprised to read this. I was looking in to colocation services (for less than a rack) and everywhere I spoke to, including Hurricane Electric, included a set number of amps (which I assume is at 120V?). Specially, HE offered me 2 amps with 7U of rack space. That seemed really low to me, just one of my 2U servers with a lot of hard drives idles at around 100W or just under 1A and easily exceeds 2A when it's really working (which admittedly is rare, it mostly idles). I didn't follow up to see how that is actually metered. I'd love to hear about other folks experiences with collocating - is this common?
- phirephly 3y ago
  The base power with any colo space is going to be minimal. You typically come back and spec out what additional power you want with the rack a la carte.
derefr 3y ago
Can someone explain why loading a 1MM-route BGP table onto a network switch is a "hard problem" that requires fancy hardware to solve, rather than something that even commodity hardware today is capable of? Presuming you do your IPv4 and IPv6 routing separately... For IPv4, an interval-treemap from uint32-pair intervals to uint8 output ports fits into the default memory config of a PC from 1994; and each lookup into said tree resolves in nanoseconds, even on a machine of the era — esp. for tree-node pages that are hot in CPU cache. And for IPv6, the tree could grow a lot larger, since the intervals are, per se, "uint128"-pairs... but there just aren't that many extant IPv6 routes yet, so the table is actually small in practice. What are the constraints on the problem that I'm missing?
- phirephly 3y ago
  When you want line rate forwarding across several Tbps of front panel ports, you need the packet pipeline to be able to make all the routing decisions without involvement from the OS. 8Bpps just doesn't give you time to be able to walk any kind of data structure in memory. Running full internet tables on a x86 server where you can only get a few Gbps up to maybe a few dozen Gbps is much easier.
  - cookiengineer 3y ago
    Isn't this what BPF and XDP try to target with their SmartNIC offloading, like the ones supported from Metronome?
    - phirephly 3y ago
      Sure. And switch ASICs hang off a CPU as a PCIe device, so the line between smartNICs and low end ASICs start to blur. The key is hardware acceleration of some sort.
- alfons_foobar 3y ago
  Speed. If your want to forward traffic at line rate (think 10/40/100 gbps per port), having the OS handle packets becomes a bottleneck.
rigidbus 3y ago
I'd like to assign a unique IPv6 address for each user of my service. Since I'm in Australia I looked to APNIC but their pricing is a bit intimidating for a side project. I'm primarily after stable addresses so that my users never have to reconfigure anything if the underlying infrastructure (Vultr to start with) changes. What options should I be looking at?
- xfer 3y ago
  Well, no such thing as stable addresses since you don't really own IP addresses, you are leasing from RIR. But, if you are fine with APNIC allocation, see if RIPE PI Allocation is sufficient for you, they are much cheaper(RIPE charges $50/yr), add some LIR cost, total would be <$100/yr.
  - rigidbus 3y ago
    Yeah I'm aware you don't really own IP addresses just as you don't really own domain names. In both cases there is degrees of stability you can expect though. I'll look take a look at RIPE, thanks.

87 points by bsilvereagle 3y ago

This article is also essentially available as a podcast. https://oxide.computer/podcasts/on-the-metal/kenneth-finnegan https://oxide.computer/podcasts/on-the-metal/kenneth-finnega...

dang 3y ago

Discussed at the time: Creating an Autonomous System for Fun and Profit - https://news.ycombinator.com/item?id=15727115 https://news.ycombinator.com/item?id=15727115 - Nov 2017 (16 comments) (p.s. reposts are fine after a year or so; links to past threads are just to satisfy extra-curious readers)

cantaloupe 3y ago

The article mentions that the Cisco router used is limited to a million addresses, which would be exceeded in “2-3 years.” Looks like the author got at least double the life out of the router, because the internet is just approaching one million BGP entries now! https://bgp.potaroo.net/bgprpts/rva-index.html https://bgp.potaroo.net/bgprpts/rva-index.html

phirephly 3y ago
I only got a few years out of it. I'm running an Arista 7280SR-48C8 now. The problem is that the million tcam entries are split between IPv4 and IPv6, so I really ran out of space.
- cantaloupe 3y ago
  Bummer! Didn’t realize the graph was not for not IPv4 and IPv6. Have you done anything fun with the AS or had an opportunity to say “Luckily, I do have an AS!” in a time of need?
  - phirephly 3y ago
    I started an Internet Exchange Point adjacent to it, use it to host mirror.fcix.net, and got a second ASN to build the anycast ns-global.zone service
    - rlaager 3y ago
      Is the map on https://ns-global.zone/ https://ns-global.zone/ actually the MicroMirror map? There seem to be a lot more dots on the map than in the traffic graph. If that map is accurate, I’m curious where the Minneapolis-ish node is (network wise). I’m not seeing 23.128.97.0/24 via MICE.
    - phirephly 3y ago
      The map is a latency heat map. I should probably replace it with a POP map

davisr 3y ago

Don't use Cisco equipment; they put (quite sloppy) backdoors in their products. Absolutely zero trust with them. Snowden: The NSA planted backdoors in Cisco products --- https://www.infoworld.com/article/2608141/snowden--the-nsa-planted-backdoors-in-cisco-products.html https://www.infoworld.com/article/2608141/snowden--the-nsa-p... Backdoors Keep Appearing In Cisco's Routers --- https://www.tomshardware.com/news/cisco-backdoor-hardcoded-accounts-software,37480.html https://www.tomshardware.com/news/cisco-backdoor-hardcoded-a... Sinister secret backdoor found in networking gear perfect for government espionage: The Chinese are – oh no, wait, it's Cisco again --- https://www.theregister.com/2019/05/02/cisco_vulnerabilities/ https://www.theregister.com/2019/05/02/cisco_vulnerabilities...

sneak 3y ago
The idea that the network is trustworthy is outdated. Assume all network connections, including those through your own equipment, are under surveillance. You shouldn't be trusting your network equipment to be not-backdoored in the first place. (Never mind that the router on the other end of the cable isn't yours and is untrusted also, whether Cisco or not.) CPUs have AES key schedule accelerator instructions for a reason now. Encrypt and authenticate your traffic.
- davisr 3y ago
  I respectfully disagree. Are you suggesting everyone just rolls over and forgets about making decisions to the best of their ability to ensure security? The switches on the outside of your network do not matter. The traffic doesn't matter. If you lose trust in your own switching equipment then it's all over. Management network? Compromised. Segmented traffic? Compromised. IPMI/BMC interfaces? Compromised. Anybody else's malformed traffic could breach your defense, and breach the very sanctity of the traffic your network is spitting out. It doesn't matter if your computer encrypts its traffic because a breached switch can just silence it. A company selling switches/routers/firewalls should _not_ have these liabilities, and as these liabilities are known, nobody should buy Cisco equipment, ever. Buy the equipment that you know is the safest. Don't just give up and roll over!

hamandcheese 3y ago

> and its biggest downsides are it's size and power, which are both not that big of issues since I've got a whole 44U rack for just a few servers and I don't get billed for my power usage. I was surprised to read this. I was looking in to colocation services (for less than a rack) and everywhere I spoke to, including Hurricane Electric, included a set number of amps (which I assume is at 120V?). Specially, HE offered me 2 amps with 7U of rack space. That seemed really low to me, just one of my 2U servers with a lot of hard drives idles at around 100W or just under 1A and easily exceeds 2A when it's really working (which admittedly is rare, it mostly idles). I didn't follow up to see how that is actually metered. I'd love to hear about other folks experiences with collocating - is this common?

phirephly 3y ago
The base power with any colo space is going to be minimal. You typically come back and spec out what additional power you want with the rack a la carte.

derefr 3y ago

Can someone explain why loading a 1MM-route BGP table onto a network switch is a "hard problem" that requires fancy hardware to solve, rather than something that even commodity hardware today is capable of? Presuming you do your IPv4 and IPv6 routing separately... For IPv4, an interval-treemap from uint32-pair intervals to uint8 output ports fits into the default memory config of a PC from 1994; and each lookup into said tree resolves in nanoseconds, even on a machine of the era — esp. for tree-node pages that are hot in CPU cache. And for IPv6, the tree could grow a lot larger, since the intervals are, per se, "uint128"-pairs... but there just aren't that many extant IPv6 routes yet, so the table is actually small in practice. What are the constraints on the problem that I'm missing?

phirephly 3y ago
When you want line rate forwarding across several Tbps of front panel ports, you need the packet pipeline to be able to make all the routing decisions without involvement from the OS. 8Bpps just doesn't give you time to be able to walk any kind of data structure in memory. Running full internet tables on a x86 server where you can only get a few Gbps up to maybe a few dozen Gbps is much easier.
- cookiengineer 3y ago
  Isn't this what BPF and XDP try to target with their SmartNIC offloading, like the ones supported from Metronome?
  - phirephly 3y ago
    Sure. And switch ASICs hang off a CPU as a PCIe device, so the line between smartNICs and low end ASICs start to blur. The key is hardware acceleration of some sort.
alfons_foobar 3y ago
Speed. If your want to forward traffic at line rate (think 10/40/100 gbps per port), having the OS handle packets becomes a bottleneck.

rigidbus 3y ago

I'd like to assign a unique IPv6 address for each user of my service. Since I'm in Australia I looked to APNIC but their pricing is a bit intimidating for a side project. I'm primarily after stable addresses so that my users never have to reconfigure anything if the underlying infrastructure (Vultr to start with) changes. What options should I be looking at?

xfer 3y ago
Well, no such thing as stable addresses since you don't really own IP addresses, you are leasing from RIR. But, if you are fine with APNIC allocation, see if RIPE PI Allocation is sufficient for you, they are much cheaper(RIPE charges $50/yr), add some LIR cost, total would be <$100/yr.
- rigidbus 3y ago
  Yeah I'm aware you don't really own IP addresses just as you don't really own domain names. In both cases there is degrees of stability you can expect though. I'll look take a look at RIPE, thanks.