11 ms·
"BGP at home": getting a DIA circuit installed at home
- c3534l 4y agoBut why?
- cobertos 4y agoWhy not? Sounds fun They mention at the top because they didn't want to colocate it. So I imagine they did it because they could?
- dboreham 4y agoProbably cheaper than bringing up a second colo if you don't need to host many machines.
- skissane 4y agoIf I had more money than sense, I’d do the same thing. If I was filthy rich (which I’m not and probably never will be), I’d build a data centre in my mansion and buy a brand new IBM mainframe and stick it in there. Because I can. For the same reason other rich people build some huge garage with dozens of cars in it.
- remram 4y agoIt sounds like they wanted 2 separate network paths but they still end up with only 1 where they can use BGP. Maybe they can still use their residential connection as failover, with a VPN to a VPS where they can use BGP?
- dboreham 4y agoThey all likey traverse the same physical path and would all be taken out by the same gravel truck hitting a pole. The two connection thing was I think more about defense against random retail connectivity flaps (someone decides to upgrade a router's firmware and just takes the town offline, meanwhile nobody answers support calls, that sort of thing).
- bombcar 4y agoYeah true multihoming is expensive and involves things like hiring people to actually inspect the physical lines and Make sure they don’t run down the same street at some point (and be subject to the same backhoe of fiber death)
- remram 4y agoTrue. I thought they wanted BGP to implement multihoming but upon re-reading they needed it anyway for other reasons.
- dboreham 4y agoBGP would be to allow fail over of some service from their primary colo.
- HyperSane 4y agoI would buy one of every high end enterprise storage array to play with. My former life as a VMWare admin made me appreciate storage.
- skissane 4y agoWhich means you'd need a IBM mainframe, since IBM mainframes require enterprise storage arrays which come with special software for emulating the old mainframe hard disks (ECKD), and if you are going to buy one of those, you'd need a mainframe to drive it.
- HyperSane 4y agoNo, most of them use fiber channel.
- skissane 4y agoYes, they do use Fibre Channel (FC), but the protocol they run on top of FC is very different from the usual FC protocols. FC has four layers, from FC-0 (the physical layer) through to FC-4 (the application layer). At level FC-4, Linux/Unix/Windows/etc servers mostly speak "Fibre Channel Protocol for SCSI" aka SCSI-FCP, which transmits SCSI commands over FC. However, while IBM mainframes do use SCSI-over-FC, they also use something else called "FICON", which instead of transmitting SCSI commands, uses the same command set as IBM's legacy non-SCSI mainframe hard drives (such as the IBM 3390, which was IBM's last non-SAN mainframe hard disk line from the late 80s / early 90s). Mainstream platforms have what IBM calls FBA (Fixed Block Array) disks, in which the disk is an array of sectors which all have the same size. While the IBM mainframe hardware supports that, most IBM mainframe operating systems don't support those, however; Linux and VSEn are the exceptions, and z/VM is a partial exception. As well as FBA, IBM has ECKD (Extended Count Key Data) disks, like the IBM 3390 was. (ECKD was preceded by CKD, which is conceptually the same – the same sector format, etc – but had a less efficient command set.) With ECKD, sectors can have variable sizes – each disk track can have different sized sectors, you can even mix different sector sizes within the same disk track. Also, sectors optionally have keys, and the disk has commands to search for sectors based on their keys. Originally, these variable sized sectors actually physically existed on disk; nowadays, the SAN only has standard hard disks (or SSDs), but ECKD disks are emulated in the SAN software on top of them. z/OS, the most popular IBM mainframe OS, only supports ECKD disks, not FBA ones. Linux and VSEn support both ECKD and FBA. For ECKD disks, you can't use the standard SCSI command set, you have to use IBM's DASD command set. Many high-end enterprise arrays do support ECKD and FICON, but you usually have to pay significant extra license fees to enable the software that supports that.
- sbierwagen 4y agoBesides the stated reason, there's also this: >(as of this writing, VZ was pricing 50Mbps commited @ $455/mo; 100Mbps @ $661/mo; 1Gbps @ $999/mo; 5Gbps @ $2,099/mo; and 10Gbps @ $3,099/mo). Ten gigabit is not yet a common residential internet service in North America. If that's what you want, this would be the only way to get it. I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc. https://www.seattleix.net/join https://www.seattleix.net/join As expensive and pointless as buying a yacht, but at least getting it online would require a lot of tedious work! (Like OP, I also jumped through the hoops for a Comcast fiber drop years ago (but to a business location) for the same eyewatering price but dramatically fewer megabits per second. The only upside is that the pair traveled directly to a CO in the same office park, so we would routinely see 1-2ms pings to anycast IPs like 4.2.2.2, 8.8.8.8, etc)
- Shank 4y ago> If that's what you want, this would be the only way to get it. 10Gbps PON is available from AT&T Fiber in some markets, and lower tiers (5Gbps) are coming online from Google Fiber in other markets. In SoCal, I've seen billboards for 10Gbps AT&T Fiber, for example. Obviously you don't get an SLA with a residential connection, though, and I highly doubt that many customers operating at full capacity will be problem-free.
- techsupporter 4y ago> I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc For what it's worth, this isn't the same as what the person who wrote the post did. They bought a DIA circuit to an ISP. What you're seeking is a point-to-point fiber link or MPLS to a panel in KOMO Plaza or the Westin building. You don't need to be near either of those locations to do it and CenturyLink (Lumen, these days) will sell you the circuit but you're not going to like what it costs. We have three of these circuits where I work and they are juuuuust a bit higher than what the article's author says they're paying for the DIA. (This, along with the eyewatering cross-connect fees that the operators of those two facilities charge, is why getting a SIX port is so expensive on retail colo. You're either paying for the privilege of running some strands of glass or you're paying to use someone's extension switch and the connection they're paying for to get back to the SIX core.)
- dboreham 4y agoThis is how any larger business (e.g. a hospital, university, anyone doing any serious networking) gets their connectivity these days. Basically the modern equivalent of a T-1. You get proper service: no random dropouts, and a clueful human on the phone and you could complain about QoS and they'd not laugh in your face. The different thing here is that the service was provided to a residential building, but that's not entirely unheard of. I've done it myself for example.
- gertrunde 4y agoPrimarily for resilience. In the event that there are issues at the primary location, internet routing protocols will re-route the relevant network prefixes to the secondary location. The public /24 that was previously routed to their NYC datacenter now goes to that Connecticut house.
- lukeh 4y agoWhy two NIDs?
- rektide 4y agoThis is one of the most literal Conway's Law instances I've seen... > organizations design systems that mirror their own communication structure. An org for the physical layer has a box, then the org offering the connectivity layer plugs their box into the first box. Like OSI layers stacking up. https://en.m.wikipedia.org/wiki/Conway%27s_law https://en.m.wikipedia.org/wiki/Conway%27s_law
- dboreham 4y agoIt's more likely to reflect a hierarchy in the network (e.g. monomode span dropping down to multimode for the demarc). The telco just colocated their box on the subscriber's premises.
- rektide 4y agoI don't buy it. You seem to be contending that the second NID is just a fancy transciever between fibers. But both boxes almost certainly have QSFP or some such, could input or output whatever fiber the job calls for. Quibbling about fiber types as an excuse for having two boxes seems absurd. Your assertion also seems contested by the words in this post: > Dispatch 5: NID #2 install. This is where I began to see the distinction between Verizon Telecom (VzT) and Verizon Business (VzB). While the layer 1 infrastructure had been installed and tested, there still remained the IP layer, which required another dispatch for another technician to install a dedicated NID for Verizon Business -- a Ciena 3903, with 1310nm single mode optics. It does mention a different fiber connection, but to me, this is describing each layer of the stack having both it's own organizational unit & it's own physical box that it manages.
- VoidWhisperer 4y agoI'm curious about this too.. it seems like it might be some intricacy because of Verizon business and Verizon telecom being two different entities, but I am curious if there is a technical reason for it too.
- NoZebra120vClip 4y agoI'm curious to know details like what sort of house or dwelling this person lives in; anyplace with enough capacity for all that (4U rack space, power and cooling, etc.) seems spacious. The article says it's right off a major street. So is it urban? Suburban? Single-family home? Inquiring minds want to know.
- cmeacham98 4y agoJust so you know, 4U of rack space is a whopping 7 inches of height. One rack holds many Us, with most "full size" racks clocking in at 42U. Open-air racks are typically less than 2 feet across and 3-4 feet deep (example [1]), hardly a significant stretch unless you live in a very small space. 1: https://www.amazon.com/StarTech-com-Open-Frame-Server-Rack/dp/B00O6GNLQE https://www.amazon.com/StarTech-com-Open-Frame-Server-Rack/d...
- zamadatix 4y ago4u of space is less volume than many PC cases. I’ve got a 22u “half rack” in my apartment (half for testing gear for work, half because I like to tinker). Bigger concern would be noise. 1u network gear tends to have high pitched 20mm fans which can drive you nuts if you don’t have it in a separate room. Noctua makes a great kit I use in my own gear but Verizon probably wouldn’t be very happy with you if you tried to do that :). I went the route of “find a dirt cheap gigabit colo” ($150/m for quarter rack with gigabit and bgp peering) and just tunnel the IP space back home. Upside is you don’t have to pay to have the circuit put to where you currently live and you don’t have to worry about space/noise. Downside is if you want to use it locally at home you get extra latency due to the tunnel.
- cmeacham98 4y ago> Downside is if you want to use it locally at home you get extra latency due to the tunnel. I guess it depends on how your tunnel is set up, but shouldn't you be able to add a static route to your rack on the "home side"?
- zamadatix 4y ago
- cobertos 4y agoCurious what they're running just on the other side of the Verizon Business NIB, and if there was any hiccups in the peering process with Verizon
- remram 4y agoBGP peering! I would be happy just to get IPv6 connectivity...
- onphonenow 4y agoThe pricing is eye watering. 2,100 for 5gbps? Att fiber is $200 per month where I am w a static block.
- cmeacham98 4y agoResidential internet service is oversubscribed and doesn't have an SLA.
- digitalsushi 4y agoOversubscribed means different things to different groups of people. (I'm replying to people in general, and not to the commenter above me) To an ISP, it means that a customer could exeperience a shortage of service, but that a group of customers would have to simultaneously utilize the infrastructure. ISPs try to keep a ratio of customers to service where this isn't likely to happen, or for discount providers, that this happens only during peak times. Ultimately it is a balance of customer attrition. To a customer, it sounds like they are being deprived of what they are paying for. (Residental customers agree to a best-effort service typically - the opposite of agreeing to a specific service level). Even with ample service capacity, it is often the first factor considered when any component in the aggregate network is failing or underperforming. I sure do not miss being a residential ISP. But I carry with me just enough sympathy for service providers that I am "always a blast at a party".
- salawat 4y ago...Eh, you'd still be okay in my book. The problem I have is the lack of transparency, obtuse terms and the huge amount of funds taken in by the bigger operations that instead of being transformed into moar infra, get turned into exec bonuses/lobbying to not have to build infra/marketing to convince people the infra that was built is the best that's possible, yes sirree Bob. When all you have is a string, and tin cans, you do the best you can. When your execs are taking big Federal bux, and setting them on fire instead of increasing overall throughput...
- salawat 4y ago
- MPSimmons 4y agoGeneral reminder that an SLA is a contractual agreement, and not a guarantee of availability. It literally describes the renumeration due when the service doesn't meet the promised availability.
- cardy31 4y agoI often hear people use SLA and SLO interchangeably. Very annoying having to clarify.
- deleted 4y ago[deleted]
- markonen 4y agoI rent a single ~600 meter fiber that connects my home to the network infrastructure of the company I run. The rent is $75/mo or so. I have passive CWDM muxes on the line to run two 10G connections to two separate edge routers. My home router is a MikroTik CRS309—it’s about $250, fanless, has 8 SFP+ ports and advertises my home network blocks over BGP to the two routers (for HA). The setup works great. The best part is how it’s small and fanless and fits inside the very small wall box the fiber terminates in.
- techsupporter 4y ago> My home router is a MikroTik CRS309 ... advertises my home network blocks over BGP How do you fit two full BGP tables into 512MB of RAM? I've looked into MikroTik boxes before and maybe they're doing something I'm not understanding. On the routers I have manage, two IPv4 feeds take up about 1.1GB and three IPv6 adds another 450MB.
- no_carrier 4y agoIf by 'full BGP tables' you mean the entire internet routing table, you don't need that to advertise a network on the internet. You can receive just a default route if you wish and still be able to advertise to any carriers you have a connection with. You won't lose out on any functionality in dual homing or anything like that.
- oarsinsync 4y ago> You can receive just a default route … You won't lose out on any functionality in dual homing or anything like that. Except you do lose out on best path routing / any other outbound TE, and you’re now restricted to rudimentary load balancing methodologies / manual prefix-specific hackery.
- iptrans 4y agoOnly for outbound traffic, tho. For eyeball networks and other stub networks this is mostly fine.
- Aachen 4y agoHuh, so per the last paragraph, they're paying a ton to have an SLA, all the ISP has to provide is basic connectivity because BGP and stuff is being done by the customer themselves, and yet they have more outages in a few months than I have on a consumer connection in years? The whole thing has a lot of hack value, it's cool and also worth something to be in control of your own networking, even if it's more expensive. Like buying apple: overpriced for the specs you're getting, but you're assured it'll be good quality (that's the idea anyway) and it looks cool (to most). Except... apparently it's still got issues, regularly? Now I'm really wondering what the point was, at least with hindsight
- rfoo 4y agoThe post said that consumer connection does not let you do BGP peering with the ISP. There are real value in doing so in addition to the hack value, by doing so you can steer the Internet traffic to whatever IP blocks you owned to your house, dynamically. For example OP mentioned "add some resilience" as motivation, i.e. anytime their services running in the DC failed he can reroute the traffic to ... their house.
- Karrot_Kream 4y agoYou're absolutely feeling the same problems on consumer connections. With DIA, you're the only subscriber on that circuit, so if upstream is having a problem then your neighboring consumers will have them too. Lower bandwidth dedicated circuits can often "feel" faster than a higher bandwidth consumer connection.
- scottgg 4y agoThere’s also DN42 [1] if you want to mess around with a bigish BGP network for free 1. https://dn42.eu/Home https://dn42.eu/Home
- jonatron 4y agoSee also a guy who used a microwave wireless link to a local data centre from his apartment: https://www.ispreview.co.uk/index.php/2020/12/video-using-a-diy-microwave-link-to-solve-slow-broadband.html https://www.ispreview.co.uk/index.php/2020/12/video-using-a-... / https://www.youtube.com/watch?v=_AAOVVmaFEo https://www.youtube.com/watch?v=_AAOVVmaFEo
- traceroute66 4y ago> While the SLA says 100%, don't expect perfection I've always considered Verizon's (and prior to that, WorldCom and/or MCI) 100% SLA to be pure marketing BS. Anyone who's ever worked with telecoms knows it's simply unrealistic. Fibres will get cut, carrier's routers will need software updates etc. etc. etc. I reckon the reason you pay a Verizon-tax is so they have spare cash floating around to pay you the inevitable SLA penalties. Personally, I prefer dealing with carriers who have more real-world SLAs.
- bcrl 4y agoThe only thing SLAs do is tell you how much you will be compensated for downtime. If a fibre cut happens, you can end up with a 16 hour outage regardless of what the SLA guarantees. It all depends on the nature, location and extent of the damage. A couple of summers ago one of my wavelengths was down for more than 16 hours. 3 x 432 count cables had to be replaced through 3 manholes because a fibre seeking backhoe ignored the locates. Splicing high count cables takes a lot of time (even with ribbon fibre).
- Habgdnv 4y agoI've been running BGP at home for the last 10 years, and in my area, most ISPs prefer to use Dedicated Internet Access (DIA) for BGP sessions. Fortunately, I live in a place where there are a few smaller ISPs with around 2000-3000 customers that are willing to run BGP with me as a home user. They're impressed that I have BGP, ASN, and IPs as a home user, and I think it's pretty cool too. Additionally, running one more BGP session makes their network appear larger, so they benefit as well. I currently have a BGP session with one of my ISPs on the 15 euro plan (1 Gbps), and another local ISP is giving me 10 Gbps for free because I'm a test user. Both of these ISPs support BGP, and it's amazing that I can have BGP sessions as a home user. However, it can be challenging to speak with regular sales reps who may not be familiar with BGP. It takes persistence and some effort to get to the network administrator. It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session, even when they're not asking for SLAs or dedicated connections. Technically, you can even run a BGP session over dial-up. It seems like these restrictions are in place to squeeze more money from customers. It's no secret that we've run out of IPv4 addresses, and I happen to have a /24 subnet just for myself. Even with all my usage, I barely use half of it. There's no rule that says ISPs should only accept /24 or shorter prefixes, and this is contributing to the depletion of IPv4 addresses. I would happily announce a /25 and return the other /25 to the world, but I must have a /24 to do it. This practice of requiring /24 subnets may have made sense in the past to preserve router memory, but with today's technology, it's nonsensical and only serves to artificially deplete available IPv4 addresses.
- traceroute66 4y ago> It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session Don't worry, it can be frustrating for businesses too ! Cogent famously nickle-and-dime their customers and consider BGP to be a chargeable extra, even on their IP Transit product which is somewhere where you would expect BGP to be a given. Of course whether you should do business with Cogent is another matter, especially given their spammy cold-calling tactics, seemingly once you're on their list you can never get off it.
- throwaway894345 4y agoI have a super rudimentary understanding of what BGP is (it allows you to advertise addresses something something), but I don't really understand why it's useful and thus I'm finding TFA to be a little inaccessible. Could you provide some context about what you're using BGP for (what is your use case; what does it enable for you; etc) such that it might fill in some gaps for me?
- tgsovlerkhgsel 4y agoThe main issue he wanted to avoid (trees falling on lines) still seems to be there. Looking at past outages, I feel like given a certain budget, you would get much better uptime with two or more diverse (i.e. not "two fibers going exactly the same path") residential providers than with one commercial one, and for anything but the most critical projects, that's the way to go IMO. Most importantly though, anything really really important that is so important to use custom fibers should be designed so it also works over the public Internet. Unless degraded service is worse than no service, it doesn't matter that the Internet is theoretically not reliable enough/doesn't provide enough guarantees - at least keep it as a backup so that when a backhoe takes out your custom fiber, you can switch over to the VPN instead of shutting down air traffic at one of Europe's largest airports. Seems like another major reason for the decision here were SLO/SLAs. Those seem rather meaningless. The residential ISP will not try to max out their 95% SLO - an ISP that's down 1.5 days every month or leaves you offline for 18 days wouldn't be very popular. The commercial SLOs, on the other hand, sound great but they're not magic. If the issue can't be fixed in that time, they will be violated, and you'll typically get a credit for a few months of service - money that you wouldn't have spent in the first place if you just went with residential. In the end, he still has countless single points of failure, e.g. the above-ground fiber line waiting for a tree, or a non-redundant power supply that he can't easily replace himself. Sure, he has an SLO, but with a dual uplink residential (which is cheaper), a CPE ("modem"/router) failure wouldn't be an outage in a first place.
- tssva 4y agoThey state at the beginning of the article that the main service location is a data center in NYC. This is a redundant service location. An outage would require failure of connectivity to his home and to the NYC data center.
- matthewmorgan 4y agoWhen did it fall out of fashion to spell out any acronyms/abbreviations you're using the first time they're used?