5 ms·

But why?

by c3534l 4y ago

But why?

cobertos 4y ago
Why not? Sounds fun They mention at the top because they didn't want to colocate it. So I imagine they did it because they could?
- dboreham 4y ago
  Probably cheaper than bringing up a second colo if you don't need to host many machines.
skissane 4y ago
If I had more money than sense, I’d do the same thing. If I was filthy rich (which I’m not and probably never will be), I’d build a data centre in my mansion and buy a brand new IBM mainframe and stick it in there. Because I can. For the same reason other rich people build some huge garage with dozens of cars in it.
- remram 4y ago
  It sounds like they wanted 2 separate network paths but they still end up with only 1 where they can use BGP. Maybe they can still use their residential connection as failover, with a VPN to a VPS where they can use BGP?
  - dboreham 4y ago
    They all likey traverse the same physical path and would all be taken out by the same gravel truck hitting a pole. The two connection thing was I think more about defense against random retail connectivity flaps (someone decides to upgrade a router's firmware and just takes the town offline, meanwhile nobody answers support calls, that sort of thing).
    - bombcar 4y ago
      Yeah true multihoming is expensive and involves things like hiring people to actually inspect the physical lines and Make sure they don’t run down the same street at some point (and be subject to the same backhoe of fiber death)
    - remram 4y ago
      True. I thought they wanted BGP to implement multihoming but upon re-reading they needed it anyway for other reasons.
      - dboreham 4y ago
        BGP would be to allow fail over of some service from their primary colo.
- HyperSane 4y ago
  I would buy one of every high end enterprise storage array to play with. My former life as a VMWare admin made me appreciate storage.
  - skissane 4y ago
    Which means you'd need a IBM mainframe, since IBM mainframes require enterprise storage arrays which come with special software for emulating the old mainframe hard disks (ECKD), and if you are going to buy one of those, you'd need a mainframe to drive it.
    - HyperSane 4y ago
      No, most of them use fiber channel.
      - skissane 4y ago
        Yes, they do use Fibre Channel (FC), but the protocol they run on top of FC is very different from the usual FC protocols. FC has four layers, from FC-0 (the physical layer) through to FC-4 (the application layer). At level FC-4, Linux/Unix/Windows/etc servers mostly speak "Fibre Channel Protocol for SCSI" aka SCSI-FCP, which transmits SCSI commands over FC. However, while IBM mainframes do use SCSI-over-FC, they also use something else called "FICON", which instead of transmitting SCSI commands, uses the same command set as IBM's legacy non-SCSI mainframe hard drives (such as the IBM 3390, which was IBM's last non-SAN mainframe hard disk line from the late 80s / early 90s). Mainstream platforms have what IBM calls FBA (Fixed Block Array) disks, in which the disk is an array of sectors which all have the same size. While the IBM mainframe hardware supports that, most IBM mainframe operating systems don't support those, however; Linux and VSEn are the exceptions, and z/VM is a partial exception. As well as FBA, IBM has ECKD (Extended Count Key Data) disks, like the IBM 3390 was. (ECKD was preceded by CKD, which is conceptually the same – the same sector format, etc – but had a less efficient command set.) With ECKD, sectors can have variable sizes – each disk track can have different sized sectors, you can even mix different sector sizes within the same disk track. Also, sectors optionally have keys, and the disk has commands to search for sectors based on their keys. Originally, these variable sized sectors actually physically existed on disk; nowadays, the SAN only has standard hard disks (or SSDs), but ECKD disks are emulated in the SAN software on top of them. z/OS, the most popular IBM mainframe OS, only supports ECKD disks, not FBA ones. Linux and VSEn support both ECKD and FBA. For ECKD disks, you can't use the standard SCSI command set, you have to use IBM's DASD command set. Many high-end enterprise arrays do support ECKD and FICON, but you usually have to pay significant extra license fees to enable the software that supports that.
sbierwagen 4y ago
Besides the stated reason, there's also this: >(as of this writing, VZ was pricing 50Mbps commited @ $455/mo; 100Mbps @ $661/mo; 1Gbps @ $999/mo; 5Gbps @ $2,099/mo; and 10Gbps @ $3,099/mo). Ten gigabit is not yet a common residential internet service in North America. If that's what you want, this would be the only way to get it. I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc. https://www.seattleix.net/join https://www.seattleix.net/join As expensive and pointless as buying a yacht, but at least getting it online would require a lot of tedious work! (Like OP, I also jumped through the hoops for a Comcast fiber drop years ago (but to a business location) for the same eyewatering price but dramatically fewer megabits per second. The only upside is that the pair traveled directly to a CO in the same office park, so we would routinely see 1-2ms pings to anycast IPs like 4.2.2.2, 8.8.8.8, etc)
- Shank 4y ago
  > If that's what you want, this would be the only way to get it. 10Gbps PON is available from AT&T Fiber in some markets, and lower tiers (5Gbps) are coming online from Google Fiber in other markets. In SoCal, I've seen billboards for 10Gbps AT&T Fiber, for example. Obviously you don't get an SLA with a residential connection, though, and I highly doubt that many customers operating at full capacity will be problem-free.
- techsupporter 4y ago
  > I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc For what it's worth, this isn't the same as what the person who wrote the post did. They bought a DIA circuit to an ISP. What you're seeking is a point-to-point fiber link or MPLS to a panel in KOMO Plaza or the Westin building. You don't need to be near either of those locations to do it and CenturyLink (Lumen, these days) will sell you the circuit but you're not going to like what it costs. We have three of these circuits where I work and they are juuuuust a bit higher than what the article's author says they're paying for the DIA. (This, along with the eyewatering cross-connect fees that the operators of those two facilities charge, is why getting a SIX port is so expensive on retail colo. You're either paying for the privilege of running some strands of glass or you're paying to use someone's extension switch and the connection they're paying for to get back to the SIX core.)
dboreham 4y ago
This is how any larger business (e.g. a hospital, university, anyone doing any serious networking) gets their connectivity these days. Basically the modern equivalent of a T-1. You get proper service: no random dropouts, and a clueful human on the phone and you could complain about QoS and they'd not laugh in your face. The different thing here is that the service was provided to a residential building, but that's not entirely unheard of. I've done it myself for example.
gertrunde 4y ago
Primarily for resilience. In the event that there are issues at the primary location, internet routing protocols will re-route the relevant network prefixes to the secondary location. The public /24 that was previously routed to their NYC datacenter now goes to that Connecticut house.

But why?

by c3534l 4y ago

But why?

Why not? Sounds fun They mention at the top because they didn't want to colocate it. So I imagine they did it because they could?

dboreham 4y ago
Probably cheaper than bringing up a second colo if you don't need to host many machines.

skissane 4y ago

If I had more money than sense, I’d do the same thing. If I was filthy rich (which I’m not and probably never will be), I’d build a data centre in my mansion and buy a brand new IBM mainframe and stick it in there. Because I can. For the same reason other rich people build some huge garage with dozens of cars in it.

remram 4y ago
It sounds like they wanted 2 separate network paths but they still end up with only 1 where they can use BGP. Maybe they can still use their residential connection as failover, with a VPN to a VPS where they can use BGP?
- dboreham 4y ago
  They all likey traverse the same physical path and would all be taken out by the same gravel truck hitting a pole. The two connection thing was I think more about defense against random retail connectivity flaps (someone decides to upgrade a router's firmware and just takes the town offline, meanwhile nobody answers support calls, that sort of thing).
  - bombcar 4y ago
    Yeah true multihoming is expensive and involves things like hiring people to actually inspect the physical lines and Make sure they don’t run down the same street at some point (and be subject to the same backhoe of fiber death)
  - remram 4y ago
    True. I thought they wanted BGP to implement multihoming but upon re-reading they needed it anyway for other reasons.
    - dboreham 4y ago
      BGP would be to allow fail over of some service from their primary colo.
HyperSane 4y ago
I would buy one of every high end enterprise storage array to play with. My former life as a VMWare admin made me appreciate storage.
- skissane 4y ago
  Which means you'd need a IBM mainframe, since IBM mainframes require enterprise storage arrays which come with special software for emulating the old mainframe hard disks (ECKD), and if you are going to buy one of those, you'd need a mainframe to drive it.
  - HyperSane 4y ago
    No, most of them use fiber channel.
    - skissane 4y ago
      Yes, they do use Fibre Channel (FC), but the protocol they run on top of FC is very different from the usual FC protocols. FC has four layers, from FC-0 (the physical layer) through to FC-4 (the application layer). At level FC-4, Linux/Unix/Windows/etc servers mostly speak "Fibre Channel Protocol for SCSI" aka SCSI-FCP, which transmits SCSI commands over FC. However, while IBM mainframes do use SCSI-over-FC, they also use something else called "FICON", which instead of transmitting SCSI commands, uses the same command set as IBM's legacy non-SCSI mainframe hard drives (such as the IBM 3390, which was IBM's last non-SAN mainframe hard disk line from the late 80s / early 90s). Mainstream platforms have what IBM calls FBA (Fixed Block Array) disks, in which the disk is an array of sectors which all have the same size. While the IBM mainframe hardware supports that, most IBM mainframe operating systems don't support those, however; Linux and VSEn are the exceptions, and z/VM is a partial exception. As well as FBA, IBM has ECKD (Extended Count Key Data) disks, like the IBM 3390 was. (ECKD was preceded by CKD, which is conceptually the same – the same sector format, etc – but had a less efficient command set.) With ECKD, sectors can have variable sizes – each disk track can have different sized sectors, you can even mix different sector sizes within the same disk track. Also, sectors optionally have keys, and the disk has commands to search for sectors based on their keys. Originally, these variable sized sectors actually physically existed on disk; nowadays, the SAN only has standard hard disks (or SSDs), but ECKD disks are emulated in the SAN software on top of them. z/OS, the most popular IBM mainframe OS, only supports ECKD disks, not FBA ones. Linux and VSEn support both ECKD and FBA. For ECKD disks, you can't use the standard SCSI command set, you have to use IBM's DASD command set. Many high-end enterprise arrays do support ECKD and FICON, but you usually have to pay significant extra license fees to enable the software that supports that.

sbierwagen 4y ago

Besides the stated reason, there's also this: >(as of this writing, VZ was pricing 50Mbps commited @ $455/mo; 100Mbps @ $661/mo; 1Gbps @ $999/mo; 5Gbps @ $2,099/mo; and 10Gbps @ $3,099/mo). Ten gigabit is not yet a common residential internet service in North America. If that's what you want, this would be the only way to get it. I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc. https://www.seattleix.net/join https://www.seattleix.net/join As expensive and pointless as buying a yacht, but at least getting it online would require a lot of tedious work! (Like OP, I also jumped through the hoops for a Comcast fiber drop years ago (but to a business location) for the same eyewatering price but dramatically fewer megabits per second. The only upside is that the pair traveled directly to a CO in the same office park, so we would routinely see 1-2ms pings to anycast IPs like 4.2.2.2, 8.8.8.8, etc)

Shank 4y ago
> If that's what you want, this would be the only way to get it. 10Gbps PON is available from AT&T Fiber in some markets, and lower tiers (5Gbps) are coming online from Google Fiber in other markets. In SoCal, I've seen billboards for 10Gbps AT&T Fiber, for example. Obviously you don't get an SLA with a residential connection, though, and I highly doubt that many customers operating at full capacity will be problem-free.
techsupporter 4y ago
> I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc For what it's worth, this isn't the same as what the person who wrote the post did. They bought a DIA circuit to an ISP. What you're seeking is a point-to-point fiber link or MPLS to a panel in KOMO Plaza or the Westin building. You don't need to be near either of those locations to do it and CenturyLink (Lumen, these days) will sell you the circuit but you're not going to like what it costs. We have three of these circuits where I work and they are juuuuust a bit higher than what the article's author says they're paying for the DIA. (This, along with the eyewatering cross-connect fees that the operators of those two facilities charge, is why getting a SIX port is so expensive on retail colo. You're either paying for the privilege of running some strands of glass or you're paying to use someone's extension switch and the connection they're paying for to get back to the SIX core.)

dboreham 4y ago

This is how any larger business (e.g. a hospital, university, anyone doing any serious networking) gets their connectivity these days. Basically the modern equivalent of a T-1. You get proper service: no random dropouts, and a clueful human on the phone and you could complain about QoS and they'd not laugh in your face. The different thing here is that the service was provided to a residential building, but that's not entirely unheard of. I've done it myself for example.

gertrunde 4y ago

Primarily for resilience. In the event that there are issues at the primary location, internet routing protocols will re-route the relevant network prefixes to the secondary location. The public /24 that was previously routed to their NYC datacenter now goes to that Connecticut house.