3 ms·

Don't get me started on that. However the two circuits are sold as a fully resilient pair (different tails from different DCs arrive into the building in Washin

Don't get me started on that. However the two circuits are sold as a fully resilient pair (different tails from different DCs arrive into the building in Washington different directions with guaranteed different paths to the DCs, then in contract it's guaranteed neither circuit will run via the same equipment or on the same submarine circuit. In the UK they arrive in two separate cities) However I know reality and contracts never meet, unfortunately in a large disfunctional organization there are other considerations in circuit procurement than technical requirements. As for internet circuits, I had two ISPs in NY on two separate paths, which is great. Something went wrong about a month ago, and the routing from one of the ISPs changed, meaning that we were back to a single point of failure who we have no business relationship with

Hikikomori 6y ago
We had a partner that bought a redundant mpls service from their DC to another DC (not their own) and to AWS direct connect via that DC. Their mpls provider was a SPOF, the provider for the second DC was a SPOF, then the direct connect circuits terminated in the same AWS router, another SPOF. All 3 had SPOF's had caused its own significant downtime within a few months. This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.
- iso1631 6y ago
  Choir here, although in our case the connectivity was designed by a third party consulting firm, against the advice of long standing employees with experience. The third party consulting firm are long gone, as are the people high up in the company who brought them in. Now there's new people high up who come up with the same problems, but in a slightly different way, arguing how their basket is far better than the previous basket. Sigh.

Hikikomori 6y ago

We had a partner that bought a redundant mpls service from their DC to another DC (not their own) and to AWS direct connect via that DC. Their mpls provider was a SPOF, the provider for the second DC was a SPOF, then the direct connect circuits terminated in the same AWS router, another SPOF. All 3 had SPOF's had caused its own significant downtime within a few months. This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.

iso1631 6y ago
Choir here, although in our case the connectivity was designed by a third party consulting firm, against the advice of long standing employees with experience. The third party consulting firm are long gone, as are the people high up in the company who brought them in. Now there's new people high up who come up with the same problems, but in a slightly different way, arguing how their basket is far better than the previous basket. Sigh.