6 ms·

Do you roll your own power grid? Do you roll your own ISP + telecoms network? As a software engineer I live by the ethos that coupling and dependency is bad, b

by yomly 5y ago

Do you roll your own power grid? Do you roll your own ISP + telecoms network? As a software engineer I live by the ethos that coupling and dependency is bad, but if you unravel the layers you start to realise much of our life is centralised: Roads, trains, water, electricity, internet These are quite consolidated and any of these going down would be very disruptive to our lives. Connected software, ie the internet, is still quite new. Being charitable, are these just growing pains in the journey to building out foundational infrastructure?

iso1210 5y ago
My datacentres have two sources of power (plus internal UPS), two main internet lines to two different exchange points (and half a dozen others), plenty of bottled water. At home I have emergency power, water and internet. If the trains stop I drive, if the car breaks I take the train.
- 411111111111111 5y ago
  But having everything redundantly available costs money. While some redundancy is easy to justify... At some point it becomes hard when the MBA wants to cut costs so he gets a bigger bonus. There is even a competitive advantage in living with the risk, as you have less costs and overhead... Sure, you might have an outage once every x years for a few minutes... But that's obviously the fault of the development team, duh
  - thayne 5y ago
    Which is one reason why things like roads, trains, water, electricty, etc. are so heavily regulated. To prevent the companies that hold monopolies over the infrastructure from cutting corners like that.
  - jjk166 5y ago
    This is a classic example of an externality. You use regulations or lawsuits to force the costs back onto the decision makers. Make it so people can collect damages from outages, the company then needs insurance to cover the potential costs of an outage; if the savings from removing a redundancy exceed the increase in insurance premium then it is actually efficient, otherwise it is a net negative. While an actuarian may make a mistake and underestimate the likelihood of an outage, they are far less incentivized to do so than the MBA looking for a bigger bonus.
    - unishark 5y ago
      If you can win a lawsuit against a company for a failure it is probably because it wasn't an externality but a contract or warranty agreement they have breached. This is present even in niches with minimal regulations. Data centers offer the highest uptime guarantees at the highest price tiers. People pay more for Toyotas, new or used, because of their reputation. Quality is a product feature. If MBA's want to decide if they can cut corners, there are already upsides and downsides, the calculation is something they need to make.
      - jjk166 5y ago
        Yeah, I'm saying add regulations so it is no longer an externality. Quality is a product feature when there is competition, monopolies don't suffer from cutting quality.
        unishark 5y ago
        whereas I'm saying it's not an externality even in the absence of manipulation by specific regulations, which you apparently agree with given your second sentence. I find that people have a tendency to be overly narrow in considering competition and declaring things monopolies. There are alternative ways to get tasks done that avoid relying on (and paying for) low-quality internet services if companies find it necessary.
        jjk166 5y ago
        An externality is a side effect or consequence of an industrial or commercial activity that affects other parties without this being reflected in the cost of the goods or services involved. Removing redundancy to increase profit margins affects other parties (ie users) without affecting the cost of the goods or services involved. If and only if the MBA's decision to increase risk to the user is reflected in the costs does it cease to be an externality. And we are specifically talking about an industry over-relying on a single provider of a service. If there were a variety of competing services, the entire point would be moot.
  - heterodoxxed 5y ago
    Right, but that's why we (ideally) put infrastructure costs under the control of an entity (the government) which doesn't have to operate within a system of profit and market competition.
- pklausler 5y ago
  What's this "emergency ... internet"? A hot-spot on a cellular telephone?
- 867-5309 5y ago
  but do all your emergency backups have emergency backups?
  - bigiain 5y ago
    My emergency backups have emergency emergency backups (but those do not have emergency emergency emergency backups).
mason55 5y ago
> Do you roll your own power grid? Do you roll your own ISP + telecoms network? > Roads, trains, water, electricity, internet I guess the difference here is that you're (mostly) talking about physical infra, which by definition must be local to where it's being used. We allow (enforce?) a monopoly on power distribution (and separate distribution from generation) because it doesn't make sense to have every power company run their own lines. But with that monopoly comes regulation. Digital services are different. The entire value prop is that you can have an infinite number and the marginal cost of "hooking up" a new customer is ~$0. This frequently leads to a natural winner-take-all market. One way to address this is to add regulation to digital services, saying that they must be up x% of the time or respond to incidents in y minutes or whatever. But another way to address it is to ensure it's easy for new companies to disrupt the incumbents if they are acting poorly. The first still leads to entrenched incumbents who act exactly as poorly as they can get away with. The second actually has a chance of pushing incumbents out, assuming the rules are being enforced. And now you've basically re-discovered the current American antitrust laws. As far as any individual company's best interests, like anything else in engineering, it's about risk vs. reward. What's the cost of having a backup CDN (cost of service, cost of extra engineering effort, opportunity cost of building that instead of something else, etc.) vs. the cost of the occasional fastly downtime? I have to imagine that for most companies the cost of being multi-CDN isn't worth what they lose with a little down time (or four hours of downtime every four years).
- ak217 5y ago
  > One way to address this is to add regulation to digital services, saying that they must be up x% of the time or respond to incidents in y minutes or whatever. This is good reasoning but I don't think it's possible to legislate service level objectives like that. > But another way to address it is to ensure it's easy for new companies to disrupt the incumbents if they are acting poorly. I agree but realistically there will be many cases when a company is far better at something than anyone else. I think the only way to avoid global infra single points of failure is competitive bidding and multi-source contracts, plus competitive pressure to force robustness (which already works quite well).
- hazmazlaz 5y ago
tw04 5y ago
> Do you roll your own power grid? I know plenty of people in Texas who will be buying solar panels and batteries after last winter. I will be doing the same. > Do you roll your own ISP + telecoms network? If I could magically get fiber directly to an IX I would gladly be my own ISP. I have confidence I would do as good a job or better than the ISPs I’ve had over the years (yes I realize having hundreds of thousands of customers to service is more difficult than a single home).
- filleduchaos 5y ago
  > I know plenty of people in Texas who will be buying solar panels and batteries after last winter. I will be doing the same. I have actually been in the position of having to rely on non-mains power all my life. It bloody sucks.
  - pengaru 5y ago
    How long is "all my life?" Because it seems like a relatively recent development that off-grid solar power solutions have become affordable and mature enough to not suck on average.
    - narism 5y ago
      Around 40% of the population of Nigeria (a country of 200+ million) does not have access to electricity. The per capita electricity consumption of Nigeria is _two orders of magnitude lower_ than the US.
      - pengaru 5y ago
        > Around 40% of the population of Nigeria (a country of 200+ million) does not have access to electricity. The per capita electricity consumption of Nigeria is _two orders of magnitude lower_ than the US. ... And how is this relevant?
        filleduchaos 5y ago

by yomly 5y ago

iso1210 5y ago

My datacentres have two sources of power (plus internal UPS), two main internet lines to two different exchange points (and half a dozen others), plenty of bottled water. At home I have emergency power, water and internet. If the trains stop I drive, if the car breaks I take the train.

411111111111111 5y ago
But having everything redundantly available costs money. While some redundancy is easy to justify... At some point it becomes hard when the MBA wants to cut costs so he gets a bigger bonus. There is even a competitive advantage in living with the risk, as you have less costs and overhead... Sure, you might have an outage once every x years for a few minutes... But that's obviously the fault of the development team, duh
- thayne 5y ago
  Which is one reason why things like roads, trains, water, electricty, etc. are so heavily regulated. To prevent the companies that hold monopolies over the infrastructure from cutting corners like that.
- jjk166 5y ago
  This is a classic example of an externality. You use regulations or lawsuits to force the costs back onto the decision makers. Make it so people can collect damages from outages, the company then needs insurance to cover the potential costs of an outage; if the savings from removing a redundancy exceed the increase in insurance premium then it is actually efficient, otherwise it is a net negative. While an actuarian may make a mistake and underestimate the likelihood of an outage, they are far less incentivized to do so than the MBA looking for a bigger bonus.
  - unishark 5y ago
    If you can win a lawsuit against a company for a failure it is probably because it wasn't an externality but a contract or warranty agreement they have breached. This is present even in niches with minimal regulations. Data centers offer the highest uptime guarantees at the highest price tiers. People pay more for Toyotas, new or used, because of their reputation. Quality is a product feature. If MBA's want to decide if they can cut corners, there are already upsides and downsides, the calculation is something they need to make.
    - jjk166 5y ago
      Yeah, I'm saying add regulations so it is no longer an externality. Quality is a product feature when there is competition, monopolies don't suffer from cutting quality.
      - unishark 5y ago
        whereas I'm saying it's not an externality even in the absence of manipulation by specific regulations, which you apparently agree with given your second sentence. I find that people have a tendency to be overly narrow in considering competition and declaring things monopolies. There are alternative ways to get tasks done that avoid relying on (and paying for) low-quality internet services if companies find it necessary.
        jjk166 5y ago
        An externality is a side effect or consequence of an industrial or commercial activity that affects other parties without this being reflected in the cost of the goods or services involved. Removing redundancy to increase profit margins affects other parties (ie users) without affecting the cost of the goods or services involved. If and only if the MBA's decision to increase risk to the user is reflected in the costs does it cease to be an externality. And we are specifically talking about an industry over-relying on a single provider of a service. If there were a variety of competing services, the entire point would be moot.
- heterodoxxed 5y ago
  Right, but that's why we (ideally) put infrastructure costs under the control of an entity (the government) which doesn't have to operate within a system of profit and market competition.
pklausler 5y ago
What's this "emergency ... internet"? A hot-spot on a cellular telephone?
867-5309 5y ago
but do all your emergency backups have emergency backups?
- bigiain 5y ago
  My emergency backups have emergency emergency backups (but those do not have emergency emergency emergency backups).

mason55 5y ago

> Do you roll your own power grid? Do you roll your own ISP + telecoms network? > Roads, trains, water, electricity, internet I guess the difference here is that you're (mostly) talking about physical infra, which by definition must be local to where it's being used. We allow (enforce?) a monopoly on power distribution (and separate distribution from generation) because it doesn't make sense to have every power company run their own lines. But with that monopoly comes regulation. Digital services are different. The entire value prop is that you can have an infinite number and the marginal cost of "hooking up" a new customer is ~$0. This frequently leads to a natural winner-take-all market. One way to address this is to add regulation to digital services, saying that they must be up x% of the time or respond to incidents in y minutes or whatever. But another way to address it is to ensure it's easy for new companies to disrupt the incumbents if they are acting poorly. The first still leads to entrenched incumbents who act exactly as poorly as they can get away with. The second actually has a chance of pushing incumbents out, assuming the rules are being enforced. And now you've basically re-discovered the current American antitrust laws. As far as any individual company's best interests, like anything else in engineering, it's about risk vs. reward. What's the cost of having a backup CDN (cost of service, cost of extra engineering effort, opportunity cost of building that instead of something else, etc.) vs. the cost of the occasional fastly downtime? I have to imagine that for most companies the cost of being multi-CDN isn't worth what they lose with a little down time (or four hours of downtime every four years).

ak217 5y ago
> One way to address this is to add regulation to digital services, saying that they must be up x% of the time or respond to incidents in y minutes or whatever. This is good reasoning but I don't think it's possible to legislate service level objectives like that. > But another way to address it is to ensure it's easy for new companies to disrupt the incumbents if they are acting poorly. I agree but realistically there will be many cases when a company is far better at something than anyone else. I think the only way to avoid global infra single points of failure is competitive bidding and multi-source contracts, plus competitive pressure to force robustness (which already works quite well).
hazmazlaz 5y ago

tw04 5y ago

> Do you roll your own power grid? I know plenty of people in Texas who will be buying solar panels and batteries after last winter. I will be doing the same. > Do you roll your own ISP + telecoms network? If I could magically get fiber directly to an IX I would gladly be my own ISP. I have confidence I would do as good a job or better than the ISPs I’ve had over the years (yes I realize having hundreds of thousands of customers to service is more difficult than a single home).

filleduchaos 5y ago
> I know plenty of people in Texas who will be buying solar panels and batteries after last winter. I will be doing the same. I have actually been in the position of having to rely on non-mains power all my life. It bloody sucks.
- pengaru 5y ago
  How long is "all my life?" Because it seems like a relatively recent development that off-grid solar power solutions have become affordable and mature enough to not suck on average.
  - narism 5y ago
    Around 40% of the population of Nigeria (a country of 200+ million) does not have access to electricity. The per capita electricity consumption of Nigeria is _two orders of magnitude lower_ than the US.
    - pengaru 5y ago
      > Around 40% of the population of Nigeria (a country of 200+ million) does not have access to electricity. The per capita electricity consumption of Nigeria is _two orders of magnitude lower_ than the US. ... And how is this relevant?
      - filleduchaos 5y ago