17 ms·

California power outage triggers chaos in science labs

264 points by McKayDavis 7y ago

atonse 7y ago
Maybe these kinds of things will push us more towards decentralized power generation like solar and wind, and bigger lab-wide or campus-wide battery battery backups?
- mc32 7y ago
  No, it's going to lead to even more regulation which in turn will constrain what PG&E can do even further. At this point, it should just become a state owned and operated enterprise and drop the dressing of being private. Once it does that, we'll come to appreciate their service as much as we appreciate the schedulekeeping of the MUNI.
  - xxpor 7y ago
    All utilities should.
  - baddox 7y ago
    PG&E has a government-granted monopoly to provide its utilities to its coverage region. That makes it indistinguishable from a state-owner enterprise, except that there are probably even more people reducing efficiency by extracting value for themselves.
    - SamReidHughes 7y ago
      Extraction of value means increased efficiency is incentivized. Obvious caveats about cutting corners with brush removal but also the shared responsibility of homes built in dangerous places apply.
      - baddox 7y ago
        Where would that incentive come from? Not from the threat of competitors, because no competitors are allowed to exist. There would perhaps be some slight incentive due to the slight elasticity of demand for electricity (i.e. people can probably reduce their energy usage in response to higher prices), but I imagine that elasticity is quite low indeed. It’s more likely that the more significant incentives would be other things, like, oh, not maintaining equipment and eventually causing wildfires.
        SamReidHughes 7y ago
        Increased efficiency increases profits.
        baddox 7y ago
        Increasing efficiency requires significant investment, which decreases profits now. And why do that now, when you have no competitors?
        SamReidHughes 7y ago
        How do they have shareholders?
- jjoonathan 7y ago
  Maybe in California. Most electric utilities in most states aren't regularly given insane contradictory legal directives that can only be reconciled by cutting power. This is not normal.
  - Blackthorn 7y ago
    PG&E is the poster child for regulatory capture. They're only in this mess because they've dragged their feet for so long on making critical upgrades and fixes.
    - jjoonathan 7y ago
      The way I hear it they were required to make the critical upgrades and forbidden from charging high enough rates to pay for it. If this is true, then the people imposing those two constraints have only themselves to blame for the blackouts. Regulatory capture is a thing that happens and is a problem, but this seems to be a case of bad regulation, not regulatory capture.
      - rodgerd 7y ago
        > If this is true, then the people imposing those two constraints have only themselves to blame for the blackouts. This is not true. PG&E have been returning phenominal divideds. They have chosen to prefer to dump their asset management budget into dividends.
        jjoonathan 7y ago
        It really doesn't look like their dividends were atypical before they went to 0 on Dec 2017: https://ycharts.com/companies/PCG/dividend_yield https://ycharts.com/companies/PCG/dividend_yield https://ycharts.com/companies/SO/dividend_yield https://ycharts.com/companies/SO/dividend_yield https://ycharts.com/companies/DUK/dividend_yield https://ycharts.com/companies/DUK/dividend_yield
        7y ago
- em3rgent0rdr 7y ago
  Or Advanced Small Modular Nuclear Reactors.
- baddox 7y ago
  Or even just a remotely competent group of people running the electricity grid.
- BurningFrog 7y ago
  Maybe this is the watershed moment when society finally becomes rational. But I wouldn't count on it :)
- mxuribe 7y ago
  This is interesting because if this happened in other states or countries, i might disagree with you...but considering it happened in California - where much innovation happens - i wonder how many existing and start-up firms will look to innovate on decentralized power generation even if only just for their own itch to scratch. I'm hoping that firms in California get pissed off enough to really increase the push for more decentralized modes of power...so instead of having engineers figure out how to invade our privacy or click online ads more, perhaps they can work on decentralized, reliable, and yes cheaper power generation.
driverdan 7y ago
If you have work that can be destroyed by a power outage you should plan accordingly. Power outages are a normal thing and can happen at any time, especially in areas with earthquake and fire risks. Not preparing is foolish.
- thrower123 7y ago
  Baffling that they couldn't have a diesel generator handy. One can rent an inverter, like for portable lighting units or a Lincoln welder, pretty readily and for relatively inconsequential sums in most cases.
  - techdevangelist 7y ago
    They are easy to acquire, as long as everyone else doesn’t need one at the same time. Heck even just securing a diesel delivery during a widespread event can be impossible, which is why you need some onsite for multiple days (if possible). Sadly you need to plan for these well in advance and be willing to sink significant resources.
    - thrower123 7y ago
      This is probably the reality now. Unless we're going to make the investments to actually maintain and upgrade the grid and infrastructure, preparedness and whatever amount of self-sufficiency that can be achieved is going to be necessary. It is a horrible cliche, but failing to prepare is preparing to fail. Possibly this is a case of California climate biting them. A lot of the big research universities in the Northeast have steam plants, originally for heat, but often with cogen facilities as well. Usually not much, a meg or two, but it is something.
  - trothamel 7y ago
    I suspect in most cases, those prices assume that demand is pretty low. When you have a regional power outage, it seems likely that limited supply will raise the price of these backup power.
- 7y ago
mensetmanusman 7y ago
People will now experience rolling black outs in California just like in South Africa? Is this just because of the lawsuit?
- latch 7y ago
  s/now/again https://en.wikipedia.org/wiki/California_electricity_crisis https://en.wikipedia.org/wiki/California_electricity_crisis
- eplanit 7y ago
  Yes (IMHO). PG&E was found responsible for the biggest/worst fire(s) in recent history, and has since declared bankruptcy. They're giving us Californians a big middle finger. I mean really, suppose a fire were to start on the Berkeley campus -- that would not trigger a forest fire miles away.
  - SamReidHughes 7y ago
    It’s not the campus, it’s the transmission lines connecting to it.
  - adrianmonk 7y ago
    I wouldn't say it's as emotional as a middle finger. Businesses manage risk in different ways, and one of the types of risk is legal risk. Based on new information that this risk is MUCH larger than they had previously accounted for, they are now making adjustments for it. They can't afford another lawsuit like that one. They couldn't afford the one that already happened. So they are taking steps to prevent it.
  - anigbrowl 7y ago
    It might be a good idea to study the physical geography of the Bay Area and read up on the history of the Oakland Hills fire. The Berkeley Campus is much larger than most people are aware of and extends up into the surrounding hills.
- mattr47 7y ago
  Most countries that experience rolling blackouts have certain levels of corruption in a socialized government. All because of a law suit. Great job California.
exabrial 7y ago
I mean, don't live in a desert? Seems like an obvious conclusion
- aodin 7y ago
  The dangerous combination that warrants blackouts is high winds, dry brush, and exposed live wires. While this combination is more likely in a desert, it's not true of all. For instance, the Salt River Project that oversees electricity transmission in Arizona switched to building underground lines in the 1970s. https://www.azcentral.com/story/news/local/asked-answered/2014/07/14/asked-answered-power-lines-above-below-ground/12571659/ https://www.azcentral.com/story/news/local/asked-answered/20...
noodlesUK 7y ago
PG&E has been doing blackouts and brownouts more than basically any other electric company I’ve ever experienced anywhere in the world since I was a little kid. These blackouts may be bigger, but if you’re an institution the size of a university, you should have significant disaster preparedness infrastructure. I mean US universities run police forces! I’m sure they could find a way of designing redundant power into their systems. They might even save money in the long run, and wouldn’t need to worry about campus hospitals or public safety going down in an outage (I’m sure some of that is already taken care of, but why not go all the way).
- cft 7y ago
  Not really: I was recently in Juticalpa, Honduras (it's a major city, like SF) and we did not have light for three days. Similar experience in Rio Dulce, Guatemala. In the third world, doing science is very difficult, because the infrastructure is just not there.
- cbkeller 7y ago
  Many universities have "cogeneration" plants that produce electricity, steam for heating, and also hot water. However, they usually don't generate enough electricity to power the whole campus, so the administration chooses who gets it and who doesn't. If these sorts of outages are going to become more common, then the management is going to have to re-evaluate the cost of providing backup power for all labs vs the cost of lost research.
  - jumpingmice 7y ago
    The university of California at Berkeley has ample cogeneration capacity but they are tied to PG&E grid and so PG&E can still turn them off.
    - dbcooper 7y ago
      Thanks for the clarification; I was wondering what had happened to their cogeneration plant. I remember some time ago when the cogeneration plant failed and took out power in one building where many biological samples were stored at -80.
quotemstr 7y ago
It's fascinating how these PG&E threads are full of recriminations, but not solutions. The power went out for 2.6 million people (on 800k accounts). No natural disaster or unanticipated software bug caused this problem. The situation is purely a human coordination failure. The state is doing nothing except issuing sternly worded statements when it could be changing policy. This odd mood of angry, passive resignation is disturbing. This whole situation represents a loss of "social technology". Just as material technology is a bag of techniques for organizing matter into useful configurations, social technology is a bag of techniques for organizing people for a useful purpose. When a society loses a technology, be it material or social, it loses a capability. It becomes unable to do what used to come easily to it. For 120 years, California was able to keep the lights on. Now it can't. How is anyone supposed to not see this situation as a kind of foreboding regression? This new power grid unreliability is far from the only example of our struggling to do something that came easily to us 20, 30, or 40 years ago. We've definitely lost something, although it's hard to pin down exactly what.
- mullingitover 7y ago
  I don't understand this comment. The grid is as reliable as it's ever been, but PG&E has been strongly incentivized to prevent wildfires. They're reacting to that incentive by preemptively shutting down the grid. There's no way of seeing the future where they didn't do that and whole cities burned to the ground, but based on recent history it's very likely that that's the future we're avoiding.
  - quotemstr 7y ago
    The reliability of the grid is a combination of the technical reliability of the grid and the social ability to reliably coordinate the resources needed to operate the grid. You need both systems to work in order for electricity to flow. As you point out, there's nothing wrong with the grid on a technical level. But the social element failed, so electricity stopped flowing. When the electricity stops, it doesn't really matter whether the power is off because of a technical or a human problem. The lights go off either way. The wind that shut grid blew at 10MPH. (Granted, it was forecast at up to 45MPH.) Did we not see winds of this strength during the 120 years* that we didn't shut down the grid over human coordination failures? If PG&E had to shut down the grid to avoid fire in light to moderate winds, it was because the people operating the grid were unable to coordinate the resources needed to operate safely under these conditions. What we've seen here is a breakdown of human systems that used to work reliably. We used to be able to operate an electricity grid reliably in moderately windy conditions. Now we can't. That should worry everyone. * Yes, I know about Enron. That we've had two grid failures in 20 years due to human-coordination breakdown is a huge red flag. Edit: phrasing, note that the winds were forecast to be stronger than what we ended up seeing
proee 7y ago
This event will increase the sales of Tesla Powerwalls.
- deleted 7y ago
  [deleted]
- twblalock 7y ago
  Pretty sure most of the people who wanted those already bought them. Sales will increase on gas generators, not Powerwalls.
  - LeoPanthera 7y ago
    I can add my anecdata to your own: I will probably buy a Powerwall because of this incident.
    - michaelt 7y ago
      This was a 5-day power outage, right? Is your house energy-efficient enough that a Powerwall will run it for that long?
      - LeoPanthera 7y ago
        Our power was out for slightly less than 24 hours.
TomMckenny 7y ago
Note on the blackout map, the county in the dead center has continuous power and it has a municipal source. Likewise the 2000 crisis where LA's DWP customers had power when PG&E did not. Although I'm sure it makes me a "socialist" for pointing these things out, it almost seems that having private monopolies in utilities is a bad idea
- aodin 7y ago
  The issue at hand is dry brush and trees being blown across live wires and sparking wild fires. The blackouts are targeted at the areas where this is most likely to occur. Namely, not heavy urban areas. Although not as widely reported, SoCal Edison is also doing rolling blackouts in at-risk areas https://www.latimes.com/california/story/2019-10-10/l-a-faces-critical-fire-danger-possible-power-outages-as-santa-ana-winds-buffet-southern-california https://www.latimes.com/california/story/2019-10-10/l-a-face...
  - TomMckenny 7y ago
    The problem is lack of investment in the infrastructure. And that damage to customers or economies is not a concern compared to lawsuits damaging the bottom-line. In particular, if you only take into account the revenue made on a given day vs the likelihood of a lawsuit and nowhere is economic damage accounted for, then you will turn the power off a lot more often and a lot more widely than you should. Also, since SoCal Edison is geographically adjacent to the DWP but Edison is investor owned while DWP is municipal owned, I believe you are adding support to my point.
    - aodin 7y ago
      Exactly, the question we should all be asking is how it got to this point. I reckon you'll find a lot to blame with both the utilities and state government, and the existing incentives for both. However, to your point about LADWP, I think you're making an overeager classification. It's not a matter of being "geographic adjacent", but being liable for the infrastructure that would start wildfires in at-risk regions. Take a look at the difference in coverage areas of Californias utilities: https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service_Areas_Detail.pdf https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service... edit: it is tangential to this issue, but LADWP does not have a very glamorous history, either.
claudeganon 7y ago
I wonder at what point California will just nationalize their grid/energy infrastructure. These “markets” always end up as some private monopoly with all the corresponding terrible incentives around upgrades and maintenance. If it was actually controlled by the state, it could be mandated to build out renewable-centered, decentralized systems as well.
- chrisco255 7y ago
  Well, in order to nationalize their power they would first need to be a nation. To be honest I don't know why one giant utility for an entire state is a desirable thing. In most places I've lived in, each community runs their own utility company. I've had cooperatives, city owned utilities, and private utilities. The best experience by far was the co-op. Since the customers owned the utility, we would get the profits back as credits on our bills usually during the middle of summer when power usage is highest in Florida.
  - CameronNemo 7y ago
    >in order to nationalize their power they would first need to be a nation A nation is a social construct that California could certainly fit into. Hell, Jerry Brown called it one. Examples of nations: Palestine, the Kurdish nation, Tibet. The legal construct is called statehood. States can bind themselves into larger legal entities by giving up some of their own rights. Examples include: the US federal government, the European Union, and the United Kingdom.
    - chrisco255 7y ago
      Sorry, no. A nation is a legal construct, and California is not a sovereign nation. If it's a social construct then I'm founding my own nation tomorrow. I don't know anything about Jerry Brown but he doesn't sound very bright.
      - selimthegrim 7y ago
        I believe J. Biafra wrote a song or two you might want to look into
twblalock 7y ago
I'm glad I live in the city of Santa Clara, where I am served by Silicon Valley Power, one of few exceptions to the PG&E monopoly in northern California. Silicon Valley Power serves many homes in addition to large companies including Intel and Nvidia, and they do so with lower prices and better quality of service than PG&E. I will be very unhappy going back to PG&E if I move to another city. Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities.
- kerkeslager 7y ago
  > Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities. Is Silicon Valley Power not a government-granted monopoly? How does their business model differ?
  - twblalock 7y ago
    Silicon Valley Power is a nonprofit operated by the city government. It's a little island in the middle of the sea of the PG&E monopoly. I wouldn't consider it a monopoly in comparison to the massive scale of PG&E, even though it is operated by a city government. Statewide utilities are bloated and inefficient, and they abuse their monopoly power. PG&E acts like the only choices are to cause fires or turn off power when it gets windy -- it faces no pressure to find a better solution because there is no competition. Small-to-medium municipal utilities that serve smaller areas can serve their customers much better, as Silicon Valley Power demonstrates -- if people don't like it they can move to a neighboring city. In contrast, PG&E is almost impossible to avoid. California should give more cities and counties the ability to experiment with this model. Competition between municipal utilities is better than granting a single company a monopoly on a utility millions of people depend on.
    - deleted 7y ago
      [deleted]
lawrenceyan 7y ago
I go to UC Berkeley. My midterms were postponed so I feel like even though I should be complaining, I really can't. PG&E basically gave me a week extra to study.
dmode 7y ago
I have no idea why this shutdown was initiated. It had been much warmer and drier earlier in the summer. It is actually quite pleasant and cold these days, with morning dew. And no winds. I have no idea why these conditions warrant power shutoff
- lightedman 7y ago
  It was this time last year that an entire town got wiped off the California map and 80+ people died because of fires sparked by PG&E's horribly-maintained power lines. You think they want that to happen again?
- aodin 7y ago
  The intensity of these blackouts is driven not just by weather, but by a major change in risk assessment by the utilities. PG&E, the utility that services most of northern California, was sued by a number of insurance entities after it was found that their equipment was the cause of several major wildfires over the last few years. Most notably the 2018 Camp Fire, which was the deadliest and most expensive wildfire in California history. But most importantly, it's what didn't happen: the state of California did not bail out or grant PG&E immunity from these lawsuits. PG&E settled with the insurance entities and has since filed for bankruptcy protection, since the possible liability is larger than their market cap. The blackouts are a way to minimize further risk and liability. But I would also guess that the intensity of blackouts is a not so subtle middle finger by the utilities to the state government for its resigned treatment of these issues.
- rolltiide 7y ago
  > I have no idea why this shutdown was initiated. Because Jeffrey Skilling just got out of prison and he wants to hold California’s energy ransom for payment to a random bitcoin address
- danans 7y ago
  > It had been much warmer and drier earlier in the summer. Warmer yes, but dryer definitely not. Earlier in the summer there is still moisture left over from the previous rainy season. By mid fall it is much dryer, and the risk is increased by offshore winds coming from inland.
unethical_ban 7y ago
Surely, PG&E will learn their lesson during this outage and send crews across the state to trim brush along the transmission lines? Strengthen the towers? Uncharacteristic edit: Why the downvotes? It must be because you took the comment seriously and think the sentiment grossly naive, or heard the skepticism and thought it unwarranted?
- twblalock 7y ago
  They blew up an entire neighborhood in San Bruno ~9 years ago. They don't learn.
- anigbrowl 7y ago
  I'm good friends with a lineman, and his view is that they just haven't been willing to make sufficient investments in resources, staff, or training.
egdod 7y ago
Power outages happen sometimes in the normal course. They are not a new thing, and the fact that this one is planned doesn’t make it worse. If you have specimens that you cannot afford to have warming up, it’s inexcusable that you don’t already have a backup power source.
- lilyball 7y ago
  Power outages don't usually last this long.
  - fsloth 7y ago
    Any dime a dozen facility with any sort of need of continuous electricity has a backup generator. The point is not that they should have bought a new generator for this planned blackout. The point is they should have had it for decades already.
- batmansmk 7y ago
  You are talking about 5 to 8 days of power outage...With one day notice. Science labs are usually NOT part of disaster recovery plans as they are R&D.
- alloutblitz85 7y ago
  Lol what are you on? This was one days notice on a week's notice. Due largely in part because they neglected to maintain their infrastructure over the years so they can shave off operating costs - for better quarterly reports. This is not normal.
  - egdod 7y ago
    You know what can provide 8 days of backup power? A generator (along with periodic trips to the gas station). Which you should already have if your power needs are this critical.
deleted 7y ago
[deleted]
whyenot 7y ago
My university is not a top tier research university like UC Berkeley by any means, but our co-generation plant is able to power the whole campus in the case of a power cut. Even when the central plant gets knocked out (a squirrel did that last year), individual buildings have backup generators. The building where I work has three of them and all of our -80s are on back up power or could be quickly shifted over. That is several layers of redundancy. We have done a lot of planning for events like this. I'm a little surprised that a more research focused (and less resource starved) institution like Berkeley wasn't more prepared.
- Mathnerd314 7y ago
  A lot of big state universities are cash starved, or at least are acting like it. Education funding was a big target during the recession and hasn't really bounced back: https://www.cbpp.org/research/state-budget-and-tax/a-lost-decade-in-higher-education-funding https://www.cbpp.org/research/state-budget-and-tax/a-lost-de... Berkely has a few generators though so I think these particular complaints are due to cost tradeoff decisions made decades ago and a general lack of foresight to predict the utility power completely shutting down.
- joshvm 7y ago
  It depends a lot on individual departments. Most astro groups I've been in have had very competent sysadmins who aggressively lobby for fault tolerance and assume that the university proper is incompetent (the very best kind of angry greybeards). So all the server rooms have enough UPS capacity to gracefully kill servers, data is properly backed up and so on. At some places like MSSL we even had an onsite diesel generator becuase there were flight critical experiments going on all the time. Many other faculties don't need that level of resilience so they don't bother. I'm not sure what the situation is for wider engineering or bio/chem. I imagine there are serious health and safety concerns for things like fume hoods that must be able to evacuate gases in the event of a power failure.
- distant_hat 7y ago
  Oftentimes, multiple failures cascade. About 2 decades back at Ohio State University, there was some renovation work going on. Due to this some sensors were switched off. This was wintertime and heating was on. During a weekend something failed and temperatures in a few rooms went as high as 50C. These rooms were used to house mice and rats for several long running experiments and all of them died or had to be euthanized as the data was unusable. Some people lost years of work.
legulere 7y ago
Meanwhile in Germany the average duration of blackouts in 2017 was around 20 minutes per customer, in quest where it was pretty high because of storms. The frequency was 0.28 in 2017, which means you can expect every three years an outage. Source: https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualitaet/versorgungszuverlaessigkeit/stoerungsstatistik-2017 https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualita...
- Tepix 7y ago
  And we pay extremely high prices for electricity.. near $.30 per kWh
  - mschuster91 7y ago
    That's because consumers subsidize the big smelters and other industrial customers who don't pay the renewables-buildout tax (EEG Umlage). The generalmtax load on energy is immense.
fyfy18 7y ago
I think it is interesting to compare PG&E to how the electricity grid is owned and run in the UK. Many people here are saying they should be nationalised, well the UK did the opposite 30 years ago and it seems to be working fine. In the UK there is National Grid Electricity Transmission plc, which as the name suggests is responsible for the transmission infrastructure across the country. They are a publicly traded company, and the parent company owns electricity & gas distribution in the UK, and some in the US. There are then smaller companies who are responsible for maintaining more local infrastructure (as I understand it's still owned by National Grid, but they are responsible for fixing and upgrading it). As a consumer you can choose to buy electricity from a long list of companies, and easily switch between them. All that happens is a different company collects your meter reading and they send you a bill. A lot of these companies also own generation facilities (e.g. SSE, who are also publically traded) while some are just resellers (e.g. So Energy).
- agf 7y ago
  There are plenty of states where that's basically the system that are a mess, if not as bad as in California. The devil is in the details of how the post-privitization market is regulated.
- perl4ever 7y ago
  I'm in NY, and the gas & electric company is National Grid, and they do give you the option of choosing a supplier. I've always thought that there must be hidden pitfalls of selecting a different supplier, since National Grid would have the ability and incentive to tilt the playing field.
ChuckMcM 7y ago
This is such a screw up. Personally I think this is PG&E trying to negotiate with the State to have it indemnify them against wildfires so that they neither have to pay to maintain their infrastructure, nor pay out large damage claims when that infrastructure burns down a town or two. I really wish the PUC wasn't captured, otherwise we might be able to fix this problem.
- networkimprov 7y ago
  100 years of US govt forest policy provided the fuel, PG&E provided the sparks. Edit: this is not an anti-government rant. There is far more vegetation in the wildlands of California than is natural due to forest fire-fighting policy dating back 100 years.
  - azinman2 7y ago
    Can you explain more?
    - mdorazio 7y ago
      Probably that many places have pursued policies of preventing forest fires entirely when they are actually a natural part of ecological development. If you keep preventing forest fires, you end up with a lot more potential fuel buildup than if you let them burn once in a while. I'm sure someone will point out that CA does controlled burns on occasion, but it's not the same thing. Edit: here's some reading on the topic: https://www.npr.org/2018/09/27/649649316/fire-ecologists-say-more-fires-should-be-left-to-burn-so-why-arent-they https://www.npr.org/2018/09/27/649649316/fire-ecologists-say...
    - iscrewyou 7y ago
      Not OP but I think this is what he/she may be referring to: https://99percentinvisible.org/episode/built-to-burn/ https://99percentinvisible.org/episode/built-to-burn/ The article is good. But podcast itself is also good.
    - avocado4 7y ago
StanislavPetrov 7y ago
We're going to spend far in excess of 700 billion dollars to maintain our global military empire this year alone (not counting legacy costs for veterans and countless other huge, related expenses) while millions of Americans sit in the dark without power because we outsource our control of our power grid and other critical infrastructure to for-profit corporations. Hopefully some of those people sitting in the dark will give that some thought while the lights are out.
ethanpil 7y ago
California has the 5th highest rate per kWh[1]. (#2 excluding Alaske and Hawaii) Just like our sales tax rates, income tax rates, our gas prices (due to taxes), sanitation costs, ad nauseum. We pay excruciatingly high rates for everything, and get mindbogglingly low returns for the money. CA gov. and utilities are a textbook case in poor management and government out of control. Where does the money go? [1] https://www.chooseenergy.com/electricity-rates-by-state/ https://www.chooseenergy.com/electricity-rates-by-state/
- tintor 7y ago
  You get sunshine in return.
  - baud147258 7y ago
    And hobos
- mxuribe 7y ago
  I'll caveat that I've never lived in California...But in New Jersey, we have the same sorts of expensive pain that you just described...but at least your weather is better, and you have a tad more elbow room. ;-)
- bontaq 7y ago
  ...PG&E is a corporation. If anything, the CA gov should make it government owned and remove the profit motive.
  - ethanpil 7y ago
    It's barely a corporation. Just another monolithic utility with government mandated monopoly and no incentive to provide service or value.
TeMPOraL 7y ago
Reminds me of a story a certain physics PhD told me about certain very known university in Poland. One day some years ago, the university administration thought, "why are we wasting money keeping power on, running lights and aircon and God knows what, when the buildings are vacant?". They soon turned that thought into action, and when a long holiday break came, they just shut off the power to the buildings as soon as everyone left for the day. About a week later, the PhD I mentioned and his co-workers came back to the lab, saw some droplets on the glass, and almost had a heart attack. Turns out, after a week of being without aircon to dry and warm it, the air was just about to cross the dew point threshold, and if it did, it would kill the optics on a million-dollar research laser they recently got installed. I hear that since then, they're a little bit less pound-foolish.
- molteanu 7y ago
  Your story now reminds me of our new plant manager back in the days of the last financial crisis. He was newly appointed as the supreme leader at this automotive plant. Uncertainty was high, this being 2007, so people were let go, and costs reduced with any means possible. So he went into production and one day decided that it is too cold in the storage area. He orders his people to increase the temperature with a few degrees since that would save tons of money. People protest, he hears only "blaa-blaa", and his request is implemented. A few days later, it turns out that barrels upon barrels of chemicals in the storage area needed to be kept under a certain temperature. So the stuff solidified and it was now only good for the dump site. Out to garbage the chemicals went, new stuff was ordered, production was stopped until the new stuff arrived and temperature was restored back to normal after that.
  - thombat 7y ago
    Seems like an exemplary case for "Chesterton's fence": https://www.chesterton.org/taking-a-fence-down/ https://www.chesterton.org/taking-a-fence-down/ Often the most important comments in a codebase are signs on such fences, preserving hard-won knowledge of pitfalls in software and human processes.
    - 7y ago
mikorym 7y ago
Not to be insensitive, but for a South African this is a kind of LOL moment. We have had power outages mess with experiments and universities since at least the early 2000's. That of course and the arson that burned down auditoriums. But in terms of experiments, if you want your MSc or PhD then you better make sure that you can go a couple of days without power to the grid. If you are in a rural area, especially a previously "white" area as the pre-1994 NP government called it, then you need to have your own > 5000 L water tank. The town where I went to primary school had 9 days without water or electricity a year or two ago. Probably half of the town (it's a small town though) never have water between 9PM and the next morning since they simply turn of the pumps (whether there is electricity or not). The government officials also steal water to go sell for $1 per 100 L to areas where there is no running water. You can think of it as a kind of osmosis between previously affluent areas and areas that had been mud huts. If you think of it that way it doesn't sound that bad, actually, but I think the net position is not the issue, the real issue to me is the corruption and apathy within the government and the rate of improvement. Africa should be ahead of China in economic growth if you look at the youth percentage and potentially massive workforce. If you want to completely redesign an entire country's power grid and make it 100% solar/wind, then South Africa is the perfect candidate. That is if you could forget about politics for a moment.
Mathnerd314 7y ago
Here on page 682 (10.3.1D.19) "High-value specimen refrigerators, freezers, etc." are listed as "optional" for emergency power: https://www.orf.od.nih.gov/TechnicalResources/Documents/DRM/DRM1.4042419.pdf https://www.orf.od.nih.gov/TechnicalResources/Documents/DRM/... Clearly it's a big inconvenience but they're not in danger of losing any grants.
bavcyc 7y ago
Rather than answer several individual posts, I'll be lazy and do a larger post which involves a lot of simplification and hand waving. I apologize in advance for any errors, but this seemed like a good way to handle insomnia instead of my normal lurking. If something is unclear or you think wrong, let me know. Power Companies circa 1900 there were lots of power companies, I've seen pictures of urban areas where there were multiple circuits run on poles, how many companies tried to serve a certain area, I do not know. Through lobbying investor owned utilities (IOU), or in most cases a single IOU gained the rights to serve an area exclusive. IOUs concentrated, for the most part, on serving dense load concentrations. As such the US Govt implemented the Rural Electrification Act, so low density areas could be served. The 2 types of electric utilities are public power (municipal, REA, RUS, PPD, UD, etc) and for profit (IOU's typically). In most cases, the IOUs give something to the government in exchange for the IOU providing service exclusively, it might be a tax or it might be free street lights. Municipals tend to subsidize the local government in some way, either through returned dollars or free power (street lights, buildings, traffic lights). IOUs typically have a defined rate of return and are supervised by a governing entity of some sort. Vegetation Management The recent SERC compliance meeting had a good presentation on vegetation management from the utility perspective and another on enforcement trends. In my opinion a lot of the issues in the W US is the result of the policy 'no fire is good', the sand pile game I think illustrates the issue where the longer sand keeps from falling results in a larger collapses (see Yellowstone fire). Since the late 1980's the issues with not burning has been known, I had an ecology class where if I recall correctly that was discussed for a couple of days. Tree trimming and clearing out undergrowth is done on a regular basis when the utility has an easement, but especially in urban areas folks tend to plant trees too close to power lines or even worse encroach on the easement with buildings. Most utilities patrol transmission lines at least once a year if not twice or monthly, sometimes this is aerial and other times it is feet on the ground walking the line. As an aside, the NESC governs clearance of electric lines to stuff and how stuff should be built; the RUS has publications on line design if you want to read about it. Distribute Generation The electric grid in the US is divided into 3 areas, Eastern Interconnect, Texas and Western Interconnect; they all function essentially the same. If I have a generator connected to the grid, it has to synchronize to the grid before closing the breaker. If it is done correctly then there is very little mechanical stress on the generator, if done incorrectly then there is a large amount of mechanical stress on the generator. One mis-operation I know about involved the A and B phases being swapped during a re-wind, when the generator was closed in at commissioning it had a large bang/clunk and the breaker opened immediately. The generator then had to be examined, i.e. taken back apart, to figure out what went wrong and if it could be put back into service. If I have a generator partially supplying a facility (this can save a lot of money for an entity) and a fault happens on the grid then my goal is to protect the generator, so the generator will either shut off or island the facility while shedding load above the generator's capacity. This happens very quickly. One instance I know of, the urban area was supplied by transmission (aka remote generation), a 30 MW generator was the closest source, the utility had a fault because equipment misoperated and the generator was suddenly trying to supply all the power to that fault such that the generator protection operated and islanded the facility. It was no issue to close the grid interconnect back in (once it was ensured it was safe to do so) but the facility had to shed load to keep the generator running without causing electrical issues to load and damaging the generator. Once a facility is islanded and running on its own generation the phase angle is a don't care until it is time to synchronize back to the grid. As long as the facility can shed load to maintain frequency (there is a NERC standard on Under Frequency Load Shed if you want to read about it) and not ruin equipment by having a power quality issue. During dynamic studies for system stability, it can be observed that a generator will diverge from the system frequency phase angle but not trip off because it is isolated from the grid which requires verification that isolation is happening and the protection scheme will indeed work that way on the actual system. My observation is that most facilities, data centers and other processing facilities (refineries) tend to be the exception, concentrate on first costs when designing their electric infrastructure. It is possible to design a resilient system but it has a cost and it will not be utilized 100% until something goes wrong. And if you are doing research then that can be an issue as you may lose a large amount of data due to the power going out or possibly being sensitive to transients on the system, e.g. a switching operation on the transmission system affects the end user equipment. Even if you have redundant systems (and/or power supplies) it is possible to have single point of failures on your system. As well if you have enough local generation to supply your load, it may be more economical to not run 100% of your generation as the market price for electricity is cheaper than your cost of production (and there are folks who don't like idle assets, not realizing the greater benefit is not using it or only having to use it infrequently). One other aspect of distributed generation is the automatic separation of the DG when loss of voltage is detected on the grid side. Utilities do not want voltage on their system if they have an outage due to worker safety (and other reasons). Utility crews in hurricane areas will typically investigate if they hear a generator running when the power is out to an area to ensure it is not back feeding the distribution line. As a reminder keep your feet together if you are near a downed power line and hop away, or even better don't go near downed power lines.
tomohawk 7y ago
Who's John Galt?
dekhn 7y ago
all the SREs on the thread are wondering why scientists don't automatically mail their complex samples to their friends across the country as a backup. and the answer is.... individual PIs optimize for the easy case (that power stays on), not the disaster recovery scenario. There is no SRE thinking at the individual PI level.
dannykwells 7y ago
Berkeley, my post doc alma mater, really has fallen on hard times. I get that PGE caused this, but UCB should have reliable back up power for their labs. Full stop. You can see why they're fighting so hard in the CRISPR case. They're desperate to raise dough.
gwbas1c 7y ago
This is something I don't understand... Portable generators are so inexpensive that I'd think an institution like this could just keep some in a shed or garage, and shlep cords through windows for extended outages. I bought a top-of-the-line tiny Honda Inverter generator so I could keep my wife's breast milk frozen in case of an extended outage, but even something half the cost would be good enough to run a refrigerator. But, even then, if you plan for it, backup generators and Solar + Battery are very affordable! I recently installed a Powerwall purely for backup. Because I have solar, it's slightly more expensive than a standby generator, but with no maintenance or fuel I suspect it'll be a wash within a few years. (Backup generators need expensive maintenance every year.) Even Generac bought a Solar + Battery company because they know they can't compete, and their solution is nicer than my Powerwall! (The Generac solution includes a critical loads panel inside of it, unlike Powerwall where the critical loads panel is outside.)
pvaldes 7y ago
Curiously, a similar problem happened last week in the Institute of Natural Products and Agrobiology from the Consejo Superior de Investigaciones Científicas in Tenerife. Temperature of ultrafrozen samples raised to -75 Celsius and years of hard work and a collection of DNA reference samples for agriculture were about to vanish
secabeen 7y ago
This isn't the best sourced article, the sources here are all random people complaining on twitter. It would be much more interesting to hear from an informed spokesperson from UC Berkeley, with actual data on the percentage of STEM research buildings without generators, etc. etc. Even then, one of the sources did describe that they have an e-power plug in their lab, for the -80 freezer. That's pretty much how it's supposed to be. If they have additional important fridges, those should also be plugged into e-power. I'm sure it's possible there are some labs at Berkeley that may be compromised in backup power, there are a lot of old buildings and infrastructure there. That still doesn't mean that that experience is common, just that the people impacted are complaining loudly. Most research labs may be totally fine, and just waiting for things to come back.
cwhittle 7y ago
All of these big research universities in CA are on major and active fault lines. My old lab at Cal sat mere hundreds of feet from the Hayward fault that runs through campus. This is a proactive power shutdown when you know it's coming. If they can't an event with advanced warning, what will happen to hundreds of millions of dollars of frozen samples and research instruments when the unexpected earthquake hits any of these universities, not only disrupting power, but also disrupting structures? Incredibly foolish not to have at least reliable power backups and containments irrespective of what PGE is doing.

264 points by McKayDavis 7y ago

Maybe these kinds of things will push us more towards decentralized power generation like solar and wind, and bigger lab-wide or campus-wide battery battery backups?

mc32 7y ago
No, it's going to lead to even more regulation which in turn will constrain what PG&E can do even further. At this point, it should just become a state owned and operated enterprise and drop the dressing of being private. Once it does that, we'll come to appreciate their service as much as we appreciate the schedulekeeping of the MUNI.
- xxpor 7y ago
  All utilities should.
- baddox 7y ago
  PG&E has a government-granted monopoly to provide its utilities to its coverage region. That makes it indistinguishable from a state-owner enterprise, except that there are probably even more people reducing efficiency by extracting value for themselves.
  - SamReidHughes 7y ago
    Extraction of value means increased efficiency is incentivized. Obvious caveats about cutting corners with brush removal but also the shared responsibility of homes built in dangerous places apply.
    - baddox 7y ago
      Where would that incentive come from? Not from the threat of competitors, because no competitors are allowed to exist. There would perhaps be some slight incentive due to the slight elasticity of demand for electricity (i.e. people can probably reduce their energy usage in response to higher prices), but I imagine that elasticity is quite low indeed. It’s more likely that the more significant incentives would be other things, like, oh, not maintaining equipment and eventually causing wildfires.
      - SamReidHughes 7y ago
        Increased efficiency increases profits.
        baddox 7y ago
        Increasing efficiency requires significant investment, which decreases profits now. And why do that now, when you have no competitors?
        SamReidHughes 7y ago
        How do they have shareholders?
jjoonathan 7y ago
Maybe in California. Most electric utilities in most states aren't regularly given insane contradictory legal directives that can only be reconciled by cutting power. This is not normal.
- Blackthorn 7y ago
  PG&E is the poster child for regulatory capture. They're only in this mess because they've dragged their feet for so long on making critical upgrades and fixes.
  - jjoonathan 7y ago
    The way I hear it they were required to make the critical upgrades and forbidden from charging high enough rates to pay for it. If this is true, then the people imposing those two constraints have only themselves to blame for the blackouts. Regulatory capture is a thing that happens and is a problem, but this seems to be a case of bad regulation, not regulatory capture.
    - rodgerd 7y ago
      > If this is true, then the people imposing those two constraints have only themselves to blame for the blackouts. This is not true. PG&E have been returning phenominal divideds. They have chosen to prefer to dump their asset management budget into dividends.
      - jjoonathan 7y ago
        It really doesn't look like their dividends were atypical before they went to 0 on Dec 2017: https://ycharts.com/companies/PCG/dividend_yield https://ycharts.com/companies/PCG/dividend_yield https://ycharts.com/companies/SO/dividend_yield https://ycharts.com/companies/SO/dividend_yield https://ycharts.com/companies/DUK/dividend_yield https://ycharts.com/companies/DUK/dividend_yield
        7y ago
em3rgent0rdr 7y ago
Or Advanced Small Modular Nuclear Reactors.
baddox 7y ago
Or even just a remotely competent group of people running the electricity grid.
BurningFrog 7y ago
Maybe this is the watershed moment when society finally becomes rational. But I wouldn't count on it :)
mxuribe 7y ago
This is interesting because if this happened in other states or countries, i might disagree with you...but considering it happened in California - where much innovation happens - i wonder how many existing and start-up firms will look to innovate on decentralized power generation even if only just for their own itch to scratch. I'm hoping that firms in California get pissed off enough to really increase the push for more decentralized modes of power...so instead of having engineers figure out how to invade our privacy or click online ads more, perhaps they can work on decentralized, reliable, and yes cheaper power generation.

driverdan 7y ago

If you have work that can be destroyed by a power outage you should plan accordingly. Power outages are a normal thing and can happen at any time, especially in areas with earthquake and fire risks. Not preparing is foolish.

thrower123 7y ago
Baffling that they couldn't have a diesel generator handy. One can rent an inverter, like for portable lighting units or a Lincoln welder, pretty readily and for relatively inconsequential sums in most cases.
- techdevangelist 7y ago
  They are easy to acquire, as long as everyone else doesn’t need one at the same time. Heck even just securing a diesel delivery during a widespread event can be impossible, which is why you need some onsite for multiple days (if possible). Sadly you need to plan for these well in advance and be willing to sink significant resources.
  - thrower123 7y ago
    This is probably the reality now. Unless we're going to make the investments to actually maintain and upgrade the grid and infrastructure, preparedness and whatever amount of self-sufficiency that can be achieved is going to be necessary. It is a horrible cliche, but failing to prepare is preparing to fail. Possibly this is a case of California climate biting them. A lot of the big research universities in the Northeast have steam plants, originally for heat, but often with cogen facilities as well. Usually not much, a meg or two, but it is something.
- trothamel 7y ago
  I suspect in most cases, those prices assume that demand is pretty low. When you have a regional power outage, it seems likely that limited supply will raise the price of these backup power.
7y ago

mensetmanusman 7y ago

People will now experience rolling black outs in California just like in South Africa? Is this just because of the lawsuit?

latch 7y ago
s/now/again https://en.wikipedia.org/wiki/California_electricity_crisis https://en.wikipedia.org/wiki/California_electricity_crisis
eplanit 7y ago
Yes (IMHO). PG&E was found responsible for the biggest/worst fire(s) in recent history, and has since declared bankruptcy. They're giving us Californians a big middle finger. I mean really, suppose a fire were to start on the Berkeley campus -- that would not trigger a forest fire miles away.
- SamReidHughes 7y ago
  It’s not the campus, it’s the transmission lines connecting to it.
- adrianmonk 7y ago
  I wouldn't say it's as emotional as a middle finger. Businesses manage risk in different ways, and one of the types of risk is legal risk. Based on new information that this risk is MUCH larger than they had previously accounted for, they are now making adjustments for it. They can't afford another lawsuit like that one. They couldn't afford the one that already happened. So they are taking steps to prevent it.
- anigbrowl 7y ago
  It might be a good idea to study the physical geography of the Bay Area and read up on the history of the Oakland Hills fire. The Berkeley Campus is much larger than most people are aware of and extends up into the surrounding hills.
mattr47 7y ago
Most countries that experience rolling blackouts have certain levels of corruption in a socialized government. All because of a law suit. Great job California.

exabrial 7y ago

I mean, don't live in a desert? Seems like an obvious conclusion

aodin 7y ago
The dangerous combination that warrants blackouts is high winds, dry brush, and exposed live wires. While this combination is more likely in a desert, it's not true of all. For instance, the Salt River Project that oversees electricity transmission in Arizona switched to building underground lines in the 1970s. https://www.azcentral.com/story/news/local/asked-answered/2014/07/14/asked-answered-power-lines-above-below-ground/12571659/ https://www.azcentral.com/story/news/local/asked-answered/20...

noodlesUK 7y ago

PG&E has been doing blackouts and brownouts more than basically any other electric company I’ve ever experienced anywhere in the world since I was a little kid. These blackouts may be bigger, but if you’re an institution the size of a university, you should have significant disaster preparedness infrastructure. I mean US universities run police forces! I’m sure they could find a way of designing redundant power into their systems. They might even save money in the long run, and wouldn’t need to worry about campus hospitals or public safety going down in an outage (I’m sure some of that is already taken care of, but why not go all the way).

cft 7y ago
Not really: I was recently in Juticalpa, Honduras (it's a major city, like SF) and we did not have light for three days. Similar experience in Rio Dulce, Guatemala. In the third world, doing science is very difficult, because the infrastructure is just not there.
cbkeller 7y ago
Many universities have "cogeneration" plants that produce electricity, steam for heating, and also hot water. However, they usually don't generate enough electricity to power the whole campus, so the administration chooses who gets it and who doesn't. If these sorts of outages are going to become more common, then the management is going to have to re-evaluate the cost of providing backup power for all labs vs the cost of lost research.
- jumpingmice 7y ago
  The university of California at Berkeley has ample cogeneration capacity but they are tied to PG&E grid and so PG&E can still turn them off.
  - dbcooper 7y ago
    Thanks for the clarification; I was wondering what had happened to their cogeneration plant. I remember some time ago when the cogeneration plant failed and took out power in one building where many biological samples were stored at -80.

quotemstr 7y ago

It's fascinating how these PG&E threads are full of recriminations, but not solutions. The power went out for 2.6 million people (on 800k accounts). No natural disaster or unanticipated software bug caused this problem. The situation is purely a human coordination failure. The state is doing nothing except issuing sternly worded statements when it could be changing policy. This odd mood of angry, passive resignation is disturbing. This whole situation represents a loss of "social technology". Just as material technology is a bag of techniques for organizing matter into useful configurations, social technology is a bag of techniques for organizing people for a useful purpose. When a society loses a technology, be it material or social, it loses a capability. It becomes unable to do what used to come easily to it. For 120 years, California was able to keep the lights on. Now it can't. How is anyone supposed to not see this situation as a kind of foreboding regression? This new power grid unreliability is far from the only example of our struggling to do something that came easily to us 20, 30, or 40 years ago. We've definitely lost something, although it's hard to pin down exactly what.

mullingitover 7y ago
I don't understand this comment. The grid is as reliable as it's ever been, but PG&E has been strongly incentivized to prevent wildfires. They're reacting to that incentive by preemptively shutting down the grid. There's no way of seeing the future where they didn't do that and whole cities burned to the ground, but based on recent history it's very likely that that's the future we're avoiding.
- quotemstr 7y ago
  The reliability of the grid is a combination of the technical reliability of the grid and the social ability to reliably coordinate the resources needed to operate the grid. You need both systems to work in order for electricity to flow. As you point out, there's nothing wrong with the grid on a technical level. But the social element failed, so electricity stopped flowing. When the electricity stops, it doesn't really matter whether the power is off because of a technical or a human problem. The lights go off either way. The wind that shut grid blew at 10MPH. (Granted, it was forecast at up to 45MPH.) Did we not see winds of this strength during the 120 years* that we didn't shut down the grid over human coordination failures? If PG&E had to shut down the grid to avoid fire in light to moderate winds, it was because the people operating the grid were unable to coordinate the resources needed to operate safely under these conditions. What we've seen here is a breakdown of human systems that used to work reliably. We used to be able to operate an electricity grid reliably in moderately windy conditions. Now we can't. That should worry everyone. * Yes, I know about Enron. That we've had two grid failures in 20 years due to human-coordination breakdown is a huge red flag. Edit: phrasing, note that the winds were forecast to be stronger than what we ended up seeing

proee 7y ago

This event will increase the sales of Tesla Powerwalls.

deleted 7y ago
[deleted]
twblalock 7y ago
Pretty sure most of the people who wanted those already bought them. Sales will increase on gas generators, not Powerwalls.
- LeoPanthera 7y ago
  I can add my anecdata to your own: I will probably buy a Powerwall because of this incident.
  - michaelt 7y ago
    This was a 5-day power outage, right? Is your house energy-efficient enough that a Powerwall will run it for that long?
    - LeoPanthera 7y ago
      Our power was out for slightly less than 24 hours.

TomMckenny 7y ago

Note on the blackout map, the county in the dead center has continuous power and it has a municipal source. Likewise the 2000 crisis where LA's DWP customers had power when PG&E did not. Although I'm sure it makes me a "socialist" for pointing these things out, it almost seems that having private monopolies in utilities is a bad idea

aodin 7y ago
The issue at hand is dry brush and trees being blown across live wires and sparking wild fires. The blackouts are targeted at the areas where this is most likely to occur. Namely, not heavy urban areas. Although not as widely reported, SoCal Edison is also doing rolling blackouts in at-risk areas https://www.latimes.com/california/story/2019-10-10/l-a-faces-critical-fire-danger-possible-power-outages-as-santa-ana-winds-buffet-southern-california https://www.latimes.com/california/story/2019-10-10/l-a-face...
- TomMckenny 7y ago
  The problem is lack of investment in the infrastructure. And that damage to customers or economies is not a concern compared to lawsuits damaging the bottom-line. In particular, if you only take into account the revenue made on a given day vs the likelihood of a lawsuit and nowhere is economic damage accounted for, then you will turn the power off a lot more often and a lot more widely than you should. Also, since SoCal Edison is geographically adjacent to the DWP but Edison is investor owned while DWP is municipal owned, I believe you are adding support to my point.
  - aodin 7y ago
    Exactly, the question we should all be asking is how it got to this point. I reckon you'll find a lot to blame with both the utilities and state government, and the existing incentives for both. However, to your point about LADWP, I think you're making an overeager classification. It's not a matter of being "geographic adjacent", but being liable for the infrastructure that would start wildfires in at-risk regions. Take a look at the difference in coverage areas of Californias utilities: https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service_Areas_Detail.pdf https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service... edit: it is tangential to this issue, but LADWP does not have a very glamorous history, either.

claudeganon 7y ago

I wonder at what point California will just nationalize their grid/energy infrastructure. These “markets” always end up as some private monopoly with all the corresponding terrible incentives around upgrades and maintenance. If it was actually controlled by the state, it could be mandated to build out renewable-centered, decentralized systems as well.

chrisco255 7y ago
Well, in order to nationalize their power they would first need to be a nation. To be honest I don't know why one giant utility for an entire state is a desirable thing. In most places I've lived in, each community runs their own utility company. I've had cooperatives, city owned utilities, and private utilities. The best experience by far was the co-op. Since the customers owned the utility, we would get the profits back as credits on our bills usually during the middle of summer when power usage is highest in Florida.
- CameronNemo 7y ago
  >in order to nationalize their power they would first need to be a nation A nation is a social construct that California could certainly fit into. Hell, Jerry Brown called it one. Examples of nations: Palestine, the Kurdish nation, Tibet. The legal construct is called statehood. States can bind themselves into larger legal entities by giving up some of their own rights. Examples include: the US federal government, the European Union, and the United Kingdom.
  - chrisco255 7y ago
    Sorry, no. A nation is a legal construct, and California is not a sovereign nation. If it's a social construct then I'm founding my own nation tomorrow. I don't know anything about Jerry Brown but he doesn't sound very bright.
    - selimthegrim 7y ago
      I believe J. Biafra wrote a song or two you might want to look into

twblalock 7y ago

I'm glad I live in the city of Santa Clara, where I am served by Silicon Valley Power, one of few exceptions to the PG&E monopoly in northern California. Silicon Valley Power serves many homes in addition to large companies including Intel and Nvidia, and they do so with lower prices and better quality of service than PG&E. I will be very unhappy going back to PG&E if I move to another city. Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities.

kerkeslager 7y ago
> Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities. Is Silicon Valley Power not a government-granted monopoly? How does their business model differ?
- twblalock 7y ago
  Silicon Valley Power is a nonprofit operated by the city government. It's a little island in the middle of the sea of the PG&E monopoly. I wouldn't consider it a monopoly in comparison to the massive scale of PG&E, even though it is operated by a city government. Statewide utilities are bloated and inefficient, and they abuse their monopoly power. PG&E acts like the only choices are to cause fires or turn off power when it gets windy -- it faces no pressure to find a better solution because there is no competition. Small-to-medium municipal utilities that serve smaller areas can serve their customers much better, as Silicon Valley Power demonstrates -- if people don't like it they can move to a neighboring city. In contrast, PG&E is almost impossible to avoid. California should give more cities and counties the ability to experiment with this model. Competition between municipal utilities is better than granting a single company a monopoly on a utility millions of people depend on.
  - deleted 7y ago
    [deleted]

lawrenceyan 7y ago

I go to UC Berkeley. My midterms were postponed so I feel like even though I should be complaining, I really can't. PG&E basically gave me a week extra to study.

dmode 7y ago

I have no idea why this shutdown was initiated. It had been much warmer and drier earlier in the summer. It is actually quite pleasant and cold these days, with morning dew. And no winds. I have no idea why these conditions warrant power shutoff

lightedman 7y ago
It was this time last year that an entire town got wiped off the California map and 80+ people died because of fires sparked by PG&E's horribly-maintained power lines. You think they want that to happen again?
aodin 7y ago
The intensity of these blackouts is driven not just by weather, but by a major change in risk assessment by the utilities. PG&E, the utility that services most of northern California, was sued by a number of insurance entities after it was found that their equipment was the cause of several major wildfires over the last few years. Most notably the 2018 Camp Fire, which was the deadliest and most expensive wildfire in California history. But most importantly, it's what didn't happen: the state of California did not bail out or grant PG&E immunity from these lawsuits. PG&E settled with the insurance entities and has since filed for bankruptcy protection, since the possible liability is larger than their market cap. The blackouts are a way to minimize further risk and liability. But I would also guess that the intensity of blackouts is a not so subtle middle finger by the utilities to the state government for its resigned treatment of these issues.
rolltiide 7y ago
> I have no idea why this shutdown was initiated. Because Jeffrey Skilling just got out of prison and he wants to hold California’s energy ransom for payment to a random bitcoin address
danans 7y ago
> It had been much warmer and drier earlier in the summer. Warmer yes, but dryer definitely not. Earlier in the summer there is still moisture left over from the previous rainy season. By mid fall it is much dryer, and the risk is increased by offshore winds coming from inland.

unethical_ban 7y ago

Surely, PG&E will learn their lesson during this outage and send crews across the state to trim brush along the transmission lines? Strengthen the towers? Uncharacteristic edit: Why the downvotes? It must be because you took the comment seriously and think the sentiment grossly naive, or heard the skepticism and thought it unwarranted?

twblalock 7y ago
They blew up an entire neighborhood in San Bruno ~9 years ago. They don't learn.
anigbrowl 7y ago
I'm good friends with a lineman, and his view is that they just haven't been willing to make sufficient investments in resources, staff, or training.

egdod 7y ago

Power outages happen sometimes in the normal course. They are not a new thing, and the fact that this one is planned doesn’t make it worse. If you have specimens that you cannot afford to have warming up, it’s inexcusable that you don’t already have a backup power source.

lilyball 7y ago
Power outages don't usually last this long.
- fsloth 7y ago
  Any dime a dozen facility with any sort of need of continuous electricity has a backup generator. The point is not that they should have bought a new generator for this planned blackout. The point is they should have had it for decades already.
batmansmk 7y ago
You are talking about 5 to 8 days of power outage...With one day notice. Science labs are usually NOT part of disaster recovery plans as they are R&D.
alloutblitz85 7y ago
Lol what are you on? This was one days notice on a week's notice. Due largely in part because they neglected to maintain their infrastructure over the years so they can shave off operating costs - for better quarterly reports. This is not normal.
- egdod 7y ago
  You know what can provide 8 days of backup power? A generator (along with periodic trips to the gas station). Which you should already have if your power needs are this critical.

deleted 7y ago

[deleted]

whyenot 7y ago

My university is not a top tier research university like UC Berkeley by any means, but our co-generation plant is able to power the whole campus in the case of a power cut. Even when the central plant gets knocked out (a squirrel did that last year), individual buildings have backup generators. The building where I work has three of them and all of our -80s are on back up power or could be quickly shifted over. That is several layers of redundancy. We have done a lot of planning for events like this. I'm a little surprised that a more research focused (and less resource starved) institution like Berkeley wasn't more prepared.

Mathnerd314 7y ago
A lot of big state universities are cash starved, or at least are acting like it. Education funding was a big target during the recession and hasn't really bounced back: https://www.cbpp.org/research/state-budget-and-tax/a-lost-decade-in-higher-education-funding https://www.cbpp.org/research/state-budget-and-tax/a-lost-de... Berkely has a few generators though so I think these particular complaints are due to cost tradeoff decisions made decades ago and a general lack of foresight to predict the utility power completely shutting down.
joshvm 7y ago
It depends a lot on individual departments. Most astro groups I've been in have had very competent sysadmins who aggressively lobby for fault tolerance and assume that the university proper is incompetent (the very best kind of angry greybeards). So all the server rooms have enough UPS capacity to gracefully kill servers, data is properly backed up and so on. At some places like MSSL we even had an onsite diesel generator becuase there were flight critical experiments going on all the time. Many other faculties don't need that level of resilience so they don't bother. I'm not sure what the situation is for wider engineering or bio/chem. I imagine there are serious health and safety concerns for things like fume hoods that must be able to evacuate gases in the event of a power failure.
distant_hat 7y ago
Oftentimes, multiple failures cascade. About 2 decades back at Ohio State University, there was some renovation work going on. Due to this some sensors were switched off. This was wintertime and heating was on. During a weekend something failed and temperatures in a few rooms went as high as 50C. These rooms were used to house mice and rats for several long running experiments and all of them died or had to be euthanized as the data was unusable. Some people lost years of work.

legulere 7y ago

Meanwhile in Germany the average duration of blackouts in 2017 was around 20 minutes per customer, in quest where it was pretty high because of storms. The frequency was 0.28 in 2017, which means you can expect every three years an outage. Source: https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualitaet/versorgungszuverlaessigkeit/stoerungsstatistik-2017 https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualita...

Tepix 7y ago
And we pay extremely high prices for electricity.. near $.30 per kWh
- mschuster91 7y ago
  That's because consumers subsidize the big smelters and other industrial customers who don't pay the renewables-buildout tax (EEG Umlage). The generalmtax load on energy is immense.

fyfy18 7y ago

I think it is interesting to compare PG&E to how the electricity grid is owned and run in the UK. Many people here are saying they should be nationalised, well the UK did the opposite 30 years ago and it seems to be working fine. In the UK there is National Grid Electricity Transmission plc, which as the name suggests is responsible for the transmission infrastructure across the country. They are a publicly traded company, and the parent company owns electricity & gas distribution in the UK, and some in the US. There are then smaller companies who are responsible for maintaining more local infrastructure (as I understand it's still owned by National Grid, but they are responsible for fixing and upgrading it). As a consumer you can choose to buy electricity from a long list of companies, and easily switch between them. All that happens is a different company collects your meter reading and they send you a bill. A lot of these companies also own generation facilities (e.g. SSE, who are also publically traded) while some are just resellers (e.g. So Energy).

agf 7y ago
There are plenty of states where that's basically the system that are a mess, if not as bad as in California. The devil is in the details of how the post-privitization market is regulated.
perl4ever 7y ago
I'm in NY, and the gas & electric company is National Grid, and they do give you the option of choosing a supplier. I've always thought that there must be hidden pitfalls of selecting a different supplier, since National Grid would have the ability and incentive to tilt the playing field.

ChuckMcM 7y ago

This is such a screw up. Personally I think this is PG&E trying to negotiate with the State to have it indemnify them against wildfires so that they neither have to pay to maintain their infrastructure, nor pay out large damage claims when that infrastructure burns down a town or two. I really wish the PUC wasn't captured, otherwise we might be able to fix this problem.

networkimprov 7y ago
100 years of US govt forest policy provided the fuel, PG&E provided the sparks. Edit: this is not an anti-government rant. There is far more vegetation in the wildlands of California than is natural due to forest fire-fighting policy dating back 100 years.
- azinman2 7y ago
  Can you explain more?
  - mdorazio 7y ago
    Probably that many places have pursued policies of preventing forest fires entirely when they are actually a natural part of ecological development. If you keep preventing forest fires, you end up with a lot more potential fuel buildup than if you let them burn once in a while. I'm sure someone will point out that CA does controlled burns on occasion, but it's not the same thing. Edit: here's some reading on the topic: https://www.npr.org/2018/09/27/649649316/fire-ecologists-say-more-fires-should-be-left-to-burn-so-why-arent-they https://www.npr.org/2018/09/27/649649316/fire-ecologists-say...
  - iscrewyou 7y ago
    Not OP but I think this is what he/she may be referring to: https://99percentinvisible.org/episode/built-to-burn/ https://99percentinvisible.org/episode/built-to-burn/ The article is good. But podcast itself is also good.
  - avocado4 7y ago

StanislavPetrov 7y ago

We're going to spend far in excess of 700 billion dollars to maintain our global military empire this year alone (not counting legacy costs for veterans and countless other huge, related expenses) while millions of Americans sit in the dark without power because we outsource our control of our power grid and other critical infrastructure to for-profit corporations. Hopefully some of those people sitting in the dark will give that some thought while the lights are out.

ethanpil 7y ago

California has the 5th highest rate per kWh[1]. (#2 excluding Alaske and Hawaii) Just like our sales tax rates, income tax rates, our gas prices (due to taxes), sanitation costs, ad nauseum. We pay excruciatingly high rates for everything, and get mindbogglingly low returns for the money. CA gov. and utilities are a textbook case in poor management and government out of control. Where does the money go? [1] https://www.chooseenergy.com/electricity-rates-by-state/ https://www.chooseenergy.com/electricity-rates-by-state/

tintor 7y ago
You get sunshine in return.
- baud147258 7y ago
  And hobos
mxuribe 7y ago
I'll caveat that I've never lived in California...But in New Jersey, we have the same sorts of expensive pain that you just described...but at least your weather is better, and you have a tad more elbow room. ;-)
bontaq 7y ago
...PG&E is a corporation. If anything, the CA gov should make it government owned and remove the profit motive.
- ethanpil 7y ago
  It's barely a corporation. Just another monolithic utility with government mandated monopoly and no incentive to provide service or value.

TeMPOraL 7y ago

Reminds me of a story a certain physics PhD told me about certain very known university in Poland. One day some years ago, the university administration thought, "why are we wasting money keeping power on, running lights and aircon and God knows what, when the buildings are vacant?". They soon turned that thought into action, and when a long holiday break came, they just shut off the power to the buildings as soon as everyone left for the day. About a week later, the PhD I mentioned and his co-workers came back to the lab, saw some droplets on the glass, and almost had a heart attack. Turns out, after a week of being without aircon to dry and warm it, the air was just about to cross the dew point threshold, and if it did, it would kill the optics on a million-dollar research laser they recently got installed. I hear that since then, they're a little bit less pound-foolish.

molteanu 7y ago
Your story now reminds me of our new plant manager back in the days of the last financial crisis. He was newly appointed as the supreme leader at this automotive plant. Uncertainty was high, this being 2007, so people were let go, and costs reduced with any means possible. So he went into production and one day decided that it is too cold in the storage area. He orders his people to increase the temperature with a few degrees since that would save tons of money. People protest, he hears only "blaa-blaa", and his request is implemented. A few days later, it turns out that barrels upon barrels of chemicals in the storage area needed to be kept under a certain temperature. So the stuff solidified and it was now only good for the dump site. Out to garbage the chemicals went, new stuff was ordered, production was stopped until the new stuff arrived and temperature was restored back to normal after that.
- thombat 7y ago
  Seems like an exemplary case for "Chesterton's fence": https://www.chesterton.org/taking-a-fence-down/ https://www.chesterton.org/taking-a-fence-down/ Often the most important comments in a codebase are signs on such fences, preserving hard-won knowledge of pitfalls in software and human processes.
  - 7y ago

mikorym 7y ago

Not to be insensitive, but for a South African this is a kind of LOL moment. We have had power outages mess with experiments and universities since at least the early 2000's. That of course and the arson that burned down auditoriums. But in terms of experiments, if you want your MSc or PhD then you better make sure that you can go a couple of days without power to the grid. If you are in a rural area, especially a previously "white" area as the pre-1994 NP government called it, then you need to have your own > 5000 L water tank. The town where I went to primary school had 9 days without water or electricity a year or two ago. Probably half of the town (it's a small town though) never have water between 9PM and the next morning since they simply turn of the pumps (whether there is electricity or not). The government officials also steal water to go sell for $1 per 100 L to areas where there is no running water. You can think of it as a kind of osmosis between previously affluent areas and areas that had been mud huts. If you think of it that way it doesn't sound that bad, actually, but I think the net position is not the issue, the real issue to me is the corruption and apathy within the government and the rate of improvement. Africa should be ahead of China in economic growth if you look at the youth percentage and potentially massive workforce. If you want to completely redesign an entire country's power grid and make it 100% solar/wind, then South Africa is the perfect candidate. That is if you could forget about politics for a moment.

Mathnerd314 7y ago

Here on page 682 (10.3.1D.19) "High-value specimen refrigerators, freezers, etc." are listed as "optional" for emergency power: https://www.orf.od.nih.gov/TechnicalResources/Documents/DRM/DRM1.4042419.pdf https://www.orf.od.nih.gov/TechnicalResources/Documents/DRM/... Clearly it's a big inconvenience but they're not in danger of losing any grants.

bavcyc 7y ago

Rather than answer several individual posts, I'll be lazy and do a larger post which involves a lot of simplification and hand waving. I apologize in advance for any errors, but this seemed like a good way to handle insomnia instead of my normal lurking. If something is unclear or you think wrong, let me know. Power Companies circa 1900 there were lots of power companies, I've seen pictures of urban areas where there were multiple circuits run on poles, how many companies tried to serve a certain area, I do not know. Through lobbying investor owned utilities (IOU), or in most cases a single IOU gained the rights to serve an area exclusive. IOUs concentrated, for the most part, on serving dense load concentrations. As such the US Govt implemented the Rural Electrification Act, so low density areas could be served. The 2 types of electric utilities are public power (municipal, REA, RUS, PPD, UD, etc) and for profit (IOU's typically). In most cases, the IOUs give something to the government in exchange for the IOU providing service exclusively, it might be a tax or it might be free street lights. Municipals tend to subsidize the local government in some way, either through returned dollars or free power (street lights, buildings, traffic lights). IOUs typically have a defined rate of return and are supervised by a governing entity of some sort. Vegetation Management The recent SERC compliance meeting had a good presentation on vegetation management from the utility perspective and another on enforcement trends. In my opinion a lot of the issues in the W US is the result of the policy 'no fire is good', the sand pile game I think illustrates the issue where the longer sand keeps from falling results in a larger collapses (see Yellowstone fire). Since the late 1980's the issues with not burning has been known, I had an ecology class where if I recall correctly that was discussed for a couple of days. Tree trimming and clearing out undergrowth is done on a regular basis when the utility has an easement, but especially in urban areas folks tend to plant trees too close to power lines or even worse encroach on the easement with buildings. Most utilities patrol transmission lines at least once a year if not twice or monthly, sometimes this is aerial and other times it is feet on the ground walking the line. As an aside, the NESC governs clearance of electric lines to stuff and how stuff should be built; the RUS has publications on line design if you want to read about it. Distribute Generation The electric grid in the US is divided into 3 areas, Eastern Interconnect, Texas and Western Interconnect; they all function essentially the same. If I have a generator connected to the grid, it has to synchronize to the grid before closing the breaker. If it is done correctly then there is very little mechanical stress on the generator, if done incorrectly then there is a large amount of mechanical stress on the generator. One mis-operation I know about involved the A and B phases being swapped during a re-wind, when the generator was closed in at commissioning it had a large bang/clunk and the breaker opened immediately. The generator then had to be examined, i.e. taken back apart, to figure out what went wrong and if it could be put back into service. If I have a generator partially supplying a facility (this can save a lot of money for an entity) and a fault happens on the grid then my goal is to protect the generator, so the generator will either shut off or island the facility while shedding load above the generator's capacity. This happens very quickly. One instance I know of, the urban area was supplied by transmission (aka remote generation), a 30 MW generator was the closest source, the utility had a fault because equipment misoperated and the generator was suddenly trying to supply all the power to that fault such that the generator protection operated and islanded the facility. It was no issue to close the grid interconnect back in (once it was ensured it was safe to do so) but the facility had to shed load to keep the generator running without causing electrical issues to load and damaging the generator. Once a facility is islanded and running on its own generation the phase angle is a don't care until it is time to synchronize back to the grid. As long as the facility can shed load to maintain frequency (there is a NERC standard on Under Frequency Load Shed if you want to read about it) and not ruin equipment by having a power quality issue. During dynamic studies for system stability, it can be observed that a generator will diverge from the system frequency phase angle but not trip off because it is isolated from the grid which requires verification that isolation is happening and the protection scheme will indeed work that way on the actual system. My observation is that most facilities, data centers and other processing facilities (refineries) tend to be the exception, concentrate on first costs when designing their electric infrastructure. It is possible to design a resilient system but it has a cost and it will not be utilized 100% until something goes wrong. And if you are doing research then that can be an issue as you may lose a large amount of data due to the power going out or possibly being sensitive to transients on the system, e.g. a switching operation on the transmission system affects the end user equipment. Even if you have redundant systems (and/or power supplies) it is possible to have single point of failures on your system. As well if you have enough local generation to supply your load, it may be more economical to not run 100% of your generation as the market price for electricity is cheaper than your cost of production (and there are folks who don't like idle assets, not realizing the greater benefit is not using it or only having to use it infrequently). One other aspect of distributed generation is the automatic separation of the DG when loss of voltage is detected on the grid side. Utilities do not want voltage on their system if they have an outage due to worker safety (and other reasons). Utility crews in hurricane areas will typically investigate if they hear a generator running when the power is out to an area to ensure it is not back feeding the distribution line. As a reminder keep your feet together if you are near a downed power line and hop away, or even better don't go near downed power lines.

tomohawk 7y ago

Who's John Galt?

dekhn 7y ago

all the SREs on the thread are wondering why scientists don't automatically mail their complex samples to their friends across the country as a backup. and the answer is.... individual PIs optimize for the easy case (that power stays on), not the disaster recovery scenario. There is no SRE thinking at the individual PI level.

dannykwells 7y ago

Berkeley, my post doc alma mater, really has fallen on hard times. I get that PGE caused this, but UCB should have reliable back up power for their labs. Full stop. You can see why they're fighting so hard in the CRISPR case. They're desperate to raise dough.

gwbas1c 7y ago

This is something I don't understand... Portable generators are so inexpensive that I'd think an institution like this could just keep some in a shed or garage, and shlep cords through windows for extended outages. I bought a top-of-the-line tiny Honda Inverter generator so I could keep my wife's breast milk frozen in case of an extended outage, but even something half the cost would be good enough to run a refrigerator. But, even then, if you plan for it, backup generators and Solar + Battery are very affordable! I recently installed a Powerwall purely for backup. Because I have solar, it's slightly more expensive than a standby generator, but with no maintenance or fuel I suspect it'll be a wash within a few years. (Backup generators need expensive maintenance every year.) Even Generac bought a Solar + Battery company because they know they can't compete, and their solution is nicer than my Powerwall! (The Generac solution includes a critical loads panel inside of it, unlike Powerwall where the critical loads panel is outside.)

pvaldes 7y ago

Curiously, a similar problem happened last week in the Institute of Natural Products and Agrobiology from the Consejo Superior de Investigaciones Científicas in Tenerife. Temperature of ultrafrozen samples raised to -75 Celsius and years of hard work and a collection of DNA reference samples for agriculture were about to vanish

secabeen 7y ago

This isn't the best sourced article, the sources here are all random people complaining on twitter. It would be much more interesting to hear from an informed spokesperson from UC Berkeley, with actual data on the percentage of STEM research buildings without generators, etc. etc. Even then, one of the sources did describe that they have an e-power plug in their lab, for the -80 freezer. That's pretty much how it's supposed to be. If they have additional important fridges, those should also be plugged into e-power. I'm sure it's possible there are some labs at Berkeley that may be compromised in backup power, there are a lot of old buildings and infrastructure there. That still doesn't mean that that experience is common, just that the people impacted are complaining loudly. Most research labs may be totally fine, and just waiting for things to come back.

cwhittle 7y ago

All of these big research universities in CA are on major and active fault lines. My old lab at Cal sat mere hundreds of feet from the Hayward fault that runs through campus. This is a proactive power shutdown when you know it's coming. If they can't an event with advanced warning, what will happen to hundreds of millions of dollars of frozen samples and research instruments when the unexpected earthquake hits any of these universities, not only disrupting power, but also disrupting structures? Incredibly foolish not to have at least reliable power backups and containments irrespective of what PGE is doing.