4 ms·

18-month $220k Powerwall solar energy installation nightmare

59 points by martyvis 4y ago

driverdan 4y ago
This link doesn't work if you DNS block trackers. The URL redirects to https://tags.news.com.au https://tags.news.com.au which is a tracker. Do you have a direct link that works without a redirect?
- Semaphor 4y ago
  The request there is for https://tags.news.com.au/prod/adblock/adblock.js https://tags.news.com.au/prod/adblock/adblock.js which is blocked by uBo, the site still works fine for me.
gjsman-1000 4y ago
I have only one comment: He should’ve sued earlier. Inexcusable. Also, the fact that the Powerwalls have processing overloads with 10 connected? Give me a break. A Nintendo Switch can coordinate with up to 11 other Switches in Mario Kart for a LAN party without internet.
- superchroma 4y ago
  well, we've seen elon's move-fast-and-break-lots-of-things attitude, so this isn't a surprise
- wccrawford 4y ago
  - gjsman-1000 4y ago
    Unless you have proof he is lying - what incentive does he have to lie? Most people adapt and are terrified of suing megacorporations. Thinking he’s lying to damage the solar movement is conspiratorial thinking, and, arguably, denial over real problems.
  - pakitan 4y ago
    > I know bad situations happen, but these articles spin it into a nightmare that just doesn't make sense. If it was FUD, Tesla's PR department would be quick to point it out. Instead they are reaching out to customer for "compensation", as per the article.
    - belltaco 4y ago
      Tesla doesn't have a PR department. https://electrek.co/2020/10/06/tesla-dissolves-pr-department/ https://electrek.co/2020/10/06/tesla-dissolves-pr-department...
  - gryf 4y ago
    It's not really anti-solar. It's that most of the companies doing solar are trying to monetize it, sell people into schemes or oversell features or capacity. It's the wild west of technologies at the moment. I know two people going through solar shit shows at the moment as well. One solar controller that won't work unless it calls home which was sold as completely off grid and another that never worked yet they are being charged for it. Those are the only two people I know who have solar so that's 100% failure rate on references so far for me.
- Tempest1981 4y ago
  Possibly using the same json parser as GTA V? "How I cut GTA Online loading times by 70%" https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
daniel-thompson 4y ago
https://archive.ph/D8HM6 https://archive.ph/D8HM6
nells 4y ago
Game_Ender 4y ago
Summary: Tesla's Powerwall controller cannot handle 10 off grid batteries and sales should not of sold this system. He is running beyond the scale of that Tesla's controller unit can really handle: > Tesla soon worked out that the system was being overloaded due to having too many Powerwalls connected at once ... The gateway is being overloaded, reaching 100 per cent CPU and RAM usage. There’s too much info when they are all online. He is doing something very uncommon among customers, and Tesla apparently doesn't even try to make it work: > But while Tesla claims it can do off-grid 10-Powerwall setups, there are only a handful around the world – most are backed up by grid electricity, papering over any outages. ... “At the beginning there were actually only three 10-Powerwall installs in the world that were off-grid,” They also apparently have poor support for off grid firmware updates, possibly related to the lack of CPU and memory to both keep the lights on and upgrade at once: > “During these upgrade cycles, the system goes into this chaotic state that takes between four and eight hours, turning off and on 50 times,”
- thr0wawayf00 4y ago
  Somebody screwed up in the sales process then, either on the installer or vendor side. It's not acceptable to request a system in an unsupported configuration like this unless he was explicitly told beforehand "this isn't a supported configuration and we'll build it for you as long as you understand that".
  - throwawaysleep 4y ago
    In every company I have worked with, sales lies to get people to sign. It is a profession for naturally dishonest people.
    - MattPalmer1086 4y ago
      A company I used to work for had an interesting solution to this. There was an internal market, and all services used had to be paid for. Sales people could keep everything after paying the company a cut and all the costs of delivering what they promised the client. So if they over promised to get a sale, it all came out of their personal profit. Seemed to work quite well.
codeflo 4y ago
> The gateway is being overloaded, reaching 100 per cent CPU and RAM usage. There’s too much info when they are all online. Maybe there’s an n^2 algorithm somewhere, maybe the hardware is simply underspecced. Maintaining voltage stability in a system without mechanical inertia (like a generator would have) requires incredibly fast response times, and a system like that should have been verified to actually achieve those. From the damage that was caused, it sounds like there are no “hardware” fuses built into this system either. That alone should be cause for regulators to take the product off the market. What a joke.
- hinkley 4y ago
  It doesn’t have to be n² if one unit has to control the others. If a single unit requires base load + 10% per unit you could be in trouble at 7-8 units. That’s more than sufficient for most sales. If anything maybe they should have a special power wall for large installations, with a beefier CPU and sturdier components. It’s not uncommon in tech and even to an extent automotive for the premium model to stay one hardware revision ahead of the consumer unit and after a few recalls iron out the reliability problems, after the R&D is paid off on whales, bring it down market and introduce something new at the high end. That’s both technologically sound and plays on baser behaviors of the well to do (namely, classism). If normal people have the same thing I have then it’s time to sell it so I can feel special again.
- throw_a_grenade 4y ago
  > From the damage that was caused, it sounds like there are no “hardware” fuses built into this system either. That alone should be cause for regulators to take the product off the market. No, it just sounds like inverter, to which there are two devices connected, one which suddenly draws a lot of current (like starting power tool, possibly the one pictured in the article) and an "electronic device" with switching power supply. When the large current draw happens, the inverter tries to compensate which results in a lot of power transmitted over higher frequency (several tens kilohertz rather than usual 50 Hz). That higher freq bypasses input cap and fries switching power supply. Matthias Wandel did a video about his own problems with portable battery setup, which did the same: https://www.youtube.com/watch?v=c1QSn3i_IEo https://www.youtube.com/watch?v=c1QSn3i_IEo. This inverter is just not suitable for powering different devices. ISTM those different "offgrid" technologies are just not ready for prod.
thr0wawayf00 4y ago
Tech has developed a really bad habit of shipping half-baked MVPs as fully fledged products with powerful marketing behind them, and that's an increasingly dangerous combination. I'm not sure where it started but I see it lot in software dev because that ethos is pretty much baked into iterative product development methodology. Don't build out something until a current or potential customer that's worth the time needs it. After a while, it feels less like "iterative" product development and more like "highest bidder gets the feature" development.
- throwawaysleep 4y ago
  > After a while, it feels less like "iterative" product development and more like "highest bidder gets the feature" development. It does work that way. I work for a unicorn security firm that internally has the rule that a bug impacting a business that pays less than 10K a year cannot be fixed. They are to be strung along until they quit on us. Whether we are actually working on the issue we claim to be working on entirely depends on your ARR. If you are a customer with less than 10K ARR, your bug is not getting any attention from engineering, just endless promises from support.
cs702 4y ago
If the OP's description of the situation is accurate, Tesla is at fault here. The company should never have sold a configuration (12 batteries + 1 controller) that no one has been able to make work despite trying again and again over 18 months. The company should apologize publicly right now, and do whatever it takes to fix this family's problems as soon as possible. I feel sorry for the family.
- gryf 4y ago
  In reality Musk will call them a pedo or something, cut their service off and silence any complaints on Twitter...
- LightG 4y ago
  But hold on just a minute! ... Tesla are incredibly cool to not have a PR/Comms Department, so what you're suggesting is so last century. Please stop with the outdated ideas.
dymk 4y ago
I've heard good stories of Tesla Powerwall, but I've heard enough horror stories to scare me away from that company's RE systems for good. Browsing the "top" posts of r/TeslaSolar, you start to get an idea for how much of a pain in the ass is to deal with the company once things go pear shaped. I'm not sure if I'm surprised or not that Tesla engineers didn't stress test their system by putting like, 50 Powerwalls on a string and seeing if the system could handle it.
- throwawaysleep 4y ago
  Not bothering to troubleshoot customer problems seems to be a business strategy these days. One of my jobs is for a security firm that internally takes the attitude that if a small biz with a low ARR comes in with a critical problem, they are just to be strung along until they quit.
- Tempest1981 4y ago
  Does Tesla have a QA department, or is engineering expected to do this type of testing? I hear some companies got rid of QA to reduce costs. Maybe this could have been simulated purely in software. Not sure.
  - hinkley 4y ago
    I really miss my QA dept. I think devs voted to get rid of them because it was often QA and Product ganging up on Dev, but if you’ve never experienced Dev and QA ganging up on Product, you’ve missed out. Getting rid of QA meant that Product cannot be outvoted except by a walkout. It’s part of why I’m not as loud about DevOps becoming its own business unit. At least it’s 2 against 1 again.
- yakak 4y ago
  I find the whole layout perplexing, more like things went off into a marketing plan to show what can be done instead of doing something reliable. If it can't handle 10 batteries and can't handle updates without backup from the grid, why did they make/sell this setup? Two controllers with ~5 batteries would make a situation where one is active/normal and the other is being the grid so the reliability would be the same as whatever they promise everyone else.. Similarly, he doesn't actually need to be off the grid, his connection just doesn't meet his peak usage so the level of chaos seems to be just to be able to say this is an off grid setup?
ilamont 4y ago
Question for people who have installed solar in a property that's used by other people: Is it easy to manage and perform basic troubleshooting? A cousin proudly showed me her solar setup at a vacation home including the app showing her electricity savings and the various hardware components and all I could think of was the things that can go wrong at our own property when I am not around and my parents, kids, or siblings are using it. After serving as my extended family's "IT support" for decades and dealing with a neverending stream of headaches relating to wifi, printers, Nest, unresponsive PCs, TV settings, car apps, and so on, I'm wary of installing a major system that requires someone to be basically on call to deal with minor problems or major outages as described in TFA.
- solardev 4y ago
  For what it's worth, most solar installations aren't offgrid. They work alongside the regular grid so if the solar system (or a part of it) goes down, the house would still keep getting regular grid electricity.
danans 4y ago
> We were probably spending nearly $10,000 on electricity a year across the business and home,” he said. “We worked out we could do a large solar system which could deliver that 200-amp feed, be able to pay itself off and be off the grid.” $220K to offset a $10k/yr electric bill is a 22 year payback period, which is quite a long time if you are using energy bills as a rationale. The primary rationalization for storage when you have grid access already is grid backup, and the secondary rationalization is for energy price arbitrage. Backup is hard to value monetarily because it depends on how much not having electricity costs you. Arbitrage is easy to value, but doesn't pay back much unless you have extremely variable retail electric rates. > “The most confusing and contentious point for me is that Tesla seems to express that the off-grid Powerwalls cannot guarantee 100 per cent uptime … [but] this was never expressed to me during deployment, and had I known this, I would not have embarked on this journey.” For a $220k installation maybe they wouldn't have been incentivized to explain it. One mode of the grid behaves a bit like insurance, increasing reliability and masking variability. Going off grid when you have the option to stay on seems like an irrational choice.
- shalmanese 4y ago
  But $150K of it was subsidized so it's only a 7 year payback period.
  - danans 4y ago
    According to the article, the $150k was a loan, not subsidized. They have to pay it back.
  - Someone 4y ago
    Not subsidized, loaned. FTA: “Of that, $150,000 was from a South Australian government-backed solar loan, with another $70,000 in cash.” There might be some subsidy in getting an attractive interest rate, but AFAICT, the article doesn’t say anything about that.
- hinkley 4y ago
  There is something to be said both financially and emotionally about getting your operating costs to be lower. And making a big capex expenditure in a year when you saw a windfall of money would flatten out your tax burden. Of course here it’s not that simple because you’ll be replacing worn equipment at some point and you will have to spin proverbial plates to keep that working well. I suspect this is about 3 parts financially sound and 2 parts Captain Ahab if you look at the full math.
hinkley 4y ago
> The IT expert had the 10-battery system installed in early 2020 in order to power both his small business and family home from the property. Nobody thought to implement this as two separate installs? Or the dad insisted it had to be done his way and is now mad that what he wanted didn’t work after they told him not to go that way? They say doctors make the worst patients. I’m getting a hint of that here.

59 points by martyvis 4y ago

This link doesn't work if you DNS block trackers. The URL redirects to https://tags.news.com.au https://tags.news.com.au which is a tracker. Do you have a direct link that works without a redirect?

Semaphor 4y ago
The request there is for https://tags.news.com.au/prod/adblock/adblock.js https://tags.news.com.au/prod/adblock/adblock.js which is blocked by uBo, the site still works fine for me.

gjsman-1000 4y ago

I have only one comment: He should’ve sued earlier. Inexcusable. Also, the fact that the Powerwalls have processing overloads with 10 connected? Give me a break. A Nintendo Switch can coordinate with up to 11 other Switches in Mario Kart for a LAN party without internet.

superchroma 4y ago
well, we've seen elon's move-fast-and-break-lots-of-things attitude, so this isn't a surprise
wccrawford 4y ago
- gjsman-1000 4y ago
  Unless you have proof he is lying - what incentive does he have to lie? Most people adapt and are terrified of suing megacorporations. Thinking he’s lying to damage the solar movement is conspiratorial thinking, and, arguably, denial over real problems.
- pakitan 4y ago
  > I know bad situations happen, but these articles spin it into a nightmare that just doesn't make sense. If it was FUD, Tesla's PR department would be quick to point it out. Instead they are reaching out to customer for "compensation", as per the article.
  - belltaco 4y ago
    Tesla doesn't have a PR department. https://electrek.co/2020/10/06/tesla-dissolves-pr-department/ https://electrek.co/2020/10/06/tesla-dissolves-pr-department...
- gryf 4y ago
  It's not really anti-solar. It's that most of the companies doing solar are trying to monetize it, sell people into schemes or oversell features or capacity. It's the wild west of technologies at the moment. I know two people going through solar shit shows at the moment as well. One solar controller that won't work unless it calls home which was sold as completely off grid and another that never worked yet they are being charged for it. Those are the only two people I know who have solar so that's 100% failure rate on references so far for me.
Tempest1981 4y ago
Possibly using the same json parser as GTA V? "How I cut GTA Online loading times by 70%" https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339

daniel-thompson 4y ago

https://archive.ph/D8HM6 https://archive.ph/D8HM6

nells 4y ago

Game_Ender 4y ago

Summary: Tesla's Powerwall controller cannot handle 10 off grid batteries and sales should not of sold this system. He is running beyond the scale of that Tesla's controller unit can really handle: > Tesla soon worked out that the system was being overloaded due to having too many Powerwalls connected at once ... The gateway is being overloaded, reaching 100 per cent CPU and RAM usage. There’s too much info when they are all online. He is doing something very uncommon among customers, and Tesla apparently doesn't even try to make it work: > But while Tesla claims it can do off-grid 10-Powerwall setups, there are only a handful around the world – most are backed up by grid electricity, papering over any outages. ... “At the beginning there were actually only three 10-Powerwall installs in the world that were off-grid,” They also apparently have poor support for off grid firmware updates, possibly related to the lack of CPU and memory to both keep the lights on and upgrade at once: > “During these upgrade cycles, the system goes into this chaotic state that takes between four and eight hours, turning off and on 50 times,”

thr0wawayf00 4y ago
Somebody screwed up in the sales process then, either on the installer or vendor side. It's not acceptable to request a system in an unsupported configuration like this unless he was explicitly told beforehand "this isn't a supported configuration and we'll build it for you as long as you understand that".
- throwawaysleep 4y ago
  In every company I have worked with, sales lies to get people to sign. It is a profession for naturally dishonest people.
  - MattPalmer1086 4y ago
    A company I used to work for had an interesting solution to this. There was an internal market, and all services used had to be paid for. Sales people could keep everything after paying the company a cut and all the costs of delivering what they promised the client. So if they over promised to get a sale, it all came out of their personal profit. Seemed to work quite well.

codeflo 4y ago

> The gateway is being overloaded, reaching 100 per cent CPU and RAM usage. There’s too much info when they are all online. Maybe there’s an n^2 algorithm somewhere, maybe the hardware is simply underspecced. Maintaining voltage stability in a system without mechanical inertia (like a generator would have) requires incredibly fast response times, and a system like that should have been verified to actually achieve those. From the damage that was caused, it sounds like there are no “hardware” fuses built into this system either. That alone should be cause for regulators to take the product off the market. What a joke.

hinkley 4y ago
It doesn’t have to be n² if one unit has to control the others. If a single unit requires base load + 10% per unit you could be in trouble at 7-8 units. That’s more than sufficient for most sales. If anything maybe they should have a special power wall for large installations, with a beefier CPU and sturdier components. It’s not uncommon in tech and even to an extent automotive for the premium model to stay one hardware revision ahead of the consumer unit and after a few recalls iron out the reliability problems, after the R&D is paid off on whales, bring it down market and introduce something new at the high end. That’s both technologically sound and plays on baser behaviors of the well to do (namely, classism). If normal people have the same thing I have then it’s time to sell it so I can feel special again.
throw_a_grenade 4y ago
> From the damage that was caused, it sounds like there are no “hardware” fuses built into this system either. That alone should be cause for regulators to take the product off the market. No, it just sounds like inverter, to which there are two devices connected, one which suddenly draws a lot of current (like starting power tool, possibly the one pictured in the article) and an "electronic device" with switching power supply. When the large current draw happens, the inverter tries to compensate which results in a lot of power transmitted over higher frequency (several tens kilohertz rather than usual 50 Hz). That higher freq bypasses input cap and fries switching power supply. Matthias Wandel did a video about his own problems with portable battery setup, which did the same: https://www.youtube.com/watch?v=c1QSn3i_IEo https://www.youtube.com/watch?v=c1QSn3i_IEo. This inverter is just not suitable for powering different devices. ISTM those different "offgrid" technologies are just not ready for prod.

thr0wawayf00 4y ago

Tech has developed a really bad habit of shipping half-baked MVPs as fully fledged products with powerful marketing behind them, and that's an increasingly dangerous combination. I'm not sure where it started but I see it lot in software dev because that ethos is pretty much baked into iterative product development methodology. Don't build out something until a current or potential customer that's worth the time needs it. After a while, it feels less like "iterative" product development and more like "highest bidder gets the feature" development.

throwawaysleep 4y ago
> After a while, it feels less like "iterative" product development and more like "highest bidder gets the feature" development. It does work that way. I work for a unicorn security firm that internally has the rule that a bug impacting a business that pays less than 10K a year cannot be fixed. They are to be strung along until they quit on us. Whether we are actually working on the issue we claim to be working on entirely depends on your ARR. If you are a customer with less than 10K ARR, your bug is not getting any attention from engineering, just endless promises from support.

cs702 4y ago

If the OP's description of the situation is accurate, Tesla is at fault here. The company should never have sold a configuration (12 batteries + 1 controller) that no one has been able to make work despite trying again and again over 18 months. The company should apologize publicly right now, and do whatever it takes to fix this family's problems as soon as possible. I feel sorry for the family.

gryf 4y ago
In reality Musk will call them a pedo or something, cut their service off and silence any complaints on Twitter...
LightG 4y ago
But hold on just a minute! ... Tesla are incredibly cool to not have a PR/Comms Department, so what you're suggesting is so last century. Please stop with the outdated ideas.

dymk 4y ago

I've heard good stories of Tesla Powerwall, but I've heard enough horror stories to scare me away from that company's RE systems for good. Browsing the "top" posts of r/TeslaSolar, you start to get an idea for how much of a pain in the ass is to deal with the company once things go pear shaped. I'm not sure if I'm surprised or not that Tesla engineers didn't stress test their system by putting like, 50 Powerwalls on a string and seeing if the system could handle it.

throwawaysleep 4y ago
Not bothering to troubleshoot customer problems seems to be a business strategy these days. One of my jobs is for a security firm that internally takes the attitude that if a small biz with a low ARR comes in with a critical problem, they are just to be strung along until they quit.
Tempest1981 4y ago
Does Tesla have a QA department, or is engineering expected to do this type of testing? I hear some companies got rid of QA to reduce costs. Maybe this could have been simulated purely in software. Not sure.
- hinkley 4y ago
  I really miss my QA dept. I think devs voted to get rid of them because it was often QA and Product ganging up on Dev, but if you’ve never experienced Dev and QA ganging up on Product, you’ve missed out. Getting rid of QA meant that Product cannot be outvoted except by a walkout. It’s part of why I’m not as loud about DevOps becoming its own business unit. At least it’s 2 against 1 again.
yakak 4y ago
I find the whole layout perplexing, more like things went off into a marketing plan to show what can be done instead of doing something reliable. If it can't handle 10 batteries and can't handle updates without backup from the grid, why did they make/sell this setup? Two controllers with ~5 batteries would make a situation where one is active/normal and the other is being the grid so the reliability would be the same as whatever they promise everyone else.. Similarly, he doesn't actually need to be off the grid, his connection just doesn't meet his peak usage so the level of chaos seems to be just to be able to say this is an off grid setup?

ilamont 4y ago

Question for people who have installed solar in a property that's used by other people: Is it easy to manage and perform basic troubleshooting? A cousin proudly showed me her solar setup at a vacation home including the app showing her electricity savings and the various hardware components and all I could think of was the things that can go wrong at our own property when I am not around and my parents, kids, or siblings are using it. After serving as my extended family's "IT support" for decades and dealing with a neverending stream of headaches relating to wifi, printers, Nest, unresponsive PCs, TV settings, car apps, and so on, I'm wary of installing a major system that requires someone to be basically on call to deal with minor problems or major outages as described in TFA.

solardev 4y ago
For what it's worth, most solar installations aren't offgrid. They work alongside the regular grid so if the solar system (or a part of it) goes down, the house would still keep getting regular grid electricity.

danans 4y ago

> We were probably spending nearly $10,000 on electricity a year across the business and home,” he said. “We worked out we could do a large solar system which could deliver that 200-amp feed, be able to pay itself off and be off the grid.” $220K to offset a $10k/yr electric bill is a 22 year payback period, which is quite a long time if you are using energy bills as a rationale. The primary rationalization for storage when you have grid access already is grid backup, and the secondary rationalization is for energy price arbitrage. Backup is hard to value monetarily because it depends on how much not having electricity costs you. Arbitrage is easy to value, but doesn't pay back much unless you have extremely variable retail electric rates. > “The most confusing and contentious point for me is that Tesla seems to express that the off-grid Powerwalls cannot guarantee 100 per cent uptime … [but] this was never expressed to me during deployment, and had I known this, I would not have embarked on this journey.” For a $220k installation maybe they wouldn't have been incentivized to explain it. One mode of the grid behaves a bit like insurance, increasing reliability and masking variability. Going off grid when you have the option to stay on seems like an irrational choice.

shalmanese 4y ago
But $150K of it was subsidized so it's only a 7 year payback period.
- danans 4y ago
  According to the article, the $150k was a loan, not subsidized. They have to pay it back.
- Someone 4y ago
  Not subsidized, loaned. FTA: “Of that, $150,000 was from a South Australian government-backed solar loan, with another $70,000 in cash.” There might be some subsidy in getting an attractive interest rate, but AFAICT, the article doesn’t say anything about that.
hinkley 4y ago
There is something to be said both financially and emotionally about getting your operating costs to be lower. And making a big capex expenditure in a year when you saw a windfall of money would flatten out your tax burden. Of course here it’s not that simple because you’ll be replacing worn equipment at some point and you will have to spin proverbial plates to keep that working well. I suspect this is about 3 parts financially sound and 2 parts Captain Ahab if you look at the full math.

hinkley 4y ago

> The IT expert had the 10-battery system installed in early 2020 in order to power both his small business and family home from the property. Nobody thought to implement this as two separate installs? Or the dad insisted it had to be done his way and is now mad that what he wanted didn’t work after they told him not to go that way? They say doctors make the worst patients. I’m getting a hint of that here.