9 ms·

Enphase Envoy-S data scraping (2016)

KaiserPro 3y ago
On of the joys of House Assistant, is that its a forcing function for getting everything interfaced into one place. There is a brilliant plugin for interfacing (https://www.home-assistant.io/integrations/enphase_envoy/ https://www.home-assistant.io/integrations/enphase_envoy/) The work exploring the APi is excellent, and a very good write up. You should still read it.
mcbishop 3y ago
As an Enphase inverter owner and installer... I'm excited about this blog post and their related posts! I assumed there was no way around Enphase's cloud for getting data. I like Enphase hardware (they also have batteries and smart EV chargers)... but I'm not a fan of their cloud / GUI experience.
- riddley 3y ago
  I wish i could see your face when you discover Home Assistant and its Enphase integration.
  - peddamat 3y ago
    the integration randomly stopped working for me recently. it now asks me to log into the enphase cloud or provide a token or something.
    - mcbishop 3y ago
      I assume the Home Assistant integration is through the Enphase cloud account. Herein lies the peril.
    - rcxdude 3y ago
      They've now made some level of online connection mandatory, as you need an oauth token to connect to the local interface, which you can only get online. That token does have an expiry time of a year, but still, it's a significant step back from where they were before, where you could get local access completely disconnected from the internet (and in fact, those who completely isolated it to prevent it from getting software updates still have this).
      - matthew1471 3y ago
        Might be of use : https://github.com/Matthew1471/Enphase-API/blob/main/Documentation/Wrappers/Python/README.adoc https://github.com/Matthew1471/Enphase-API/blob/main/Documen... The IQ Gateway authentication is done entirely offline - but you do have to have a device online (laptop, pi, mobile etc) to obtain a JWT to present to it. Installer tokens are 12 hours, Owner tokens are a year. Some endpoints are only accessible with roles higher than Owner however, see https://github.com/Matthew1471/Enphase-API/blob/main/Documentation/IQ%20Gateway%20API/Authentication.adoc https://github.com/Matthew1471/Enphase-API/blob/main/Documen.... for my scripts (available as the "examples") they're set to renew the tokens automatically where required.
        rcxdude 3y ago
        It is about as sensible a design as you could come up with while still tying the access to be gated by the manufacturer. I still don't really get why it's done this way: the stated reason of security against a previous owner of the system doesn't make sense: this can more easily be accomplished by being able to reset the password. (As an aside, assuming the system gets its time from NTP, I wonder if you could extend a token's access time indefinitely by returning a looping timestamp from a local server)
- deleted 3y ago
  [deleted]
serbrech 3y ago
I was very disappointed when I realized that the data showing from the app was 20min behind, and going through the cloud, while the Envoy box is connected to my local network and has a per-second stream. their documentation is close to inexistant too. There were a couple opensource projects, with some snippets of code to get inspiration from to connect and scrape the data directly from the local API. I did this and pushed data to grafana, but never took the time to share it. Since my last attempt, it seems that more projects showed up that did a better job than me making it reusable :) - https://github.com/petercable/solar-observatory https://github.com/petercable/solar-observatory - https://github.com/vk2him/Enphase-Envoy-mqtt-json https://github.com/vk2him/Enphase-Envoy-mqtt-json - https://github.com/Matthew1471/Enphase-API https://github.com/Matthew1471/Enphase-API
- mcbishop 3y ago
  Thanks so much for your note and these links! Can't wait to give it a shake.
  - matthew1471 3y ago
    Thanks for the shoutout to my wrapper and docs too :) My GitHub repo serves 2 purposes: 1. To provide a reusable wrapper (currently only Python, https://github.com/Matthew1471/Enphase-API/blob/main/Documentation/Wrappers/Python/README.adoc https://github.com/Matthew1471/Enphase-API/blob/main/Documen...) to talk to the LAN API, it's even installable via pip. 2. To document what the LAN API returns and all the URL endpoints (https://github.com/Matthew1471/Enphase-API/tree/main/Documentation/IQ%20Gateway%20API https://github.com/Matthew1471/Enphase-API/tree/main/Documen...). I would like to see more people using the wrapper so we have a common way of talking to the IQ Gateway (there's a lot of tightly coupled projects on GitHub), but most people seem more interested in the docs I provide and rolling their own wrapper. I always have plans to add more to it - I found multiple security vulnerabilities in the IQ Gateway and that allowed me to get a dump of the firmware - so there's plenty more URLs for me to write about :)
    - mcbishop 3y ago

On of the joys of House Assistant, is that its a forcing function for getting everything interfaced into one place. There is a brilliant plugin for interfacing (https://www.home-assistant.io/integrations/enphase_envoy/ https://www.home-assistant.io/integrations/enphase_envoy/) The work exploring the APi is excellent, and a very good write up. You should still read it.

mcbishop 3y ago

As an Enphase inverter owner and installer... I'm excited about this blog post and their related posts! I assumed there was no way around Enphase's cloud for getting data. I like Enphase hardware (they also have batteries and smart EV chargers)... but I'm not a fan of their cloud / GUI experience.

riddley 3y ago
I wish i could see your face when you discover Home Assistant and its Enphase integration.
- peddamat 3y ago
  the integration randomly stopped working for me recently. it now asks me to log into the enphase cloud or provide a token or something.
  - mcbishop 3y ago
    I assume the Home Assistant integration is through the Enphase cloud account. Herein lies the peril.
  - rcxdude 3y ago
    They've now made some level of online connection mandatory, as you need an oauth token to connect to the local interface, which you can only get online. That token does have an expiry time of a year, but still, it's a significant step back from where they were before, where you could get local access completely disconnected from the internet (and in fact, those who completely isolated it to prevent it from getting software updates still have this).
    - matthew1471 3y ago
      Might be of use : https://github.com/Matthew1471/Enphase-API/blob/main/Documentation/Wrappers/Python/README.adoc https://github.com/Matthew1471/Enphase-API/blob/main/Documen... The IQ Gateway authentication is done entirely offline - but you do have to have a device online (laptop, pi, mobile etc) to obtain a JWT to present to it. Installer tokens are 12 hours, Owner tokens are a year. Some endpoints are only accessible with roles higher than Owner however, see https://github.com/Matthew1471/Enphase-API/blob/main/Documentation/IQ%20Gateway%20API/Authentication.adoc https://github.com/Matthew1471/Enphase-API/blob/main/Documen.... for my scripts (available as the "examples") they're set to renew the tokens automatically where required.
      - rcxdude 3y ago
        It is about as sensible a design as you could come up with while still tying the access to be gated by the manufacturer. I still don't really get why it's done this way: the stated reason of security against a previous owner of the system doesn't make sense: this can more easily be accomplished by being able to reset the password. (As an aside, assuming the system gets its time from NTP, I wonder if you could extend a token's access time indefinitely by returning a looping timestamp from a local server)
deleted 3y ago
[deleted]

serbrech 3y ago

I was very disappointed when I realized that the data showing from the app was 20min behind, and going through the cloud, while the Envoy box is connected to my local network and has a per-second stream. their documentation is close to inexistant too. There were a couple opensource projects, with some snippets of code to get inspiration from to connect and scrape the data directly from the local API. I did this and pushed data to grafana, but never took the time to share it. Since my last attempt, it seems that more projects showed up that did a better job than me making it reusable :) - https://github.com/petercable/solar-observatory https://github.com/petercable/solar-observatory - https://github.com/vk2him/Enphase-Envoy-mqtt-json https://github.com/vk2him/Enphase-Envoy-mqtt-json - https://github.com/Matthew1471/Enphase-API https://github.com/Matthew1471/Enphase-API

mcbishop 3y ago
Thanks so much for your note and these links! Can't wait to give it a shake.
- matthew1471 3y ago
  Thanks for the shoutout to my wrapper and docs too :) My GitHub repo serves 2 purposes: 1. To provide a reusable wrapper (currently only Python, https://github.com/Matthew1471/Enphase-API/blob/main/Documentation/Wrappers/Python/README.adoc https://github.com/Matthew1471/Enphase-API/blob/main/Documen...) to talk to the LAN API, it's even installable via pip. 2. To document what the LAN API returns and all the URL endpoints (https://github.com/Matthew1471/Enphase-API/tree/main/Documentation/IQ%20Gateway%20API https://github.com/Matthew1471/Enphase-API/tree/main/Documen...). I would like to see more people using the wrapper so we have a common way of talking to the IQ Gateway (there's a lot of tightly coupled projects on GitHub), but most people seem more interested in the docs I provide and rolling their own wrapper. I always have plans to add more to it - I found multiple security vulnerabilities in the IQ Gateway and that allowed me to get a dump of the firmware - so there's plenty more URLs for me to write about :)
  - mcbishop 3y ago