5 ms·

Reading this gave me an idea to extend this even further. What if the AI could look at your logbook history and suggest automations? For example, I have an auto

Reading this gave me an idea to extend this even further. What if the AI could look at your logbook history and suggest automations? For example, I have an automation that turns the lights on when it's dark based on a light sensor. It would be neat if AI could see "hey, you tend to manually turn on the lights when the light level is below some value, want to create an automation for that?"

balloob 3y ago
That's a good one. We might take it one step further and ask the user if they want to add a rule that certain rooms have a certain level of light. Although light level would tie it to a specific sensor. A smart enough system might also be able to infer this from the position of the sun + weather (ie cloudy) + direction of the windows in the room + curtains open/closed.
- blagie 3y ago
  I can write a control system easy enough to do this. I'm kind of an expert at that, for oddball reasons, and that's a trivial amount of work for me. The "smart enough" part, I'm more than smart enough for. What's not a trivial amount of work is figuring out how to integrate that into HA. I can guarantee that there is an uncountably infinite number of people like me, and very few people like you. You don't need to do my work for me; you just need to enable me to do it easily. What's really needed are decent APIs. If I go into Settings->Automation, I get a frustrating trigger/condition/action system. This should instead be: 1) Allow me to write (maximally declarative) Python / JavaScript, in-line, to script HA. To define "maximally declarative," see React / Redux, and how they trigger code with triggers 2) Allow my kid(s) to do the same with Blockly 3) Ideally, start to extend this to edge computing, where I can push some of the code into devices (e.g. integrating with ESPHome and standard tools like CircuitPython and MakeCode). This would have the upside of also turning HA into an educational tool for families with kids, much like Logo, Microsoft BASIC, HyperCard, HTML 2.0, and other technologies of yesteryear. Specifically controlling my lights to give constant light was one of the first things I wanted to do with HA, but the learning curve meant there was never enough time. I'm also a big fan of edge code, since a lot of this could happen much more gradually and discreetly. That's especially true for things with motors, like blinds, where a very slow stepper could make it silent.
  - windexh8er 3y ago
    1) You can basically do this today with Blueprints. There's also things like Pyscript [0]. 2) The Node-RED implementation in HA is phenomenal and kids can very easily use with a short introduction. 3) Again, already there. ESPHome is a first class citizen in HA. I feel like you've not read the HA docs [1,] or took the time to understand the architecture [2]. And, for someone who has more than enough self-proclaimed skills, this should be a very understandable system. [0] https://github.com/custom-components/pyscript https://github.com/custom-components/pyscript [1] https://www.home-assistant.io/docs/ https://www.home-assistant.io/docs/ [2] https://developers.home-assistant.io/ https://developers.home-assistant.io/
    - blagie 3y ago
      I think we are talking across each other. (1) You are correct that I have not read the docs or discovered everything there is. I have had HA for a few weeks now. I am figuring stuff out. I am finding the learning curve to be steep. (2) However, I don't think you understand the level of usability and integration I'm suggesting. For most users, "read the docs" or "there's a github repo somewhere" is no longer a sufficient answer. That worked fine for 1996-era Linux. In 2023, this needs to be integrated into the user interface, and you need discoverability and on-ramps. This means actually treating developers as customers. Take a walk through Micro:bit and MakeCode to understand what a smooth on-ramp looks like. Or the Scratch ecosystem. This contrasts with the macho "for someone who has more than enough self-proclaimed skills, this should be a very understandable system" -- no, it is not a very understandable system for me. Say what you will about my skills, that means it will also not be an understandable system for most e.g. kids and families. That said, if you're correct, a lot of this may just be a question of relatively surface user-interface stuff, configuration and providing good in-line documentation. (3) Skills are not universal. A martial artist might be a great athlete, but unless you're Kareem Abdul-Jabbar, that doesn't make you a great basketball player. My skills do include (1) designing educational experiences for kids; and (2) many semesters of graduate-level coursework on control theory. That's very different from being fluid at e.g. managing docker containers, which I know next to nothing about. My experience trying to add things to HA has not been positive. I spent a lot of time trying to add extensions which would show me a Zigbee connectivity map to debug some connectivity issues. None worked. I eventually found a page which told me this was already in the system *shrug*. I still don't know why the ones I installed didn't work, or where to get started debugging. For me, that was harder than doing a root-locus plot, implementing a system identification, designing a lag or lead compensator, or running the Bode obstacle course. Seriously. If I went into HA, and there was a Python console with clear documentation and examples, this would be built. That's my particular skills, but a userbase brings very diverse other skills.
sprobertson 3y ago
I've been working on something like this but it's of course harder than it sounds, mostly due to how few example use cases there are. A dumb false positive for yours might be "you tend to turn off the lights when the outside temperature is 50º" Anyone know of a database of generic automations to train on?
- hxypqr 3y ago
  Temperature and light may create illusions in LLM. A potential available solution to this is to establish a knowledge graph based on sensor signals, where LLM is used to understand the speech signals given by humans and then interpret these signals as operations on the graph using similarity calculations.
weebull 3y ago
Machine learning can tackle this for sure, but that's surely separate to LLMs. A language model deals with language, not logic.
- hxypqr 3y ago
  This is a very insightful viewpoint. In this situation, I believe it is necessary to use NER to connect the LLM module and the ML module.
- vidarh 3y ago
  At least higher-end LLMs are perfectly capable of making quite substantive logical inferences from data. I'd argue that an LLM is likely to be better than many other methods if the dataset is small, while other methods will be better once you're dealing with data that pushes the context window. E.g. I just tested w/ChatGPT, gave it a selection of instructions about playing music, the time and location, and a series of hypothetical responses, and then asked it to deduce what went right and wrong about the response, and it correctly deduced what the user intent I implied was a user that given the time (10pm) and place (the bedroom) and rejection of loud music possibly just preferred calmer music, but who at least wanted something calmer for bedtime. I also asked it to propose a set of constrained rules, and it proposed rules that'd certainly make me a lot happier by e.g. starting with calmer music if asked an unconstrained "play music" in the evening, and transition artists or genres more aggressively the more the user skips to try to find something the user will stick with. In other words, you absolutely can get an LLM to look at even very constrained history and get it to apply logic to try to deduce a better set of rules, and you can get it to produce rules in a constrained grammar to inject into the decision making process without having to run everything past the LLM. While given enough data you can train a model to try to produce the same result, one possible advantage of the above is that it's far easier to introspect. E.g. my ChatGPT session had it suggest a "IF <user requests to play music> AND <it is late evening> THEN <start with a calming genre>" rule. If it got it wrong (maybe I just disliked the specific artists I used in my example, or loved what I asked for instead), then correcting its mistake is far easier if it produces a set of readable rules, and if it's told to e.g. produce something that stays consistent with user-provided rules. (the scenario I gave it, btw. is based on my very real annoyance with current music recommendation that all to often does fail to take into account things like avoiding abrupt transitions, paying attention to the time of day and volume settings, and changing tack or e.g. asking questions if the user skips multiple tracks in quick succession)

balloob 3y ago

That's a good one. We might take it one step further and ask the user if they want to add a rule that certain rooms have a certain level of light. Although light level would tie it to a specific sensor. A smart enough system might also be able to infer this from the position of the sun + weather (ie cloudy) + direction of the windows in the room + curtains open/closed.

blagie 3y ago
I can write a control system easy enough to do this. I'm kind of an expert at that, for oddball reasons, and that's a trivial amount of work for me. The "smart enough" part, I'm more than smart enough for. What's not a trivial amount of work is figuring out how to integrate that into HA. I can guarantee that there is an uncountably infinite number of people like me, and very few people like you. You don't need to do my work for me; you just need to enable me to do it easily. What's really needed are decent APIs. If I go into Settings->Automation, I get a frustrating trigger/condition/action system. This should instead be: 1) Allow me to write (maximally declarative) Python / JavaScript, in-line, to script HA. To define "maximally declarative," see React / Redux, and how they trigger code with triggers 2) Allow my kid(s) to do the same with Blockly 3) Ideally, start to extend this to edge computing, where I can push some of the code into devices (e.g. integrating with ESPHome and standard tools like CircuitPython and MakeCode). This would have the upside of also turning HA into an educational tool for families with kids, much like Logo, Microsoft BASIC, HyperCard, HTML 2.0, and other technologies of yesteryear. Specifically controlling my lights to give constant light was one of the first things I wanted to do with HA, but the learning curve meant there was never enough time. I'm also a big fan of edge code, since a lot of this could happen much more gradually and discreetly. That's especially true for things with motors, like blinds, where a very slow stepper could make it silent.
- windexh8er 3y ago
  1) You can basically do this today with Blueprints. There's also things like Pyscript [0]. 2) The Node-RED implementation in HA is phenomenal and kids can very easily use with a short introduction. 3) Again, already there. ESPHome is a first class citizen in HA. I feel like you've not read the HA docs [1,] or took the time to understand the architecture [2]. And, for someone who has more than enough self-proclaimed skills, this should be a very understandable system. [0] https://github.com/custom-components/pyscript https://github.com/custom-components/pyscript [1] https://www.home-assistant.io/docs/ https://www.home-assistant.io/docs/ [2] https://developers.home-assistant.io/ https://developers.home-assistant.io/
  - blagie 3y ago
    I think we are talking across each other. (1) You are correct that I have not read the docs or discovered everything there is. I have had HA for a few weeks now. I am figuring stuff out. I am finding the learning curve to be steep. (2) However, I don't think you understand the level of usability and integration I'm suggesting. For most users, "read the docs" or "there's a github repo somewhere" is no longer a sufficient answer. That worked fine for 1996-era Linux. In 2023, this needs to be integrated into the user interface, and you need discoverability and on-ramps. This means actually treating developers as customers. Take a walk through Micro:bit and MakeCode to understand what a smooth on-ramp looks like. Or the Scratch ecosystem. This contrasts with the macho "for someone who has more than enough self-proclaimed skills, this should be a very understandable system" -- no, it is not a very understandable system for me. Say what you will about my skills, that means it will also not be an understandable system for most e.g. kids and families. That said, if you're correct, a lot of this may just be a question of relatively surface user-interface stuff, configuration and providing good in-line documentation. (3) Skills are not universal. A martial artist might be a great athlete, but unless you're Kareem Abdul-Jabbar, that doesn't make you a great basketball player. My skills do include (1) designing educational experiences for kids; and (2) many semesters of graduate-level coursework on control theory. That's very different from being fluid at e.g. managing docker containers, which I know next to nothing about. My experience trying to add things to HA has not been positive. I spent a lot of time trying to add extensions which would show me a Zigbee connectivity map to debug some connectivity issues. None worked. I eventually found a page which told me this was already in the system *shrug*. I still don't know why the ones I installed didn't work, or where to get started debugging. For me, that was harder than doing a root-locus plot, implementing a system identification, designing a lag or lead compensator, or running the Bode obstacle course. Seriously. If I went into HA, and there was a Python console with clear documentation and examples, this would be built. That's my particular skills, but a userbase brings very diverse other skills.

sprobertson 3y ago

I've been working on something like this but it's of course harder than it sounds, mostly due to how few example use cases there are. A dumb false positive for yours might be "you tend to turn off the lights when the outside temperature is 50º" Anyone know of a database of generic automations to train on?

hxypqr 3y ago
Temperature and light may create illusions in LLM. A potential available solution to this is to establish a knowledge graph based on sensor signals, where LLM is used to understand the speech signals given by humans and then interpret these signals as operations on the graph using similarity calculations.

weebull 3y ago

Machine learning can tackle this for sure, but that's surely separate to LLMs. A language model deals with language, not logic.

hxypqr 3y ago
This is a very insightful viewpoint. In this situation, I believe it is necessary to use NER to connect the LLM module and the ML module.
vidarh 3y ago
At least higher-end LLMs are perfectly capable of making quite substantive logical inferences from data. I'd argue that an LLM is likely to be better than many other methods if the dataset is small, while other methods will be better once you're dealing with data that pushes the context window. E.g. I just tested w/ChatGPT, gave it a selection of instructions about playing music, the time and location, and a series of hypothetical responses, and then asked it to deduce what went right and wrong about the response, and it correctly deduced what the user intent I implied was a user that given the time (10pm) and place (the bedroom) and rejection of loud music possibly just preferred calmer music, but who at least wanted something calmer for bedtime. I also asked it to propose a set of constrained rules, and it proposed rules that'd certainly make me a lot happier by e.g. starting with calmer music if asked an unconstrained "play music" in the evening, and transition artists or genres more aggressively the more the user skips to try to find something the user will stick with. In other words, you absolutely can get an LLM to look at even very constrained history and get it to apply logic to try to deduce a better set of rules, and you can get it to produce rules in a constrained grammar to inject into the decision making process without having to run everything past the LLM. While given enough data you can train a model to try to produce the same result, one possible advantage of the above is that it's far easier to introspect. E.g. my ChatGPT session had it suggest a "IF <user requests to play music> AND <it is late evening> THEN <start with a calming genre>" rule. If it got it wrong (maybe I just disliked the specific artists I used in my example, or loved what I asked for instead), then correcting its mistake is far easier if it produces a set of readable rules, and if it's told to e.g. produce something that stays consistent with user-provided rules. (the scenario I gave it, btw. is based on my very real annoyance with current music recommendation that all to often does fail to take into account things like avoiding abrupt transitions, paying attention to the time of day and volume settings, and changing tack or e.g. asking questions if the user skips multiple tracks in quick succession)