10 ms·
ChatGPT in an iOS Shortcut: Smart HomeKit Voice Assistant
- nobrains 4y agoThe best eye opener for me was how he programmed / primed ChatGPT using plain English. It's not even psuedocode. It's just human language (structured well).
- ma2rten 4y agoWhy does the title and beginning of the article talk about ChatGPT if he actually used InstructGPT? As someone working in the space, it's quite strange to observe how ChatGPT blew up, when OpenAI had similar models for quite a while.
- mach1ne 4y agoAccessibility and presentation
- urbandw311er 4y agoThe mistitling is likely deliberate SEO to pull in traffic off the back of the popularity of ChatGPT
- d0mine 4y agoChatGPT plays the role of a generic word here (like Xerox, Kleenex, Google).
- swyx 4y agoarguably still inappropriate - you wouldnt put Bing in an ios shortcut and call it Google. we need to be less tolerant of lying to technical people because we fear they are too dumb to understand that software has variants or has layers
- DeWilde 4y agoWhere does it say he is using InstructGPT?
- moneywoes 4y agoSEO
- atorodius 4y agoThis is so cool, and ofc impressive. Also, I learned a lot about how powerful Shortcuts is. Every since I played with ChatGPT (just with the chat UI) I am kind of amazed every time I use Siri: it’s crazy how far we have come.
- schappim 4y agoI actually just did something similar with whisper.cpp, and hooked it up to GPT-3 Davinci-3 via the API, and then the answer piped via Microsoft's text to speech. The mic I'm using is a cheap USB omni-mic designed for conference calls.
- allisdust 4y agoDoes whisper model support real time transcription?
- parkuman 4y agowhisper.cpp is an open source project that has a feature to turn whisper into a real time transcription program, it works surprisingly well by feeding small segments of audio into whisper. take a look at their GitHub page!
- schappim 4y agoYou're right, and it also is apparently optimised for m1.
- unit_circle 4y agoHow long has it been running? Have you figured out anything to use it for beyond what other assistants can do yet? I'm very curious as I had the same idea but don't have time to implement...
- schappim 4y agoA few hours... It is constantly listening. In terms of what other assistance can't do, it can integrate w/ my desktop computer in a way that other assistance can't easily do. The M$ voices take things to another level. Apple has better quality voices for Siri on desktop, but they don't let you access them through the 'say' command.
- sorry_outta_gas 4y agoreminds of me https://infiniteconversation.com/ https://infiniteconversation.com/
- jkhdigital 4y agoRead this while sitting on the couch next to my 8 year-old son. He’s gotten into Scratch and Python recently and seemed very impressed when I explained the gist of the article. Can’t help but wonder how different software development will be when he enters the workforce in 10-15 years… and everything else for that matter.
- mymyairduster 4y agoIt's not lookg very bright, but it's still better than getting into something like graphic design that will be devalued even quicker
- godzillabrennus 4y agoThe only ways to earn money I can conceive of are owning robots and/or real estate. Going to be a ton of disruption as labor is collectively devalued further by the deflationary forces known as technological advances.
- moconnor 4y agoA brutal example of commoditise your complements: you can see an economy as capital, land and labour. Labour employs capital to extract value from land. The fear here is that capital replaces labour, leading to the value of labour dropping to zero and the value of capital and land increasing alone. But is labour being replaced, or augmented as it always is by capital? Industrialisation augmented labour more efficiently, but the value of labour has still increased in real terms and has not dropped to zero. That said, I would invest in capital and land now and not in labour.
- moconnor 4y agoIf it helps, a blacksmith who made their daily living producing nails could not conceive of ways to earn money post-industrialisation other than owning a mine or a factory either. But it turns out there are many unimaginable but better ways humans can contribute to our collective success than by banging hot metal into the same shape again and again in a post industrial economy.
- 4y ago
- urbandw311er 4y agoGosh. Can somebody help me to understand how a LLM has achieved this capability. I had thought that a LLM was essentially only doing completions under the hood using statistical likelihood of next words, wrapped in some lexical sugar/clever prompt mods. But evidently far more is going on here since, in OP’s example, some of its output (eg future time stamps) will not be present in its training data. Even with several billion parameters that seems impossible to me. (Clearly not!) Could somebody join the dots for me on the nature of whatever framework the LLM is embedded inside that allows it to achieve these behaviours that emulate logic and allow generation of unique content.
- Zee2 4y agoI mean, that's really the mystery of it. One of the most notable advances of recent LLMs has been emergent, one-shot, and highly-contextually-aware behaviors that seem to only manifest in models with extremely large numbers of parameters. Cleary the latent space of the model is able to encode some sort of reasoning around how time flows, typical information about houses, understanding how to transform and format information in a JSON snippet, etc. That's the "magic" of it all; amazing and powerful emergent behaviors from billions of weights. Of course, that's also the limitation. They're opaque and incredibly difficult (impossible?) to inspect and reason about.
- urbandw311er 4y agoThanks for the pointer. Any idea of how clever the wrapper around these things is? For example, would OP’s use case simply get forwarded to the neural network as one single input (string of words) or is there some clever preprocessing going on.
- tarvaina 4y agoThe only significant bit of preprocessing by the language model is tokenization. You can see how it works here: https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer
- punnerud 4y ago
- rcarmo 4y agoOne of the big eye openers here is not the use of OpenAI, but how dumb and limited Siri seems by comparison (especially its inability to grasp context). Apple, Amazon and (to a lesser extent) Google’s voice assistants never evolved past simple intent and keyword matching, and this hack puts their product teams to shame…
- chaoz_ 4y agoI really want to agree, but those are different tasks. Apple and Amazon attempt to focus on monetizable parts of NLP flows meaning optimization for narrow use cases (play music, order me a book, etc) and OpenAI can share an impressive chat bot, to just see how that plays out and generate extra PR. Pretty sure Apple and Amazon are capable of improving their voice-assistants, the question is whether they decide to invest into it. OpenAI is relatively young and is "quicker" than large orgs, as it's a startup with top-notch talent.
- krsdcbl 4y agoI think that's rather a case of hindsight as we now realise and agree that Alexa, Siri etc won't evolve past this crude monetization scheme. They began as very much what you describe though, to generate pr and make their underlying ecosystems seem more attractive and advanced.
- ignoramous 4y agoNot really. Amazon has already started working on "generalizable intelligence" [0] for Alexa inspired by GPT-3 (as they say so themselves) and released to production at least one model (viz Alexa Teacher Model) based on that effort: https://archive.is/gItZq https://archive.is/gItZq / https://www.amazon.science/blog/scaling-multilingual-virtual-assistants-to-1-000-languages https://www.amazon.science/blog/scaling-multilingual-virtual... [0] https://archive.is/UlCpM https://archive.is/UlCpM / https://www.amazon.science/blog/alexas-head-scientist-on-conversational-exploration-ambient-ai https://www.amazon.science/blog/alexas-head-scientist-on-con...
- trentearl 4y agoVery cool and impressive. Yesterday, I created a subreddit to discuss and curate these types of Ai use-cases, if anyone else is interested please join: https://www.reddit.com/r/AiAppDev/ https://www.reddit.com/r/AiAppDev/
- LeoPanthera 4y agoOpenAI charges per-word. You get an initial credit, but the "setup" prompt is so long that each request will be far more expensive than "normal" requests, since you'll be charged for the entire setup prompt with every single request.
- mrtksn 4y agoYep, have been playing with the 18USD API credit and indeed the cost is not trivial enough for casual use cases. That said, these new developments give us a new reason to have powerful computers. The last 10 years computers were way more powerful for most of the task for their conventional use cases but now computers are yet again way less powerful for the new frontier use cases. Apparently GPT-3 runs on 350GB VRAM, so it requires specialised hardware, so the best we can do today is to sneak peak into the future of on-device GPT-3 like LLMs.
- eknkc 4y agoThat input is 610 tokens. With the actual prompt and the response let's say you hit 1000 tokens. That would be 2 cents per question then. Not sure how often do people ask Siri questions but that seems like it would not be a significant cost for one user. If you were to build a service on top of it, that might be expensive though.
- PebblesRox 4y agoIn a home with only responsible adults, it would be fine. With kids it would be a repeat of the olden days before unlimited texting when the parents would get the cellphone bill at the end of the month. But I wonder how soon we’ll have fine-tuned or otherwise customized specialized LLMs that are cheaper to run? In the mean time this kind of thing is a really neat proof-of-concept of how ChatGPT’s abilities can be harnessed.
- KMnO4 4y agoGPT-3 is really impressive with the Davinci model, but this seems like overkill for the tasks here. You can save 10x [0] by just switching to the Curie model. More information on the models: https://subscription.packtpub.com/book/data/9781800563193/2/ch02lvl1sec07/introducing-davinci-babbage-curie-and-ada https://subscription.packtpub.com/book/data/9781800563193/2/... [0]: https://openai.com/api/pricing/ https://openai.com/api/pricing/
- IshKebab 4y agoCalled it two months ago! https://news.ycombinator.com/item?id=33892842 https://news.ycombinator.com/item?id=33892842 I can't find the actually query I tried but I told it a Typescript API and told it to generate a script to run in response to the query. That's much more powerful than JSON because it can pass data around from one thing to another e.g. "send a message to my girlfriend telling her my ETA". You just give it the maps and messages API and it figures it out. Edit: Found the query I tried back then! https://imgur.com/a/yfEJYKf https://imgur.com/a/yfEJYKf
- pelorat 4y agoI had the same idea as well. I run home-assistant for my home, and the dumbness of Google Assistant is infuriating. You have to manually add keywords and phrases to make it understand anything. So I was experimenting with having ChatGPT generate JSON instructions based on a prompt similar to this (though the prompt in the blog is way better than mine). The real power of these models are not the information that they can fuzzily regurgitate, but how they can be given instructions on how to manipulate unstructured data.
- odedbend 4y agoThis is great, how is it going to be affected by the pricing change that OpenAi doing those days?
- pyentropy 4y agoA decade of Siri research & development... overshadowed by a single clever GPT-3 prompt. Love it.
- toxicFork 4y agoTbh there's a lot of research that went into LLMs
- finneganscat 4y ago[dead]
- Waterluvian 4y agoLast week I asked Siri in my car for the name of a song with the lyrics, “hello darkness my old friend.” I know the song and title. I was just having a brain fart. Siri had zero ability to help me. She just repeated the currently playing song. ChatGPT would have been perfect, especially since it was a question that I could instantly verify an answer to. I want my Ship’s Computer in my car.
- capableweb 4y agoUsing Siri is an exercise in how easily frustrated you can be. Most of the commands of misunderstood, the rest of them ends up with the reply "Sorry, I can't help you with that while you're in the car", even when you ask the most basic questions. CarPlay in general feels like such an afterthought, so not that surprising that Siri sucks there too.
- wintermutestwin 4y ago>"Sorry, I can't help you with that while you're in the car" I was on the way to an important meeting that didn't have enough time to prepare for. I needed to read a very long blog post and so I pulled up the page before I got in the car and once I was on the highway, I asked Siri to "speak screen"and got this asinine response. WTH? This is the primary use case for having Siri read something to me. What idiot thought this was somehow necessary for safety? That day, Siri was the tipping point from Apple being a company that delighted me to one that infuriated me.
- d0mine 4y agoI’ve tried it with the exact prompt from your comment: «what is the song with the lyrics, “hello darkness my old friend.” » and Siri answered correctly with “The Sound of Silence.”
- Waterluvian 4y agoFair enough! It must have misheard me and then did its best guess with some of the words it did hear.
- noodlesUK 4y agoI feel like the logical next step is to plug something like this into something a bit more powerful than Siri Shortcuts. I expect you could easily hook this up to the home assistant API or similar and have it work properly programatically.
- silentguy 4y agoSomeone already tried it with homeassistant.io (https://mikegrant.org.uk/2022/12/22/gpt3-and-homeassistant.html https://mikegrant.org.uk/2022/12/22/gpt3-and-homeassistant.h...) . I predict that there will be a homeassistant integration module for openai soon.
- satvikpendem 4y agoChatGPT now has a paid plan (and a limited free plan), so I wonder how apps like these that integrate into ChatGPT are going to handle it, I assume they'd have to start being monetized as well. https://news.ycombinator.com/item?id=34476842 https://news.ycombinator.com/item?id=34476842
- cal85 4y agoCan someone help me understand: is ChatGPT essentially just using the GPT-3 text-davinci-003 under the hood, with a bunch of prompt-prefixing (similar to what is going on in this guy's Siri Shortcut)? Or is it using a significantly more powerful model?
- reustle 4y agoFrom what I understand, they refer to it as GPT 3.5. So, that plus the prompt prefixing as you mentioned.
- koalala 4y agoAFAIK that's exactly what it is.
- cal85 4y agoIf so, doesn’t that make it trivial for anyone to make a ChatGPT-like app using the GPT-3 API? If so, I would’ve expected several ChatGPT-like tools coming out in the days following GPT-3’s release. I don’t understand why it took so many months - and then, why it was OpenAI themselves that finally released the first interactive thing on top of it. It’s puzzling.
- jldugger 4y agoChatGPT is GPT-3.5 plus some human feedback reinforcement learning[1] to steer it away from the things that tanked Tay. Meaning they had a bunch humans test the thing out, rate responses, and incorporate that into future training. [1]: https://en.wikipedia.org/wiki/ChatGPT#Training
- anticensor 4y agoIt was either text-davinci-002-render or text-davinci-003-render
- deleted 4y ago[deleted]
- novakinblood 4y agoI’ve been slowly integrating more devices to my home assistant setup—most recent a garage door controller—and I thought I was done for now and content with the setup. After reading this I can’t help but plan out the next several weekends to try to replicate this in hass.
- junon 4y agoI've gone down this road before, setting up a protocol with ChatGPT. You can see some early revisions of it on a gist[0]. It instructed chatgpt to communicate via a protocol and for the most part it worked. It also bypassed a lot of the neutering openai has done as a side effect. Might be interesting for someone to play with. Since chatgpt doesn't have API access this was as far as I decided to take it. Feed the prompts in order, one after the other. If you do something cool with this, let me know! Just a tip if you want to modify it: Changing the rules or adding commands later on seems not to work very well at all. It starts to break the "fourth wall" and starts issuing responses without using the protocol framing. In my experimentation it worked best when establishing everything at the beginning. Changes to the prompts worked best when starting with new sessions. Also, don't expect it to remember things well. E.g. don't try to tell it to remember the last timer value, for example. It simple can't handle it, not entirely sure why. You'd need to rely on the external "software handler" for storage/retrieval. [0] https://gist.github.com/Qix-/4f3b3f249192caa140c95ce4c38a2327 https://gist.github.com/Qix-/4f3b3f249192caa140c95ce4c38a232...
- thegabriele 4y agoFunny, isn't this the exact current version of the "Egg" in the Black Mirror Christmas Special? The most self aware thing we could build nowadays to manage our homes. https://en.wikipedia.org/wiki/White_Christmas_(Black_Mirror) https://en.wikipedia.org/wiki/White_Christmas_(Black_Mirror)
- MasterYoda 4y agoHaha, its funny and interesting how importent it is _when_ you submit something on hn, it seems. I submitted the exact same link, with the same title for 1 day ago [1], but no comments and 1 vote :) This post a couple of hours later got alot of comments and traction. [1] https://news.ycombinator.com/item?id=34460202 https://news.ycombinator.com/item?id=34460202
- anonymouse008 4y agoTruly brilliant combination. Not the author's fault, but this implementation reveals one of Apple Shortcut's worst traits: after the target "Ok, Smarthome," one must wait for Shortcuts to link the function and ask for the next input. There's no way to go "Ok, Smarthome what's the temperature in the oven?" One must wait for the "Yes?" in this example - even though it's just one body of text, the Shortcut parser can't delineate anything after the target command. Frustrating.
- Ajedi32 4y agoI've been wanting something like this ever since GPT-2 was a thing. Language models seem like a pretty natural fit for voice assistants. It's just a question of actually building one, and of being comfortable enough with all the new failure modes that enables to actually release the product despite that. Hopefully the existence of ChatGPT has popularized the idea enough that we'll start to see companies actually implementing this in short order. Or at the very least, open source solutions taking advantage of LLM APIs outcompeting proprietary assistants from companies too terrified of the idea of Alexa or Siri being caught on camera saying something racist to implement a system where there's even a remote possibility of that happening.
- mensetmanusman 4y agoThis seems like a moon landing without the fanfare. I’m shocked this is possible.
- quitit 4y agoChatGPT stands out in its ability manage meandering conversational language to find the interwoven commands. The pearl is that the user doesn't need to accomodate the AI's limitations. The core issue with current AI assistants is each has an invisible wall of limitations that the user can only discover via experimentation. This ultimately results in the system training the user on how to speak to it, and what commands it can accept. Since the system doesn't have any way of notifying the user of growing functionality, most users possess a dated idea of the capabilities of their AI assistant. I don't feel it's warranted to use ChatGPT as evidence that AI assistants are bad or that people at FAANG companies are to be ashamed. AI assistants operate under far greater restraints such as a necessity to reply quickly and run computationally light. ChatGPT currently can't provide that at the moment, and making the conversational interaction smarter won't make the whole AI system smarter. E.g. If I say "remind me to get bananas", ChatGPT won't pop up and alert me when I pass a greengrocer or supermarket. Instead one should direct their criticism more fairly: such as why don't AI assistants provide a better means of demonstrating their growing functionality, or demonstrate the flexibility of language that they do understand. I think most people would be pleasantly pleased by what an AI assistant could do if they just knew what they could ask it. Side note: Siri is does understand basic context. If you say "it's dark in here", it will turn on the lights. For those that put a homepod in each room, Siri is smart enough to turn on a light in just that room. It will also handle longer commands that tie into system functionality "remind me to get my prescription when I'm near CVS" will build a reminder that's tied to a geo-fence around the pharmacy. What it doesn't do (and I can't see how ChatGPT would help here) is recommend any pharmacy that I walk past, instead it has to be just that CVS, and if there are multiple CVS's nearby, I would have had to elect one.)