9 ms·

I worked for an AI startup that got bought by a big tech company and I've seen the hype up close. In the inner tech circles it's not exactly a big lie. The tech

I worked for an AI startup that got bought by a big tech company and I've seen the hype up close. In the inner tech circles it's not exactly a big lie. The tech is good enough to make incredible demos but not good enough to generalize into reliable tools. The gulf between demo and useful tool is much wider than we thought.

beoberha 2y ago
I work at Microsoft, though not in AI. This describes Copilot to a T. The demos are spectacular and get you so excited to go use it, but the reality is so underwhelming.
- moose_man 2y ago
  It's a lot like AR before Vision Pro. The situation for the demo and reality didn't meet. I'm not trying to claim Vision Pro is perfect but it seems to do AR in the real world without the circumstances needing to be absolutely ideal.
  - Animats 2y ago
    The Vision Pro is not doing well. Apple has cancelled the next version.[1] As Carmack says, AR/VR will be a small niche until the headgear gets down to swim goggle size, and will not go mainstream until it gets down to eyeglass size. [1] https://www.msn.com/en-us/lifestyle/shopping/apple-shelves-new-model-of-vision-headset-as-sales-decline-report/ar-BB1oszvS https://www.msn.com/en-us/lifestyle/shopping/apple-shelves-n...
    - numpad0 2y ago
      Apple approved ALVR few days ago too, clearly they're having issues at least wrt getting developer attention. 1: https://apps.apple.com/us/app/alvr/id6479728026 https://apps.apple.com/us/app/alvr/id6479728026
    - namaria 2y ago
      I think both hardware and software in AR have to become unobtrusive for people to adopt it. And then it will be a specialized tool for stuff like maintenance. Keeping large amounts of information in context without requiring frequent changes in context. But I also think that the information overload will put a premium on non-AR time. Once it becomes a common work tool, people using it will be very keen to touch grass and watch clouds afterwards. I don't think it will ever become the mainstream everyday carry proponents want it to be. But only time will tell...
      - Turskarama 2y ago
        Until there is an interface for it that allows you to effectively touch type (or equivalent) then 99% of jobs won't be able to use it away from a desk anyway. Speech to text would be good enough for writing (non technical) documentation but probably not for things like filling spreadsheets or programming.
    - addandsubtract 2y ago
      It was always the plan for Apple to release a cheaper version of the Vision Pro next. That the next version of the PRO has been postponed isn't a huge sign. It just seems that the technology isn't evolving quickly enough to warrant a new version any time soon.
    - salad-tycoon 2y ago
      Your article states this differently. The development has not been canceled fully but re focused. “and now hopes to release a more standard headset with fewer abilities by the end of next year.
      - Animats 2y ago
        That's marketing-speak for "cancelled".
    - Yizahi 2y ago
      Which it probably won't, because real life physics are not aware about roadmaps and corporate ads.
      - matthewdgreen 2y ago
        What physics are you talking about? Limits on power? Display? Sensor size? I ask because I’ve had similar feelings about things like high speed mobile Internet or mobile device screen size (over a couple of decades) and lived to see all my intuition blown away, so I really don’t believe in limits that don’t have explicit physical constraints behind them.
        throwup238 2y ago
        Lens diffraction limits. VR needs lenses that are small and thin enough while still being powerful enough to bend the needed light towards the eyes. Modern lenses need more distance between the screen and the eyes and they’re quite thick. Theoretically future lenses may make it possible, but the visible light metamaterials needed are still very early research stage.
    - falcor84 2y ago
      > swim google size The "Bigscreen Beyond" [0] is quite close, but doesn't have cameras - so at this stage it's only really good for watching movies and the like. [0] https://store.bigscreenvr.com/products/bigscreen-beyond https://store.bigscreenvr.com/products/bigscreen-beyond
      - numpad0 2y ago
        That one does have 6DoF tracking, it's just based on the Valve Lighthouse system. Upside of that system is it's more privacy respecting.
  - pbmonster 2y ago
    But does what Apple has shown in its demos of the Vision Pro actually meet reality? Does it provide any value at all? In my eyes, it's exactly the same as AI. The demos work. You can play around with it, and its impressive for an hour. But there's just very little value.
    - MrScruff 2y ago
      The value would come if it was something you would feel comfortable wearing all day. So it would need perfect pass through, be much much lighter and more comfortable. If they achieved that and can do multiple high resolution virtual displays then people would use it. The R&D required to get to that point is vast though.
      - pbmonster 2y ago
        > can do multiple high resolution virtual displays In most applications, it then would need to compete on price with multiple high resolution displays, and undercut them quite significantly to break the inertia of the old tech (and other various advantages - like not wearing something all day and being able to allow other people to look at what you have on your screen).
        MrScruff 2y ago
        I take your point but living in a London flat I don't have the room for multiple high resolution displays. Nor are they very portable, I have a MBP rather than an iMac because mobility is important. I do think we're 4+ years until it gets to the 'iPhone 1' level of utility though, so we'll see how committed Apple are to it.
      - Yizahi 2y ago
        That's what all these companies are peddling though. The question is - do humans actually NEED a display before their eyes for all awake time? Or even most of it? Maybe, but today I have some doubts.
        2y ago
  - timeon 2y ago
    It is not really AR. Reality is not just augmented but captured first with camera. It can make someone dizzy.
    - yencabulator 2y ago
      It's the opposite of AR, it's VR augmented with real imagery.
  - deleted 2y ago
    [deleted]
- throwaway2037 2y ago
  I never considered this angle. (Yeah, I am a sucker -- I know.) Are you saying that they cherry pick the best samples for the demo? Damn. I _still_ have high hopes for something like Copilot. I work on CRUD apps. There are so many cases where I want Copilot to provide some sample code to do X.
  - Slyfox33 2y ago
    They absolutely do. Check out this video https://youtu.be/tNmgmwEtoWE https://youtu.be/tNmgmwEtoWE
  - beoberha 2y ago
    Sorry I didn’t mean GitHub Copilot. Code generation is definitely one of the better use cases for AI. I meant the “Copilot” brand that Microsoft has trotted out into pretty much everyone of its products and rolled together in this generic “Copilot” app on windows.
- TeMPOraL 2y ago
  Copilot isn't underwhelming, it's shit. What's impressive is how Microsoft managed to gut GPT-4 to the point of near-uselessness. It refuses to do work even more than OpenAI models refuse to advise on criminal behavior. In my experience, the only thing it does well is scan documents on corporate SharePoint. For anything else, it's better to copy-paste to a proper GPT-4 yourself. (Ask Office Copilot in PowerPoint to create you a slide. I dare you! I double dare you!!) The problem with demos is that they're staged, they showcase integrations that are never delivered, and probably never existed. But you know what's not hype and fluff? The models themselves. You could hack a more useful Copilot with AutoHotkey, today. I have GPT-4o hooked up as a voice assistant via Home Assistant, and what a breeze that is. Sure, every interaction costs me some $0.03 due to inefficient use of context (HA generates too much noise by default in its map of available devices and their state), but I can walk around the house and turn devices on and off by casually chatting with my watch, and it work, works well, and works faster than it takes to turn on Google Assistant. So no, I honestly don't think AI advances are oversold. It's just that companies large and small race to deploy "AI-enabled" features, no matter how badly made they are.
  - dsauerbrun 2y ago
    whoa that's very cool. can you share some info about how you set up the integration in ha? would love to explore doing something like this for myself
    - TeMPOraL 2y ago
      With the most recent update, it's actually very simple. You need three things: 1) Add OpenAI Conversation integration - https://www.home-assistant.io/integrations/openai_conversation https://www.home-assistant.io/integrations/openai_conversati... - and configure it with your OpenAI API key. In there, you can control part of the system prompt (HA will add some stuff around it) and configure model to use. With the newest HA, there's now an option to enable "Assist" mode (under "Control Home Assistant" header). Enable this. 2) Go to "Settings/Voice assistants". Under "Assist", you can add a new assistant. You'll be asked to pick a name, language to use, then choose a conversation model - here you pick the one you configured in step 1) - and Speech-to-Text and Text-to-Speech models. I have a subscription to Home Assistant Cloud, so I can choose "Home Assistant Cloud" models for STT and TTS; it would be great to integrate third party ones here, but I'm not sure if and how. 3) Still in "Settings/Voice assistants", look for a line saying "${some number} entities exposed", under "Add assistant" button. Click that, and curate the list of devices and sensors you want "exposed" to the assistant - "exposed" here means that HA will make a large YAML dump out of selected entities and paste that into the conversation for you[0]. There's also other stuff (I heard docs mentioning "intents") that you can expose, but I haven't look into it yet[1]. That's it. You can press the Assist button and start typing. Or, for much better experience, install HA's mobile app (and if you have a smartwatch, the watch companion app), and configure Home Assistant as your voice assistant on the device(s). That's how you get the full experience of randomly talking to your watch, "oh hey, make the home feel more like a Borg cube", and witnessing lights turning green and climate control pumping heat. I really recommend everyone who can to try that. It's a night-and-day difference compared to Siri, Alexa or Google Now. It finally fulfills those promises of voice-activated interfaces. (I'm seriously considering making a Home Assistant to Tasker bridge via HA app notification, just to enable the assistant to do things on my phone - experience is just that good, that I bet it'll, out of the box, work better than Google stuff.) -- [0] - That's the inefficient token waster I mentioned in the previous comment. I have some 60 entities exposed, and best I can tell, it generates a couple thousand token's worth of YAML, most of which is noise like entity IDs and YAML structure. This could be cut down significantly if you named your devices and entities cleverly (and concisely), but I think my best bet is to dig into the code and trim it down. And/or create a synthetic entities that stand for multiple entities representing a single device or device group, like e.g. one "A/C" entity that combines multiple sensor entities from all A/C units. [1] - Outside the YAML dump that goes with each message (and a preamble with current date/time), which is how the Assistant know current state of every exposed entity, there's also an extra schema exposing controls via "function calling" mechanism of OpenAI API, which is how the assistant is able to control devices at home. I assume those "intents" go there. I'll be looking into it today, because there's a bunch of interactions I could simplify if I could expose automation scripts to the assistant.
xwolfi 2y ago
So a large cluster of nvidia cards cannot predict the future, generate correct http links, rotate around objects with only a picture at source with the right lighting or program a million lines of code from 3 lines of prompt ? Color me surprised. Maybe we should ask Mira Murati to step aside from her inspiring essays about the future of poetry and help us figure out why the world spent trillions on nvidia equity and how to unwind this pending disaster...
- waciki 2y ago
  it also can't reliably add two numbers to each other. > help us figure out why the world spent trillions on nvidia equity and how to unwind this pending disaster.. There are many documented examples of the market being irrational.
surfingdino 2y ago
> The tech is good enough to make incredible demos but not good enough to generalize into reliable tools. The gulf between demo and useful tool is much wider than we thought. One thing it is good at is scaring people into paying to feed it all the data they have for a promise of an unquantifiable improvement.
yawnxyz 2y ago
I just used Groq / llama-7b to classify 20,000 rows of Google sheets data (Sidebar archive links) that would have taken me way longer... Every one I've spot checked right now has been correct, and I might write another checker to scan the results just in case. Even w/ a 20% failure it's better than not having the classifications
- lifeisstillgood 2y ago
  Sorry this actually sounds like a real use case. What was the classification? (I tried google “sidebar archive”). I assume somehow you visited 20,000 web pages and it classified the text on the page? How was that achieved ? You ran a local llama?
  - karles 2y ago
    We had ChatGPT look at 200.000 products, and make a navigation structure in 3 tiers based on the attributes of each product. The validation took 2% of the time it would have taken to manually create the hierarchy ourselves. I think that even the simple LLM's are very well suited for classification-tasks, where very little prompting is needed.
    - lifeisstillgood 2y ago
      Sorry to harp on. So you had a list of products (what sort - I am thinking like widgets from a wholesaler and you want to have a three tier menu for an e-commerce site?) I am guessing each product has a description - like from Amazon, and chatgpt read the description and said “aha this is a Television/LCD/50inch or Underwear/flimsy/bra I assume you sent in 200,000 different queries - but how did you get it to return three tiers? (Maybe I need to read one of those “become a ChatGPt expert” blogs
      - t-writescode 2y ago
        I'm not this person; but, I've been working on LLMs pretty aggressively for the last 6ish months and I have some ideas of how this __could__ be done. You could plainly ask the LLM something like this as the query goes on: "Please provide 3 categories that this product could exist under, with increasing specificity in the following format: { "broad category": "a broad category that would encompass this product, as well as others, for example 'televisions' for a 50" OLED LG with Roku integration", "category": "a narrower category that describes this product more aggressively, for example 'Smart Televisions'", "narrow category": "an even narrower category that describes this product and its direct competitors, for example OLED Smart televisions" } A next question you'll have pretty quick is, "Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category?" And that's a good and valid question, so you then have another query that takes the categories that have been provided to you and asks for synonyms, alternative spellings, etc, such as: "Given a product categorization of a specific level of specificity, please provide a list of words and phrases that mean the same thing". In OpenAI's backend - and many of them, I think, you can have the api run the query multiple times and get back multiple answers. enumerate over those answers, build the graph, and you can have all that data in an easy to read and follow format! It might not be perfect, but it should be pretty good!
rob74 2y ago
Looks like this is another application of the ninety-ninety rule: getting to the stage where you can make incredible demos has required 90% of the effort, and the last 10% to make it actually reliable will require the other 90% (https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule).
- veqq 2y ago
  80/20, 90/10 etc. are all just https://en.wikipedia.org/wiki/Zipf%27s_law https://en.wikipedia.org/wiki/Zipf%27s_law
  - com 2y ago
    Excellent find, I’d never heard of Zipf’s law. GP was talking about something else though, the 90:90 rule is related to an extremely common planning optimism fallacy around work required to demo, and work required to productise.
    - Sakos 2y ago
      It's not just demos though. It's that the final 10% of any project, which largely consists of polishing, implementing feedback, ironing out bugs or edge cases, and finalization and getting to a point where it's "done" can end up taking as much effort as what you needed to complete the first 90% of the project.
    - 0xEF 2y ago
      Can you elaborate? I am curious. In my line of work, the 80/20 rule is often throw about, that being "to do 80% of the work, you only need 20% of the knowledge." I thought the other reply was talking about the same diminutive axiom, but now I am not sure.
      - com 2y ago
        The sibling post gives a good account of the 90:90 challenge. The last part of any semi-difficult project nearly always takes much longer than the officially difficult “main problem” to solve. It leads to the last 10% of the deliverables costing at least 90% of the total effort for the project (not the planned amount; the total ad calculated after completion, if that ever occurs) This seems to endlessly surprise people in tech, but also many other semi-professional project domains (home renovations are a classic)
rsynnott 2y ago
> The gulf between demo and useful tool is much wider than we thought. This is _always_ the problem with these things. Voice transcription was a great tech demo in the 1990s (remember DragonDictate?), and there was much hype for a couple of years that, by the early noughties, speech would be the main way that people use computes. In the real world, 30 years on, it has finally reached the point where you might be able to use it for things provided that accuracy doesn't matter at all.
- jraph 2y ago
  Assuming it works perfectly, speech still couldn't possibly be the main way to use a computer: - hearing people next to you speaking to the computer would be tiring and annoying. Though remote work might be a partial solution to this - hello voice extinction after days of using a computer :-)
  - acuozzo 2y ago
    Same here, but I'm hoping it takes off for other people. I get requests all the time from colleagues to have discussions via telephone instead of chat because they are bad at typing.
  - rsynnott 2y ago
    Oh, yeah, I mean, it would've been awful had it actually happened, even if it worked properly. But try telling that to Microsoft's marketing department circa 1998. (MS had a bit of a fetish for alternate interfaces at the time; before voice they spent a few years desperately trying to make Windows for Pen Computing a thing).

beoberha 2y ago

I work at Microsoft, though not in AI. This describes Copilot to a T. The demos are spectacular and get you so excited to go use it, but the reality is so underwhelming.

moose_man 2y ago
It's a lot like AR before Vision Pro. The situation for the demo and reality didn't meet. I'm not trying to claim Vision Pro is perfect but it seems to do AR in the real world without the circumstances needing to be absolutely ideal.
- Animats 2y ago
  The Vision Pro is not doing well. Apple has cancelled the next version.[1] As Carmack says, AR/VR will be a small niche until the headgear gets down to swim goggle size, and will not go mainstream until it gets down to eyeglass size. [1] https://www.msn.com/en-us/lifestyle/shopping/apple-shelves-new-model-of-vision-headset-as-sales-decline-report/ar-BB1oszvS https://www.msn.com/en-us/lifestyle/shopping/apple-shelves-n...
  - numpad0 2y ago
    Apple approved ALVR few days ago too, clearly they're having issues at least wrt getting developer attention. 1: https://apps.apple.com/us/app/alvr/id6479728026 https://apps.apple.com/us/app/alvr/id6479728026
  - namaria 2y ago
    I think both hardware and software in AR have to become unobtrusive for people to adopt it. And then it will be a specialized tool for stuff like maintenance. Keeping large amounts of information in context without requiring frequent changes in context. But I also think that the information overload will put a premium on non-AR time. Once it becomes a common work tool, people using it will be very keen to touch grass and watch clouds afterwards. I don't think it will ever become the mainstream everyday carry proponents want it to be. But only time will tell...
    - Turskarama 2y ago
      Until there is an interface for it that allows you to effectively touch type (or equivalent) then 99% of jobs won't be able to use it away from a desk anyway. Speech to text would be good enough for writing (non technical) documentation but probably not for things like filling spreadsheets or programming.
  - addandsubtract 2y ago
    It was always the plan for Apple to release a cheaper version of the Vision Pro next. That the next version of the PRO has been postponed isn't a huge sign. It just seems that the technology isn't evolving quickly enough to warrant a new version any time soon.
  - salad-tycoon 2y ago
    Your article states this differently. The development has not been canceled fully but re focused. “and now hopes to release a more standard headset with fewer abilities by the end of next year.
    - Animats 2y ago
      That's marketing-speak for "cancelled".
  - Yizahi 2y ago
    Which it probably won't, because real life physics are not aware about roadmaps and corporate ads.
    - matthewdgreen 2y ago
      What physics are you talking about? Limits on power? Display? Sensor size? I ask because I’ve had similar feelings about things like high speed mobile Internet or mobile device screen size (over a couple of decades) and lived to see all my intuition blown away, so I really don’t believe in limits that don’t have explicit physical constraints behind them.
      - throwup238 2y ago
        Lens diffraction limits. VR needs lenses that are small and thin enough while still being powerful enough to bend the needed light towards the eyes. Modern lenses need more distance between the screen and the eyes and they’re quite thick. Theoretically future lenses may make it possible, but the visible light metamaterials needed are still very early research stage.
  - falcor84 2y ago
    > swim google size The "Bigscreen Beyond" [0] is quite close, but doesn't have cameras - so at this stage it's only really good for watching movies and the like. [0] https://store.bigscreenvr.com/products/bigscreen-beyond https://store.bigscreenvr.com/products/bigscreen-beyond
    - numpad0 2y ago
      That one does have 6DoF tracking, it's just based on the Valve Lighthouse system. Upside of that system is it's more privacy respecting.
- pbmonster 2y ago
  But does what Apple has shown in its demos of the Vision Pro actually meet reality? Does it provide any value at all? In my eyes, it's exactly the same as AI. The demos work. You can play around with it, and its impressive for an hour. But there's just very little value.
  - MrScruff 2y ago
    The value would come if it was something you would feel comfortable wearing all day. So it would need perfect pass through, be much much lighter and more comfortable. If they achieved that and can do multiple high resolution virtual displays then people would use it. The R&D required to get to that point is vast though.
    - pbmonster 2y ago
      > can do multiple high resolution virtual displays In most applications, it then would need to compete on price with multiple high resolution displays, and undercut them quite significantly to break the inertia of the old tech (and other various advantages - like not wearing something all day and being able to allow other people to look at what you have on your screen).
      - MrScruff 2y ago
        I take your point but living in a London flat I don't have the room for multiple high resolution displays. Nor are they very portable, I have a MBP rather than an iMac because mobility is important. I do think we're 4+ years until it gets to the 'iPhone 1' level of utility though, so we'll see how committed Apple are to it.
    - Yizahi 2y ago
      That's what all these companies are peddling though. The question is - do humans actually NEED a display before their eyes for all awake time? Or even most of it? Maybe, but today I have some doubts.
      - 2y ago
- timeon 2y ago
  It is not really AR. Reality is not just augmented but captured first with camera. It can make someone dizzy.
  - yencabulator 2y ago
    It's the opposite of AR, it's VR augmented with real imagery.
- deleted 2y ago
  [deleted]
throwaway2037 2y ago
I never considered this angle. (Yeah, I am a sucker -- I know.) Are you saying that they cherry pick the best samples for the demo? Damn. I _still_ have high hopes for something like Copilot. I work on CRUD apps. There are so many cases where I want Copilot to provide some sample code to do X.
- Slyfox33 2y ago
  They absolutely do. Check out this video https://youtu.be/tNmgmwEtoWE https://youtu.be/tNmgmwEtoWE
- beoberha 2y ago
  Sorry I didn’t mean GitHub Copilot. Code generation is definitely one of the better use cases for AI. I meant the “Copilot” brand that Microsoft has trotted out into pretty much everyone of its products and rolled together in this generic “Copilot” app on windows.
TeMPOraL 2y ago
Copilot isn't underwhelming, it's shit. What's impressive is how Microsoft managed to gut GPT-4 to the point of near-uselessness. It refuses to do work even more than OpenAI models refuse to advise on criminal behavior. In my experience, the only thing it does well is scan documents on corporate SharePoint. For anything else, it's better to copy-paste to a proper GPT-4 yourself. (Ask Office Copilot in PowerPoint to create you a slide. I dare you! I double dare you!!) The problem with demos is that they're staged, they showcase integrations that are never delivered, and probably never existed. But you know what's not hype and fluff? The models themselves. You could hack a more useful Copilot with AutoHotkey, today. I have GPT-4o hooked up as a voice assistant via Home Assistant, and what a breeze that is. Sure, every interaction costs me some $0.03 due to inefficient use of context (HA generates too much noise by default in its map of available devices and their state), but I can walk around the house and turn devices on and off by casually chatting with my watch, and it work, works well, and works faster than it takes to turn on Google Assistant. So no, I honestly don't think AI advances are oversold. It's just that companies large and small race to deploy "AI-enabled" features, no matter how badly made they are.
- dsauerbrun 2y ago
  whoa that's very cool. can you share some info about how you set up the integration in ha? would love to explore doing something like this for myself
  - TeMPOraL 2y ago
    With the most recent update, it's actually very simple. You need three things: 1) Add OpenAI Conversation integration - https://www.home-assistant.io/integrations/openai_conversation https://www.home-assistant.io/integrations/openai_conversati... - and configure it with your OpenAI API key. In there, you can control part of the system prompt (HA will add some stuff around it) and configure model to use. With the newest HA, there's now an option to enable "Assist" mode (under "Control Home Assistant" header). Enable this. 2) Go to "Settings/Voice assistants". Under "Assist", you can add a new assistant. You'll be asked to pick a name, language to use, then choose a conversation model - here you pick the one you configured in step 1) - and Speech-to-Text and Text-to-Speech models. I have a subscription to Home Assistant Cloud, so I can choose "Home Assistant Cloud" models for STT and TTS; it would be great to integrate third party ones here, but I'm not sure if and how. 3) Still in "Settings/Voice assistants", look for a line saying "${some number} entities exposed", under "Add assistant" button. Click that, and curate the list of devices and sensors you want "exposed" to the assistant - "exposed" here means that HA will make a large YAML dump out of selected entities and paste that into the conversation for you[0]. There's also other stuff (I heard docs mentioning "intents") that you can expose, but I haven't look into it yet[1]. That's it. You can press the Assist button and start typing. Or, for much better experience, install HA's mobile app (and if you have a smartwatch, the watch companion app), and configure Home Assistant as your voice assistant on the device(s). That's how you get the full experience of randomly talking to your watch, "oh hey, make the home feel more like a Borg cube", and witnessing lights turning green and climate control pumping heat. I really recommend everyone who can to try that. It's a night-and-day difference compared to Siri, Alexa or Google Now. It finally fulfills those promises of voice-activated interfaces. (I'm seriously considering making a Home Assistant to Tasker bridge via HA app notification, just to enable the assistant to do things on my phone - experience is just that good, that I bet it'll, out of the box, work better than Google stuff.) -- [0] - That's the inefficient token waster I mentioned in the previous comment. I have some 60 entities exposed, and best I can tell, it generates a couple thousand token's worth of YAML, most of which is noise like entity IDs and YAML structure. This could be cut down significantly if you named your devices and entities cleverly (and concisely), but I think my best bet is to dig into the code and trim it down. And/or create a synthetic entities that stand for multiple entities representing a single device or device group, like e.g. one "A/C" entity that combines multiple sensor entities from all A/C units. [1] - Outside the YAML dump that goes with each message (and a preamble with current date/time), which is how the Assistant know current state of every exposed entity, there's also an extra schema exposing controls via "function calling" mechanism of OpenAI API, which is how the assistant is able to control devices at home. I assume those "intents" go there. I'll be looking into it today, because there's a bunch of interactions I could simplify if I could expose automation scripts to the assistant.

xwolfi 2y ago

So a large cluster of nvidia cards cannot predict the future, generate correct http links, rotate around objects with only a picture at source with the right lighting or program a million lines of code from 3 lines of prompt ? Color me surprised. Maybe we should ask Mira Murati to step aside from her inspiring essays about the future of poetry and help us figure out why the world spent trillions on nvidia equity and how to unwind this pending disaster...

waciki 2y ago
it also can't reliably add two numbers to each other. > help us figure out why the world spent trillions on nvidia equity and how to unwind this pending disaster.. There are many documented examples of the market being irrational.

surfingdino 2y ago

> The tech is good enough to make incredible demos but not good enough to generalize into reliable tools. The gulf between demo and useful tool is much wider than we thought. One thing it is good at is scaring people into paying to feed it all the data they have for a promise of an unquantifiable improvement.

yawnxyz 2y ago

I just used Groq / llama-7b to classify 20,000 rows of Google sheets data (Sidebar archive links) that would have taken me way longer... Every one I've spot checked right now has been correct, and I might write another checker to scan the results just in case. Even w/ a 20% failure it's better than not having the classifications

lifeisstillgood 2y ago
Sorry this actually sounds like a real use case. What was the classification? (I tried google “sidebar archive”). I assume somehow you visited 20,000 web pages and it classified the text on the page? How was that achieved ? You ran a local llama?
- karles 2y ago
  We had ChatGPT look at 200.000 products, and make a navigation structure in 3 tiers based on the attributes of each product. The validation took 2% of the time it would have taken to manually create the hierarchy ourselves. I think that even the simple LLM's are very well suited for classification-tasks, where very little prompting is needed.
  - lifeisstillgood 2y ago
    Sorry to harp on. So you had a list of products (what sort - I am thinking like widgets from a wholesaler and you want to have a three tier menu for an e-commerce site?) I am guessing each product has a description - like from Amazon, and chatgpt read the description and said “aha this is a Television/LCD/50inch or Underwear/flimsy/bra I assume you sent in 200,000 different queries - but how did you get it to return three tiers? (Maybe I need to read one of those “become a ChatGPt expert” blogs
    - t-writescode 2y ago
      I'm not this person; but, I've been working on LLMs pretty aggressively for the last 6ish months and I have some ideas of how this __could__ be done. You could plainly ask the LLM something like this as the query goes on: "Please provide 3 categories that this product could exist under, with increasing specificity in the following format: { "broad category": "a broad category that would encompass this product, as well as others, for example 'televisions' for a 50" OLED LG with Roku integration", "category": "a narrower category that describes this product more aggressively, for example 'Smart Televisions'", "narrow category": "an even narrower category that describes this product and its direct competitors, for example OLED Smart televisions" } A next question you'll have pretty quick is, "Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category?" And that's a good and valid question, so you then have another query that takes the categories that have been provided to you and asks for synonyms, alternative spellings, etc, such as: "Given a product categorization of a specific level of specificity, please provide a list of words and phrases that mean the same thing". In OpenAI's backend - and many of them, I think, you can have the api run the query multiple times and get back multiple answers. enumerate over those answers, build the graph, and you can have all that data in an easy to read and follow format! It might not be perfect, but it should be pretty good!

rob74 2y ago

Looks like this is another application of the ninety-ninety rule: getting to the stage where you can make incredible demos has required 90% of the effort, and the last 10% to make it actually reliable will require the other 90% (https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule).

veqq 2y ago
80/20, 90/10 etc. are all just https://en.wikipedia.org/wiki/Zipf%27s_law https://en.wikipedia.org/wiki/Zipf%27s_law
- com 2y ago
  Excellent find, I’d never heard of Zipf’s law. GP was talking about something else though, the 90:90 rule is related to an extremely common planning optimism fallacy around work required to demo, and work required to productise.
  - Sakos 2y ago
    It's not just demos though. It's that the final 10% of any project, which largely consists of polishing, implementing feedback, ironing out bugs or edge cases, and finalization and getting to a point where it's "done" can end up taking as much effort as what you needed to complete the first 90% of the project.
  - 0xEF 2y ago
    Can you elaborate? I am curious. In my line of work, the 80/20 rule is often throw about, that being "to do 80% of the work, you only need 20% of the knowledge." I thought the other reply was talking about the same diminutive axiom, but now I am not sure.
    - com 2y ago
      The sibling post gives a good account of the 90:90 challenge. The last part of any semi-difficult project nearly always takes much longer than the officially difficult “main problem” to solve. It leads to the last 10% of the deliverables costing at least 90% of the total effort for the project (not the planned amount; the total ad calculated after completion, if that ever occurs) This seems to endlessly surprise people in tech, but also many other semi-professional project domains (home renovations are a classic)

rsynnott 2y ago

> The gulf between demo and useful tool is much wider than we thought. This is _always_ the problem with these things. Voice transcription was a great tech demo in the 1990s (remember DragonDictate?), and there was much hype for a couple of years that, by the early noughties, speech would be the main way that people use computes. In the real world, 30 years on, it has finally reached the point where you might be able to use it for things provided that accuracy doesn't matter at all.

jraph 2y ago
Assuming it works perfectly, speech still couldn't possibly be the main way to use a computer: - hearing people next to you speaking to the computer would be tiring and annoying. Though remote work might be a partial solution to this - hello voice extinction after days of using a computer :-)
- acuozzo 2y ago
  Same here, but I'm hoping it takes off for other people. I get requests all the time from colleagues to have discussions via telephone instead of chat because they are bad at typing.
- rsynnott 2y ago
  Oh, yeah, I mean, it would've been awful had it actually happened, even if it worked properly. But try telling that to Microsoft's marketing department circa 1998. (MS had a bit of a fetish for alternate interfaces at the time; before voice they spent a few years desperately trying to make Windows for Pen Computing a thing).