4 ms·

Vision is the ultimate API. The historical progression from text to still images to audio to moving images will hold true for AI as well. Just look at OpenAI'

Vision is the ultimate API. The historical progression from text to still images to audio to moving images will hold true for AI as well. Just look at OpenAI's progression as well from LLM to multi-modal to the realtime API. A co-worker almost 20 years ago said something interesting to me as we were discussing Al Gore's CurrentTV project: the history of information is constrained by "bandwidth". He mentioned how broadcast television went from 72 hours of "bandwidth" (3 channels x 24h) per day to now having so much bandwidth that we could have a channel with citizen journalists. Of course, this was also the same time that YouTube was taking off. The pattern holds true for AI. AI is going to create "infinite bandwidth".

ToDougie 2y ago
So long as the spectrum is open for infinity, yes. Was listening to a Seth Godin interview where he pointed out that there was a time when you had to purchase a slice of spectrum to share your voice on the radio. Nowadays you can put your thoughts on a platform, but that platform is owned by corporations who can and will put their thumb on thoughtcrime or challenges. I really do love your comment. Cheers.
- CharlieDigital 2y ago
  Thanks! There's a related concept as well which is that as "bandwidth" increases, the ratio of producers to consumers pushes upwards towards 1. My take is that generative AI will accelerate this I write a bit more in depth about it here: https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and-why-you-should-be-too/ https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and...
ricardo81 2y ago
You could call it bandwidth, or call it entropy. I'd lean towards the more physical definition. I think of how the USA had cable TV and hundreds of channels projecting all kinds of whatever in the 80s while here in the UK we were limited to our finite channels. To be fair those finite channels gave people something to talk about the next day, because millions of people saw the same thing. Surely a lot of what mankind has done is to tame entropy, like steam engines etc. With AI and everyone having a prompt, it's surely a game changer. How it works out, we'll see.
swatcoder 2y ago
> The historical progression from text to still images to audio to moving images will hold true for AI as well. You'll have to explain what you mean by this. Direct speech, text, illustrations, photos, abstract sounds, music, recordings, videos, circuits, programs, cells... these are all just different mediums with different characteristics. There is no "progression" apparent among them. Why should there be? They each fulfill different ends and have different occasions for which they best suit. We seem to have discovered a new family of tools that help lossilly transform content or intent from one of these mediums to some others, which is sure to be useful in its own ways. But it's not a medium like the above in the first place, and with none of them representing a progression, it certainly doesn't either.
- CharlieDigital 2y ago
  > You'll have to explain what you mean by this The progression of distribution. Printing press, photos, radio, movies, television. The early web was text, then came images, then audio (Napster age), and then video (remember that Netflix used to ship DVDs?). The flip side of that is production and the ratio of producers to consumers. As the bandwidth for distribution increases, there is a decrease in the cost and complexity for producers and naturally, we see the same progression with producers on each new platform and distribution technology: text, still images, audio, moving images.
  - margalabargala 2y ago
    > The progression of distribution. Printing press, photos, radio, movies, television. Your history is incorrect, though. Still images predate text, by a lot. Cave paintings came before writing. Woodcuts came before the printing press.
    - earthnail 2y ago
      Not in the information age. This cascade just corresponds to how much data and processing power is needed for each. It is entirely logical to say that AI development will follow the same progression as the early internet or as broadcast since they all fall under the same data constraints.
croes 2y ago
Vision especially GUIs are a pretty limited API.
- abirch 2y ago
  It reminded me of these old Unix lessons of Master Foo https://prirai.github.io/books/unix-koans/#master-foo-discourses-on-the-graphical-user-interface https://prirai.github.io/books/unix-koans/#master-foo-discou...
- CharlieDigital 2y ago
  I mean vision in the most general sense, not just a GUI. Imagine OpenAI can not only read the inflection in your voice, but also nuances in your facial expressions and how you're using your hands to understand your state of mind. And instead of merely responding as an audio stream, a real-time avatar.
  - corobo 2y ago
    Sweet! I'll have my own Holly from Red Dwarf! or maybe my own HAL9000.. A little bit ambivalent on this haha, looking forward to seeing what comes of it either way though :)
  - croes 2y ago
    Imagine how this could be abused.
    - CharlieDigital 2y ago
      Or how it could be used.
      - pixl97 2y ago
        Used and abused. That is the nature of intelligence.
skydhash 2y ago
It is not. Text is very dense information wise and recursive and you can formalize it. And easily coupled with interactions method. And more apt for automation. You can easily see this with software like Autocad which have both. There's a reason all protocol are texts. Vision and audio plays a nice role, but that's because of humans and reality. real world <-> vision|audio <-> processing pipeline makes sense. But processing <-> data <-> vision|audio <-> data <-> processing cycle is just non sense and a waste of resources.
- cooper_ganglia 2y ago
  Not a waste of resources, just an increase in use. This is why need more resources.
  - skydhash 2y ago
    It's a waste, because you could have just resolve the middleman and have data <-> processing cycle. When you increase resources use, that means some other metrics should be increased with an higher factor (car vs carriage and horses, computers vs doing computation by hand), otherwise it's a waste
- ricardo81 2y ago
  >Text is very dense information wise and recursive and you can formalize it. There's been a lot of attempts over the years and varying degrees of accuracy, but I don't know if you can go as far as to "formalize" it. Beyond the syntax, (tokenising, syntactic chunking and ...beyond) there is the intent, and that is super hard to measure. And possibly, the problem with these prompts is they get things right a lot of the time but wrong say 5% of the time. Purely because they couldn't formalize it. My web hosting has 99.99% uptime which is a bit more reassuring than 95%.
layer8 2y ago
Text is still a predominant medium of communication and information processing, and I don’t see that changing substantially. TFA was an article, not a video, and you wouldn’t want the HN comment section to be composed of videos or images. Similarly, video calls haven’t replaced texting.
- CharlieDigital 2y ago
  It's not that it will be replaced, but there's a natural progression of the types of media that is available on a given platform of distribution. RF: text (telegram), audio, still images (fax), moving images Web had the same progression: text, still images (inverted here), audio age (MP3s, Napster), video (Netflix, YouTube) AI: text, images, audio (realtime API), ...? Vision is the obvious next medium.
  - wwweston 2y ago
    Lately I've been realizing that as much as I value YouTube, much of the content is distorted by various incentives favoring length and frequency and a tech culture which overfocuses on elaborating steps as an equivalent to an explanation. This contributes to more duplicate content (often even within the same channel) and less in terms of refined conceptualization. I find myself often wishing I had outlines, summaries, and hyperlinks to greater details. Of course, I can use AI tools to get approximations of such things, and it'll probably get better, which means we will now be using this increased bandwidth or progression to produce more video to be pushed out through the pipes and distilled by an AI tool into shorter video or something like hypertext. Progress!
  - layer8 2y ago
    That progression is obviously a consequence of being able to tackle increasingly large volumes of data, but that doesn’t make video an “ultimate API”, unless you just mean “the most demanding one in terms of data volume”.
    - CharlieDigital 2y ago
      That's the point: as bandwidth and capacity increases, there's a natural progression. So it's obvious that vision is next after audio.
wwweston 2y ago
The information coursing through the world around us already exceeds our ability to grasp it by high orders of magnitude. Three channels of television over 8 hours was already more than anyone had time to take in. AI might be able to create a summarizing layers and relays that help manage that. AI isn't going to create infinite bandwidth. It's as likely to increase entropy and introduce noise.
- CharlieDigital 2y ago
  It's not that it makes more channels for all of us, it creates a channel for each of us.
  - mathgeek 2y ago
    There are already more channels than that, as bots training off the media produced by bots will just continue to grow.
IAmGraydon 2y ago
The 'bandwidth' analogy breaks down because it assumes AI's value lies in processing more information, rather than processing information more intelligently. Increasing broadcast bandwidth added linear value, but AI's advancements come from complexity, nuance, and understanding - not just sheer volume of data. 'Infinite bandwidth' doesn't guarantee better insights or decision-making; it may even lead to information overload and decreased relevance.
- CharlieDigital 2y ago
  Gen AI's value lies in producing more variants of information. Infinitely many. If you and I prompt OpenAI to generate an image of a woman holding a candle, we'll get two totally novel instances.
robotresearcher 2y ago
> The pattern holds true for AI. AI is going to create "infinite bandwidth". I’ve worked in AI for more than 30 years and I have no idea what you mean by this. Can you explain?
- CharlieDigital 2y ago
  There is a ratio of producers to consumers in all media. There's two ways to think about bandwidth. One is the physical capacity. The other is the content that can be produced and distributed. We once had 3 channels equating to a maximum of 72h of content in a 24h period. Now we have YouTube which is orders of magnitude more content and bandwidth. The constraint now is the ratio of producers to consumers. Some creator had to create the exact content that you want. What if gen AI can create the exact content and media experience that you want? Effectively pushing the ratio of producers and consumers towards 1 so that every experience is unique? It is effectively as if there was infinite bandwidth to create and distribute content. You are no longer constrained by physical bandwidth and no longer constrained by production bandwidth (actual creators making the content). You want your AI generated news reel delivered by Walter Cronkite. I want mine delivered by Barbara Walters wearing a fake mustache while standing on one hand on the moon. It is as if there are infinite producers. I write a bit more on this topic here: https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and-why-you-should-be-too/ https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and...
  - pixl97 2y ago
    What happens after the effective end of society?
slowmovintarget 2y ago
> AI is going to create "infinite bandwidth". For whom? If you mean infinite outpouring, then yes, but it will drown us in a sea of noise. We've constructed a Chinese Room for the mind. The computer was a bicycle, but this is something different. Bandwidth is carrying ability, and the current incarnation of "AI" does not increase signal. It takes vastly more resources to produce something close enough, but not quite... it.
- CharlieDigital 2y ago
  Just as there are thousands of videos on YouTube for making pancakes, there will one day be infinite videos for making pancakes. That visual interface will watch as you prep your pancakes and give you tips, suggest a substitute if you are missing an ingredient. Your experience with that recipe will be one of "infinitely" many.
- pixl97 2y ago
  Eh even current AI where it makes a summary of your preferences is ever so slightly increasing signal. I don't expect this ability to diminish over time, but instead increase, which would lead to more personalized signaling.
diffeomorphism 2y ago
Vision fails the discoverability test: Buttons that don't look like buttons. Fake download buttons. Long press to do something. Swipe in the shape of a hexagon. Grey text on light grey background. Also: thanks for tuning in, raid shadow legends, many people ask, but how... Anyway, you need these two lines of text (20 minutes YouTube videos could have been a half page of text) Finally: Huge output of bad quality and very, very limited input capacity. So "infinite bandwidth in" and then horrible traffic jam out.

ToDougie 2y ago

So long as the spectrum is open for infinity, yes. Was listening to a Seth Godin interview where he pointed out that there was a time when you had to purchase a slice of spectrum to share your voice on the radio. Nowadays you can put your thoughts on a platform, but that platform is owned by corporations who can and will put their thumb on thoughtcrime or challenges. I really do love your comment. Cheers.

CharlieDigital 2y ago
Thanks! There's a related concept as well which is that as "bandwidth" increases, the ratio of producers to consumers pushes upwards towards 1. My take is that generative AI will accelerate this I write a bit more in depth about it here: https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and-why-you-should-be-too/ https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and...

ricardo81 2y ago

You could call it bandwidth, or call it entropy. I'd lean towards the more physical definition. I think of how the USA had cable TV and hundreds of channels projecting all kinds of whatever in the 80s while here in the UK we were limited to our finite channels. To be fair those finite channels gave people something to talk about the next day, because millions of people saw the same thing. Surely a lot of what mankind has done is to tame entropy, like steam engines etc. With AI and everyone having a prompt, it's surely a game changer. How it works out, we'll see.

swatcoder 2y ago

> The historical progression from text to still images to audio to moving images will hold true for AI as well. You'll have to explain what you mean by this. Direct speech, text, illustrations, photos, abstract sounds, music, recordings, videos, circuits, programs, cells... these are all just different mediums with different characteristics. There is no "progression" apparent among them. Why should there be? They each fulfill different ends and have different occasions for which they best suit. We seem to have discovered a new family of tools that help lossilly transform content or intent from one of these mediums to some others, which is sure to be useful in its own ways. But it's not a medium like the above in the first place, and with none of them representing a progression, it certainly doesn't either.

CharlieDigital 2y ago
> You'll have to explain what you mean by this The progression of distribution. Printing press, photos, radio, movies, television. The early web was text, then came images, then audio (Napster age), and then video (remember that Netflix used to ship DVDs?). The flip side of that is production and the ratio of producers to consumers. As the bandwidth for distribution increases, there is a decrease in the cost and complexity for producers and naturally, we see the same progression with producers on each new platform and distribution technology: text, still images, audio, moving images.
- margalabargala 2y ago
  > The progression of distribution. Printing press, photos, radio, movies, television. Your history is incorrect, though. Still images predate text, by a lot. Cave paintings came before writing. Woodcuts came before the printing press.
  - earthnail 2y ago
    Not in the information age. This cascade just corresponds to how much data and processing power is needed for each. It is entirely logical to say that AI development will follow the same progression as the early internet or as broadcast since they all fall under the same data constraints.

croes 2y ago

Vision especially GUIs are a pretty limited API.

abirch 2y ago
It reminded me of these old Unix lessons of Master Foo https://prirai.github.io/books/unix-koans/#master-foo-discourses-on-the-graphical-user-interface https://prirai.github.io/books/unix-koans/#master-foo-discou...
CharlieDigital 2y ago
I mean vision in the most general sense, not just a GUI. Imagine OpenAI can not only read the inflection in your voice, but also nuances in your facial expressions and how you're using your hands to understand your state of mind. And instead of merely responding as an audio stream, a real-time avatar.
- corobo 2y ago
  Sweet! I'll have my own Holly from Red Dwarf! or maybe my own HAL9000.. A little bit ambivalent on this haha, looking forward to seeing what comes of it either way though :)
- croes 2y ago
  Imagine how this could be abused.
  - CharlieDigital 2y ago
    Or how it could be used.
    - pixl97 2y ago
      Used and abused. That is the nature of intelligence.

skydhash 2y ago

It is not. Text is very dense information wise and recursive and you can formalize it. And easily coupled with interactions method. And more apt for automation. You can easily see this with software like Autocad which have both. There's a reason all protocol are texts. Vision and audio plays a nice role, but that's because of humans and reality. real world <-> vision|audio <-> processing pipeline makes sense. But processing <-> data <-> vision|audio <-> data <-> processing cycle is just non sense and a waste of resources.

cooper_ganglia 2y ago
Not a waste of resources, just an increase in use. This is why need more resources.
- skydhash 2y ago
  It's a waste, because you could have just resolve the middleman and have data <-> processing cycle. When you increase resources use, that means some other metrics should be increased with an higher factor (car vs carriage and horses, computers vs doing computation by hand), otherwise it's a waste
ricardo81 2y ago
>Text is very dense information wise and recursive and you can formalize it. There's been a lot of attempts over the years and varying degrees of accuracy, but I don't know if you can go as far as to "formalize" it. Beyond the syntax, (tokenising, syntactic chunking and ...beyond) there is the intent, and that is super hard to measure. And possibly, the problem with these prompts is they get things right a lot of the time but wrong say 5% of the time. Purely because they couldn't formalize it. My web hosting has 99.99% uptime which is a bit more reassuring than 95%.

layer8 2y ago

Text is still a predominant medium of communication and information processing, and I don’t see that changing substantially. TFA was an article, not a video, and you wouldn’t want the HN comment section to be composed of videos or images. Similarly, video calls haven’t replaced texting.

CharlieDigital 2y ago
It's not that it will be replaced, but there's a natural progression of the types of media that is available on a given platform of distribution. RF: text (telegram), audio, still images (fax), moving images Web had the same progression: text, still images (inverted here), audio age (MP3s, Napster), video (Netflix, YouTube) AI: text, images, audio (realtime API), ...? Vision is the obvious next medium.
- wwweston 2y ago
  Lately I've been realizing that as much as I value YouTube, much of the content is distorted by various incentives favoring length and frequency and a tech culture which overfocuses on elaborating steps as an equivalent to an explanation. This contributes to more duplicate content (often even within the same channel) and less in terms of refined conceptualization. I find myself often wishing I had outlines, summaries, and hyperlinks to greater details. Of course, I can use AI tools to get approximations of such things, and it'll probably get better, which means we will now be using this increased bandwidth or progression to produce more video to be pushed out through the pipes and distilled by an AI tool into shorter video or something like hypertext. Progress!
- layer8 2y ago
  That progression is obviously a consequence of being able to tackle increasingly large volumes of data, but that doesn’t make video an “ultimate API”, unless you just mean “the most demanding one in terms of data volume”.
  - CharlieDigital 2y ago
    That's the point: as bandwidth and capacity increases, there's a natural progression. So it's obvious that vision is next after audio.

wwweston 2y ago

The information coursing through the world around us already exceeds our ability to grasp it by high orders of magnitude. Three channels of television over 8 hours was already more than anyone had time to take in. AI might be able to create a summarizing layers and relays that help manage that. AI isn't going to create infinite bandwidth. It's as likely to increase entropy and introduce noise.

CharlieDigital 2y ago
It's not that it makes more channels for all of us, it creates a channel for each of us.
- mathgeek 2y ago
  There are already more channels than that, as bots training off the media produced by bots will just continue to grow.

IAmGraydon 2y ago

The 'bandwidth' analogy breaks down because it assumes AI's value lies in processing more information, rather than processing information more intelligently. Increasing broadcast bandwidth added linear value, but AI's advancements come from complexity, nuance, and understanding - not just sheer volume of data. 'Infinite bandwidth' doesn't guarantee better insights or decision-making; it may even lead to information overload and decreased relevance.

CharlieDigital 2y ago
Gen AI's value lies in producing more variants of information. Infinitely many. If you and I prompt OpenAI to generate an image of a woman holding a candle, we'll get two totally novel instances.

robotresearcher 2y ago

> The pattern holds true for AI. AI is going to create "infinite bandwidth". I’ve worked in AI for more than 30 years and I have no idea what you mean by this. Can you explain?

CharlieDigital 2y ago
There is a ratio of producers to consumers in all media. There's two ways to think about bandwidth. One is the physical capacity. The other is the content that can be produced and distributed. We once had 3 channels equating to a maximum of 72h of content in a 24h period. Now we have YouTube which is orders of magnitude more content and bandwidth. The constraint now is the ratio of producers to consumers. Some creator had to create the exact content that you want. What if gen AI can create the exact content and media experience that you want? Effectively pushing the ratio of producers and consumers towards 1 so that every experience is unique? It is effectively as if there was infinite bandwidth to create and distribute content. You are no longer constrained by physical bandwidth and no longer constrained by production bandwidth (actual creators making the content). You want your AI generated news reel delivered by Walter Cronkite. I want mine delivered by Barbara Walters wearing a fake mustache while standing on one hand on the moon. It is as if there are infinite producers. I write a bit more on this topic here: https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and-why-you-should-be-too/ https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and...
- pixl97 2y ago
  What happens after the effective end of society?

slowmovintarget 2y ago

> AI is going to create "infinite bandwidth". For whom? If you mean infinite outpouring, then yes, but it will drown us in a sea of noise. We've constructed a Chinese Room for the mind. The computer was a bicycle, but this is something different. Bandwidth is carrying ability, and the current incarnation of "AI" does not increase signal. It takes vastly more resources to produce something close enough, but not quite... it.

CharlieDigital 2y ago
Just as there are thousands of videos on YouTube for making pancakes, there will one day be infinite videos for making pancakes. That visual interface will watch as you prep your pancakes and give you tips, suggest a substitute if you are missing an ingredient. Your experience with that recipe will be one of "infinitely" many.
pixl97 2y ago
Eh even current AI where it makes a summary of your preferences is ever so slightly increasing signal. I don't expect this ability to diminish over time, but instead increase, which would lead to more personalized signaling.

diffeomorphism 2y ago

Vision fails the discoverability test: Buttons that don't look like buttons. Fake download buttons. Long press to do something. Swipe in the shape of a hexagon. Grey text on light grey background. Also: thanks for tuning in, raid shadow legends, many people ask, but how... Anyway, you need these two lines of text (20 minutes YouTube videos could have been a half page of text) Finally: Huge output of bad quality and very, very limited input capacity. So "infinite bandwidth in" and then horrible traffic jam out.