5 ms·

Let's put it this way: Before 2023 I always thought that all the bugs and glitches of technology in Star Trek were totally made up and would never happen this

Let's put it this way: Before 2023 I always thought that all the bugs and glitches of technology in Star Trek were totally made up and would never happen this way. Post-LLM I am absolutely certain that they will happen exactly that way. I am not sure what LLM integrations have to do with engineering anymore, or why it makes sense to essentially put all your company's infrastructure into external control. And that is not even scratching the surface with the lack of reproducibility at every single step of the way. It "somehow works" isn't engineering.

withinboredom 1y ago
For someone who isn't a trek fan -- can you elaborate on this?
- whoknowsidont 1y ago
  The computer, at least aboard the enterprise, is kind of portrayed as a singular monolithic AI that can access the majority of the ship's subsystems (different networks, other computer/control units, etc) and functions. It can control nearly every aspect of the ship while talking with its human crew / commanding officers. So very much like an LLM accessing multiple pieces of functionality across different tools and API endpoints (if you want to imagine it that way). While it is seemingly very knowledgeable, it is rather stupid. It gets duped by nefarious actors or has a class of bugs that are elementary that put the crew into awkward positions. Most professional software engineers might have previously looked as these scenarios as implausible, given the "failure model" of current software is quite blunt, and especially given how far into the future the series took place. Now we see that computational tasks are becoming less predictable, less straight-forward, with cascading failures instead of blunt, direct failures. Interacting with an LLM might be compared to talking with a person in psychosis when it starts to hallucinate. So you get things like this in the Star Trek universe: https://www.youtube.com/watch?v=kUJh7id0lK4 https://www.youtube.com/watch?v=kUJh7id0lK4 Which make a lot more sense, become a lot more plausible and a lot more relatable with our current implementations of AI/LLM's.
  - cookiengineer 1y ago
    Excellent comment, couldn't have described it better. I wanted to add that in Star Trek they always talk with techno babble things like "Computer, create a matrix from a historic person who was knowledgeable in a specialized surgery field" and then the Hologram room creates that avatar's approximation, with the programming and simulated/hallucinated expertise. The holodeck is a special kind of weird because sooo many accidents happen because of sloppy coding that the AI of the ship's computer created as flawed programs that later then hurt the crew members because of failing or ignored/bypassed safety protocols, which we see now as the rising field of prompt engineering in redteams. Additionally, in Star Trek instead of coding on tablets, they usually just show analytics data or debug views of what the ship's computer created. The crew never actually code on a computer, and if they do they primarily just "vibe code" it by saying absurd things like "Computer, analyze the enemy ship's frequency and create a phasing shield emitter to block their phasers" (or something like that) and the computer generates those programs on the fly. The cool part that I liked the most is when Voyager's neural packs (think of them as the AI-to-system control adapters) actually got sick with a biological virus because they were essentially made out of brain matter.
    - rdtsc 1y ago
      > The cool part that I liked the most is when Voyager's neural packs (think of them as the AI-to-system control adapters) actually got sick with a biological virus because they were essentially made out of brain matter. I liked that part too. I hadn't paid attention much before but that was a fun revelation that the computer is run by a bunch of brain tissue pouches. The LLM "guts" to speak is pretty much a collection of brain tissue clumps semantically, with weights and connections as opposed, to some database of logical assertions, like an expert systems people envisioned in the 1980s.
    - mewpmewp2 1y ago
      These are such great points. I'm truly mind boggled how they got those ideas so right while people previously wouldn't have believed this direction to be correct at all. Because people would think if we reach AGI there's nothing we have to worry about because AI will be able to handle it, but whether we reach AGI or what kind of steps are there in between, for a period of time the behaviour displayed in Star Trek will be very plausible. Asking AI to create elaborate debug views is something that I definitely spend a lot of time doing when vibe coding. And trying to orchestrate seemingly ridiculous scenarios to either keep AI in its tracks, or brainstorm future directions, etc. AI generates close to 100% of my code, but I have to also ask it to create guardrails for itself, special linting rules that I would never use myself so it avoids the common errors it does. It can generate the code, but it out of the box won't keep itself in its tracks, which leads to very interesting scenarios and potential absurd stories.
wredcoll 1y ago
> It "somehow works" isn't engineering. But it sure is fast.
- drweevil 1y ago
  It somehow works, until it doesn't. That's when the pain starts.
  - jay_kyburz 1y ago
    Like Capitalism, or even Democracy :)
- _kb 1y ago
  Move faster and break more things.
  - HighGoldstein 1y ago
    Does this mean moving fastest would break infinitely many things?
    - mewpmewp2 1y ago
      Well if you think about it, if you do things infinitely fast, there's still infinitely many things that are broken about a theoretical feature set.
  - Henchman21 1y ago
    Everything is already broken. Can we maybe stop breaking stuff and start fixing whats been broken?
- whateveracct 1y ago
  For some definition of "fast" haha
zeristor 1y ago
“Somehow Palpatine survived” broke Star Wars, although no AI was used in that.
mewpmewp2 1y ago
> It "somehow works" isn't engineering. Consider this - everything will "somehow work" if the system has been there for generations and is complex enough that no single human brain can keep everything about it in the brain at any given time. It is easy to keep a system high quality, well maintained, well understood for a year with a small team, but imagine doing that for 100+ years with a system constantly evolving in complexity with generations of maintainers, people being rotated.
- cookiengineer 1y ago
  Maybe what's happening right now is that grumpy old developers like me are starting to realize that despite our efforts for professionalism (with methodologies like TDD, DDD etc) the "good enough" approach is what companies have been doing all along. Maybe we are stuck in the past and are in the process of disillusioning ourselves, and that's why the older generation is so reluctant to use vibe coding tools? What's great is that a whole new area of research is available, because previous methodologies don't seem to work anymore and we have to find different ways to get better results out of vibe coding workflows.
bdangubic 1y ago
You just described every company and every system before we had llms…
BiraIgnacio 1y ago
When I look at how AI and LM systems, sw, platforms, have been built over the last decade (and are still being built), I can't help but to think that what really every mattered was the response the system produced. Never mind the quality or if it's even going to work in production. And maybe that's all that's needed, I don't really know. I'm sure that's just me being the old curmudgeon of a software engineer I am, wishing people thought about more than one user using a system and 2 engineers supporting it.

withinboredom 1y ago

For someone who isn't a trek fan -- can you elaborate on this?

whoknowsidont 1y ago
The computer, at least aboard the enterprise, is kind of portrayed as a singular monolithic AI that can access the majority of the ship's subsystems (different networks, other computer/control units, etc) and functions. It can control nearly every aspect of the ship while talking with its human crew / commanding officers. So very much like an LLM accessing multiple pieces of functionality across different tools and API endpoints (if you want to imagine it that way). While it is seemingly very knowledgeable, it is rather stupid. It gets duped by nefarious actors or has a class of bugs that are elementary that put the crew into awkward positions. Most professional software engineers might have previously looked as these scenarios as implausible, given the "failure model" of current software is quite blunt, and especially given how far into the future the series took place. Now we see that computational tasks are becoming less predictable, less straight-forward, with cascading failures instead of blunt, direct failures. Interacting with an LLM might be compared to talking with a person in psychosis when it starts to hallucinate. So you get things like this in the Star Trek universe: https://www.youtube.com/watch?v=kUJh7id0lK4 https://www.youtube.com/watch?v=kUJh7id0lK4 Which make a lot more sense, become a lot more plausible and a lot more relatable with our current implementations of AI/LLM's.
- cookiengineer 1y ago
  Excellent comment, couldn't have described it better. I wanted to add that in Star Trek they always talk with techno babble things like "Computer, create a matrix from a historic person who was knowledgeable in a specialized surgery field" and then the Hologram room creates that avatar's approximation, with the programming and simulated/hallucinated expertise. The holodeck is a special kind of weird because sooo many accidents happen because of sloppy coding that the AI of the ship's computer created as flawed programs that later then hurt the crew members because of failing or ignored/bypassed safety protocols, which we see now as the rising field of prompt engineering in redteams. Additionally, in Star Trek instead of coding on tablets, they usually just show analytics data or debug views of what the ship's computer created. The crew never actually code on a computer, and if they do they primarily just "vibe code" it by saying absurd things like "Computer, analyze the enemy ship's frequency and create a phasing shield emitter to block their phasers" (or something like that) and the computer generates those programs on the fly. The cool part that I liked the most is when Voyager's neural packs (think of them as the AI-to-system control adapters) actually got sick with a biological virus because they were essentially made out of brain matter.
  - rdtsc 1y ago
    > The cool part that I liked the most is when Voyager's neural packs (think of them as the AI-to-system control adapters) actually got sick with a biological virus because they were essentially made out of brain matter. I liked that part too. I hadn't paid attention much before but that was a fun revelation that the computer is run by a bunch of brain tissue pouches. The LLM "guts" to speak is pretty much a collection of brain tissue clumps semantically, with weights and connections as opposed, to some database of logical assertions, like an expert systems people envisioned in the 1980s.
  - mewpmewp2 1y ago
    These are such great points. I'm truly mind boggled how they got those ideas so right while people previously wouldn't have believed this direction to be correct at all. Because people would think if we reach AGI there's nothing we have to worry about because AI will be able to handle it, but whether we reach AGI or what kind of steps are there in between, for a period of time the behaviour displayed in Star Trek will be very plausible. Asking AI to create elaborate debug views is something that I definitely spend a lot of time doing when vibe coding. And trying to orchestrate seemingly ridiculous scenarios to either keep AI in its tracks, or brainstorm future directions, etc. AI generates close to 100% of my code, but I have to also ask it to create guardrails for itself, special linting rules that I would never use myself so it avoids the common errors it does. It can generate the code, but it out of the box won't keep itself in its tracks, which leads to very interesting scenarios and potential absurd stories.

wredcoll 1y ago

> It "somehow works" isn't engineering. But it sure is fast.

drweevil 1y ago
It somehow works, until it doesn't. That's when the pain starts.
- jay_kyburz 1y ago
  Like Capitalism, or even Democracy :)
_kb 1y ago
Move faster and break more things.
- HighGoldstein 1y ago
  Does this mean moving fastest would break infinitely many things?
  - mewpmewp2 1y ago
    Well if you think about it, if you do things infinitely fast, there's still infinitely many things that are broken about a theoretical feature set.
- Henchman21 1y ago
  Everything is already broken. Can we maybe stop breaking stuff and start fixing whats been broken?
whateveracct 1y ago
For some definition of "fast" haha

zeristor 1y ago

“Somehow Palpatine survived” broke Star Wars, although no AI was used in that.

mewpmewp2 1y ago

> It "somehow works" isn't engineering. Consider this - everything will "somehow work" if the system has been there for generations and is complex enough that no single human brain can keep everything about it in the brain at any given time. It is easy to keep a system high quality, well maintained, well understood for a year with a small team, but imagine doing that for 100+ years with a system constantly evolving in complexity with generations of maintainers, people being rotated.

cookiengineer 1y ago
Maybe what's happening right now is that grumpy old developers like me are starting to realize that despite our efforts for professionalism (with methodologies like TDD, DDD etc) the "good enough" approach is what companies have been doing all along. Maybe we are stuck in the past and are in the process of disillusioning ourselves, and that's why the older generation is so reluctant to use vibe coding tools? What's great is that a whole new area of research is available, because previous methodologies don't seem to work anymore and we have to find different ways to get better results out of vibe coding workflows.

bdangubic 1y ago

You just described every company and every system before we had llms…

BiraIgnacio 1y ago

When I look at how AI and LM systems, sw, platforms, have been built over the last decade (and are still being built), I can't help but to think that what really every mattered was the response the system produced. Never mind the quality or if it's even going to work in production. And maybe that's all that's needed, I don't really know. I'm sure that's just me being the old curmudgeon of a software engineer I am, wishing people thought about more than one user using a system and 2 engineers supporting it.