4 ms·

What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).

by FloorEgg 1y ago

jvanderbot 1y ago
Play in the real world generates a data point every few minutes. Seems a bit slow?
- FloorEgg 1y ago
  What are you basing that statement on? What exactly are you considering a "data point"? Are you assuming one model = one agent instance? I am pretty sure that there is more information (molecular structure) and functional information (I(Ex )) just in the room I am sitting in than all the unique, useful, digitized information on earth.
- pizzly 1y ago
  Humans experience (play in the real world) is multi modal though vision, sound, touch, pressure, muscle feedback, gravitational, etc. Its extremely rich in data. Its also not a data point its continuous stream of information. Also I would bet that humans synthesize data at the same time. Everytime we run multiple scenarios in our mind before choosing the one we execute without even thinking about it is synthesizing data. Also humans dream which is another form of data synthesizing. Allowing AI to interact with the real world is definitely a way to go.
  - mannykannot 1y ago
    That's true, but still, a single individual or small group living isolated in the real world will, over a lifetime, learn only a tiny fraction of what we can learn from the written knowledge accumulated over millennia. Having said that, I tend to agree that having AI interact with the world may be key: for one thing, I'm not sure whether there is any sense in which LLMs understand that most of the information content of language is about an external world.
tliltocatl 1y ago
That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.
- FloorEgg 1y ago
  Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis. So while it's expensive, we can trade scope-constrained robustness (replication and redundancy at many levels of abstraction), for broader cognitive scale and fragility (data centers can't repair themselves and self-replicate). Going to be interesting to see it all unfold... my bet is on stacking S-curves all the way.
  - airstrike 1y ago
    > Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. I believe you but I would love to know where this number came from just so I can read more about it
    - FloorEgg 1y ago
      It's napkin math so take it with a pinch of salt, but I am calculating the information stored in genome, assuming 2 bits per base pair, reducing to estimated 88% coding fraction to get the functional bits, and then dividing by cell volume. Did this for a few different types of cells and then averaged the result to around 1–10 Mbit/μm³ # If there are any bioinformaticians around please come eviscerate or confirm this calc # Then compared it to TSMC 2nm research macro of (38.1 Mbit/mm^2) normalized to cell scale: 0.00019 Mbit/μm³ Living Cells: 1–10 Mbit/μm³ Current best chips: 0.00019 Mbit/μm³ https://research.tsmc.com/page/memory/4.html https://research.tsmc.com/page/memory/4.html

What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).

jvanderbot 1y ago

Play in the real world generates a data point every few minutes. Seems a bit slow?

FloorEgg 1y ago
What are you basing that statement on? What exactly are you considering a "data point"? Are you assuming one model = one agent instance? I am pretty sure that there is more information (molecular structure) and functional information (I(Ex )) just in the room I am sitting in than all the unique, useful, digitized information on earth.
pizzly 1y ago
Humans experience (play in the real world) is multi modal though vision, sound, touch, pressure, muscle feedback, gravitational, etc. Its extremely rich in data. Its also not a data point its continuous stream of information. Also I would bet that humans synthesize data at the same time. Everytime we run multiple scenarios in our mind before choosing the one we execute without even thinking about it is synthesizing data. Also humans dream which is another form of data synthesizing. Allowing AI to interact with the real world is definitely a way to go.
- mannykannot 1y ago
  That's true, but still, a single individual or small group living isolated in the real world will, over a lifetime, learn only a tiny fraction of what we can learn from the written knowledge accumulated over millennia. Having said that, I tend to agree that having AI interact with the world may be key: for one thing, I'm not sure whether there is any sense in which LLMs understand that most of the information content of language is about an external world.

tliltocatl 1y ago

That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.

FloorEgg 1y ago
Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis. So while it's expensive, we can trade scope-constrained robustness (replication and redundancy at many levels of abstraction), for broader cognitive scale and fragility (data centers can't repair themselves and self-replicate). Going to be interesting to see it all unfold... my bet is on stacking S-curves all the way.
- airstrike 1y ago
  > Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. I believe you but I would love to know where this number came from just so I can read more about it
  - FloorEgg 1y ago
    It's napkin math so take it with a pinch of salt, but I am calculating the information stored in genome, assuming 2 bits per base pair, reducing to estimated 88% coding fraction to get the functional bits, and then dividing by cell volume. Did this for a few different types of cells and then averaged the result to around 1–10 Mbit/μm³ # If there are any bioinformaticians around please come eviscerate or confirm this calc # Then compared it to TSMC 2nm research macro of (38.1 Mbit/mm^2) normalized to cell scale: 0.00019 Mbit/μm³ Living Cells: 1–10 Mbit/μm³ Current best chips: 0.00019 Mbit/μm³ https://research.tsmc.com/page/memory/4.html https://research.tsmc.com/page/memory/4.html