4 ms·

I can’t see how this is possible. You’re losing so much information.

by bilsbie 7mo ago

MarsIronPI 7mo ago
It's because they're natively trained with 1 bit, so it's not losing anything. Now, the question might be how they manage to get decent predictive performance with such little precision. That I don't know.
- syntaxpr 7mo ago
  Not training. Transposing rows/columns of matrices to group 128 parameters with similar (shared) scale factor. Qwen-3 model.
  - MarsIronPI 7mo ago
    I'm not sure what you mean. Could you please elaborate?
txrx0000 7mo ago
I always remind myself and everyone else that human DNA is "only" 1.6 GB of data, and yet it encodes all of the complex systems of the human body including the brain, and can replicate itself. Our intuitive feel of how much stuff can be packed into how many bits are probably way off from the true limits of physics.
- kennywinker 7mo ago
  And anybody who’s ever met a baby can tell you, they score very poorly on most llm benchmarks.
- humanjir 7mo ago
  That's not strictly true - DNA doesnt replicate itself, a cell with DNA replicates itself. You need to count the information contained in the non-DNA part of the cell too. Just in case it's not obvious, you can't take human DNA and put it in a cat cell, it won't work, that cell won't replicate.
  - adrian_b 7mo ago
    True. For now, the DNA replication and the synthesis of RNA and proteins using the information stored in DNA are the best understood parts about how a cell grows and divides, but how other complex cellular structures, e.g. membranes or non-ribosomal peptides, are assembled and replicated is much less understood. We need more years of research, perhaps up to a decade or two, until we will be able to know the entire amount of information describing a simple bacterial cell, and perhaps more than that for a much more complex eukaryotic cell.
- freakynit 7mo ago
  Human DNA has 3.2 billion base pairs, and with 2x the information density compared to binary systems (due to 4-letters as opposed 2), that's roughly 800MB of informational data. Second, what's even more crazy is that roughly 98% of that DNA is actually non-coding.. just junk. So, we are talking about encoding entirety of the logic to construct a human body in just around 16MB of data!!! That's some crazy levels of recursive compression.. maybe it's embedding "varying" parsing logic, mixed with data, along the chain.
  - kgeist 7mo ago
    >Second, what's even more crazy is that roughly 98% of that DNA is actually non-coding.. just junk. I think it's a myth that non-coding DNA is junk. Say: https://www.nature.com/articles/444130a https://www.nature.com/articles/444130a >'Non-coding' DNA may organize brain cell connections.
  - adrian_b 7mo ago
    As another poster has said, much of the "junk" is not junk. The parts of the DNA with known functions encode either proteins or RNA molecules, being templates for their synthesis. The parts with unknown functions include some amount of true junk caused by various historical accidents that have been replicated continuously until now, but they also include a lot of DNA that seems to have a role in controlling how the protein or RNA genes are expressed (i.e. turning off or on the synthesis of specific proteins or RNAs), by mechanisms not well understood yet.
- orbital-decay 7mo ago
  It encodes the data on top of locally optimal trajectories in the physical world that were learned in millions of years of evolution. Treat this as context, not weights.
in-silico 7mo ago
In theory you do lose information compared to parameters with more bits. In practice, neural networks aren't able to store much more than 2-4 bits of useful information per parameter (regardless of the precision), so models like this are mostly getting rid of redundancy.

I can’t see how this is possible. You’re losing so much information.

by bilsbie 7mo ago

I can’t see how this is possible. You’re losing so much information.

It's because they're natively trained with 1 bit, so it's not losing anything. Now, the question might be how they manage to get decent predictive performance with such little precision. That I don't know.

syntaxpr 7mo ago
Not training. Transposing rows/columns of matrices to group 128 parameters with similar (shared) scale factor. Qwen-3 model.
- MarsIronPI 7mo ago
  I'm not sure what you mean. Could you please elaborate?

txrx0000 7mo ago

I always remind myself and everyone else that human DNA is "only" 1.6 GB of data, and yet it encodes all of the complex systems of the human body including the brain, and can replicate itself. Our intuitive feel of how much stuff can be packed into how many bits are probably way off from the true limits of physics.

kennywinker 7mo ago
And anybody who’s ever met a baby can tell you, they score very poorly on most llm benchmarks.
humanjir 7mo ago
That's not strictly true - DNA doesnt replicate itself, a cell with DNA replicates itself. You need to count the information contained in the non-DNA part of the cell too. Just in case it's not obvious, you can't take human DNA and put it in a cat cell, it won't work, that cell won't replicate.
- adrian_b 7mo ago
  True. For now, the DNA replication and the synthesis of RNA and proteins using the information stored in DNA are the best understood parts about how a cell grows and divides, but how other complex cellular structures, e.g. membranes or non-ribosomal peptides, are assembled and replicated is much less understood. We need more years of research, perhaps up to a decade or two, until we will be able to know the entire amount of information describing a simple bacterial cell, and perhaps more than that for a much more complex eukaryotic cell.
freakynit 7mo ago
Human DNA has 3.2 billion base pairs, and with 2x the information density compared to binary systems (due to 4-letters as opposed 2), that's roughly 800MB of informational data. Second, what's even more crazy is that roughly 98% of that DNA is actually non-coding.. just junk. So, we are talking about encoding entirety of the logic to construct a human body in just around 16MB of data!!! That's some crazy levels of recursive compression.. maybe it's embedding "varying" parsing logic, mixed with data, along the chain.
- kgeist 7mo ago
  >Second, what's even more crazy is that roughly 98% of that DNA is actually non-coding.. just junk. I think it's a myth that non-coding DNA is junk. Say: https://www.nature.com/articles/444130a https://www.nature.com/articles/444130a >'Non-coding' DNA may organize brain cell connections.
- adrian_b 7mo ago
  As another poster has said, much of the "junk" is not junk. The parts of the DNA with known functions encode either proteins or RNA molecules, being templates for their synthesis. The parts with unknown functions include some amount of true junk caused by various historical accidents that have been replicated continuously until now, but they also include a lot of DNA that seems to have a role in controlling how the protein or RNA genes are expressed (i.e. turning off or on the synthesis of specific proteins or RNAs), by mechanisms not well understood yet.
orbital-decay 7mo ago
It encodes the data on top of locally optimal trajectories in the physical world that were learned in millions of years of evolution. Treat this as context, not weights.

in-silico 7mo ago

In theory you do lose information compared to parameters with more bits. In practice, neural networks aren't able to store much more than 2-4 bits of useful information per parameter (regardless of the precision), so models like this are mostly getting rid of redundancy.