2 ms·
https://x.com/sukjun_hwang/status/1943703574908723674 https://x.com/sukjun_hwang/status/1943703574908723674 > Tokenization has been the final barrier to truly
by hatthew 1y ago
https://x.com/sukjun_hwang/status/1943703574908723674 https://x.com/sukjun_hwang/status/1943703574908723674
> Tokenization has been the final barrier to truly end-to-end language models.
> We developed the H-Net: a hierarchical network that replaces tokenization with a dynamic chunking process directly inside the model, automatically discovering and operating over meaningful units of data
- b112 1y agoGood to see. Will be better to see once the scope is clear. But something such as this is required to move towards actual intelligence.