3 ms·
I built a tool called Meaning Machine to let you see how language models "read" your words. It walks through the core stages — tokenization, POS tagging, depen
by jdspiral 1y ago
I built a tool called Meaning Machine to let you see how language models "read" your words.
It walks through the core stages — tokenization, POS tagging, dependency parsing, embeddings — and visualizes how meaning gets fragmented and simulated along the way.
Built with Streamlit, spaCy, BERT, and Plotly. It’s fast, interactive, and aimed at anyone curious about how LLMs turn your sentence into structured data.
Would love thoughts and feedback from the HN crowd — especially devs, linguists, or anyone working with or thinking about NLP systems.
GitHub: https://github.com/jdspiral/meaning-machine https://github.com/jdspiral/meaning-machine
Live Demo: https://meaning-machine.streamlit.app https://meaning-machine.streamlit.app
- macleginn 1y agoThe presentation is nice! The main point, however, is a bit misleading. From the title, one would assume that we will see something about how LMs do all these things implicitly (as was famously shown for syntax in this paper: https://arxiv.org/pdf/2005.04511 https://arxiv.org/pdf/2005.04511, for example), but instead the input is simply given to a bunch of pretrained task-specific models, which may not have much in common and definitely do not have very much in common with what today's LLMs are doing under the hood.
- toxik 1y agoYou shouldn’t link directly to the pdf, here is the abs page https://arxiv.org/abs/2005.04511 https://arxiv.org/abs/2005.04511
- selfhoster11 1y agoI''m getting an error message with Streamlit: You do not have access to this app or it does not exist
- jdspiral 1y agoI moved the app, it’s now tokenizer-machine.streamlit.app.