4 ms·

Offtopic sort of, but does anyone know if folks are working on combining vision and natural language in one model? I think that could wield some interesting re

Offtopic sort of, but does anyone know if folks are working on combining vision and natural language in one model? I think that could wield some interesting results.

abalaji 5y ago
yeah there has definitely been work done in that space: it’s called multi-modal models not sure if this is the latest work but here’s some results from Google’s AI Blog https://ai.googleblog.com/2017/06/multimodel-multi-task-machine-learning.html?m=1 https://ai.googleblog.com/2017/06/multimodel-multi-task-mach...
- im3w1l 5y ago
  What would be really cool is neural networks with routing. Like circuit switching or packet switching. No idea how you would train such a beast though. Like imagine the vision part making a phonecall to the natural language part to ask it for help with something.
  - TylerLives 5y ago
    Sounds like The Society of Mind - https://en.m.wikipedia.org/wiki/Society_of_Mind https://en.m.wikipedia.org/wiki/Society_of_Mind
  - wittenator 5y ago
    Capsule networks have a routing algorithm as far as I know
sillysaurusx 5y ago
https://github.com/openai/CLIP https://github.com/openai/CLIP The results are quite interesting: https://www.reddit.com/r/Art/comments/p866wv/deep_dive_meai_model_digital_2021/ https://www.reddit.com/r/Art/comments/p866wv/deep_dive_meai_...
- codetrotter 5y ago
  And here is a short guide and a link to a Google Collab notebook that anyone can use to create their own AI-powered art using VQGAN+CLIP: https://sourceful.us/doc/935/introduction-to-vqganclip https://sourceful.us/doc/935/introduction-to-vqganclip