2 ms·

I don’t train LLMs from scratch, but I have: 3x4090s 1xTesla A100 Lots of fine tuning, attention visualisation, evaluation of embeddings and different embeddi

I don’t train LLMs from scratch, but I have: 3x4090s 1xTesla A100 Lots of fine tuning, attention visualisation, evaluation of embeddings and different embedding generation methods, not just LLMs though I use them a lot for deep nets of many kinds Both for my day job (hedge fund) and my hobby project https://atomictessellator.com https://atomictessellator.com It’s summer here in NZ and I have these in servers mounted in a freestanding server rack beside my desk, and it is very hot in here XD

2099miles 3y ago
Can you talk about the attention visualization a bit. What’s the benefit. I have been plotting attention masks and even visualizing them over sentences but it obv doesn’t result in a ton of insight due to the sentences changing from words to much more complex vectors that represent those words after being processed through attentions 32x.
- malux85 3y ago
  I am using attention weighted pooling strategy to generate cross-attentions across N transformer heads, this allows us to visualize which parts of an output were caused by which part of the input -- this is used to debug summarization. Only works for encoder/decoder networks - i.e. ones where you have cross attentions
bavell 3y ago
Very cool website, love the aesthetic! Would love to know how you made the animations and the algos behind them.
- malux85 3y ago
  Thank you! The animation at the top is a GPU shader of the transition between 1s, 2p and 3d electron wave functions in a hydrogen atom, the animated crystal of sulpher I made using Maya and then animated into/out of wireframe by loading it and spinning it in WebGL, the other ones further down are just screen recordings. You might like this too - this last weekend I finished writing a topological contour GPU shader for visualizing electron density grids, here is a molecule of buckminsterfullerene - the contour generator is running on the GPU, which is how I can keep the framerate so high. https://drive.google.com/file/d/1xZkXsWDZtpe0H3BKSkdzvjOA3UEfp1gk/view?usp=sharing https://drive.google.com/file/d/1xZkXsWDZtpe0H3BKSkdzvjOA3UE...
  - ProllyInfamous 3y ago
    WOW! I was undergraduate chemistry (decades ago) and if these visualizations had existed back then... I may have actually done something with my chemistry training! The `SMILES` number text input had me in flashbacks about some of the fledgling research I half-assed waybackwhen. Thanks for providing such a wonderful resource for future chemists!

2099miles 3y ago

Can you talk about the attention visualization a bit. What’s the benefit. I have been plotting attention masks and even visualizing them over sentences but it obv doesn’t result in a ton of insight due to the sentences changing from words to much more complex vectors that represent those words after being processed through attentions 32x.

malux85 3y ago
I am using attention weighted pooling strategy to generate cross-attentions across N transformer heads, this allows us to visualize which parts of an output were caused by which part of the input -- this is used to debug summarization. Only works for encoder/decoder networks - i.e. ones where you have cross attentions

bavell 3y ago

Very cool website, love the aesthetic! Would love to know how you made the animations and the algos behind them.

malux85 3y ago
Thank you! The animation at the top is a GPU shader of the transition between 1s, 2p and 3d electron wave functions in a hydrogen atom, the animated crystal of sulpher I made using Maya and then animated into/out of wireframe by loading it and spinning it in WebGL, the other ones further down are just screen recordings. You might like this too - this last weekend I finished writing a topological contour GPU shader for visualizing electron density grids, here is a molecule of buckminsterfullerene - the contour generator is running on the GPU, which is how I can keep the framerate so high. https://drive.google.com/file/d/1xZkXsWDZtpe0H3BKSkdzvjOA3UEfp1gk/view?usp=sharing https://drive.google.com/file/d/1xZkXsWDZtpe0H3BKSkdzvjOA3UE...
- ProllyInfamous 3y ago
  WOW! I was undergraduate chemistry (decades ago) and if these visualizations had existed back then... I may have actually done something with my chemistry training! The `SMILES` number text input had me in flashbacks about some of the fledgling research I half-assed waybackwhen. Thanks for providing such a wonderful resource for future chemists!