33 ms·

SynJax: Jax library for efficient probabilistic modeling of structured objects

59 points by pizza 3y ago

uoaei 3y ago
Link should go to: https://github.com/deepmind/synjax https://github.com/deepmind/synjax The repo isn't even linked in the X post, which is strange.
- sampo 3y ago
  > The repo isn't even linked in the X post It's a thread of 4 tweets. Twitter has a limit of 280 characters per tweet, so usually people split their text into multiple tweets. The link is in the 4th one, as you might expect. https://nitter.net/milosstanojevic/status/1688896558790520832 https://nitter.net/milosstanojevic/status/168889655879052083...
  - uoaei 3y ago
    I'm logged out and have adblockers on. I used to be able to see the posts that come after, but now I only see the first one. Maybe that's a new change. Thanks for the nitter link.
    - zote 3y ago
      For anyone else not in the know, Twitter(X?) now limits your view to only the tweet a link points to, if you're not logged in.
      - pizza 3y ago
        Annoying.. I posted the thread since I felt it motivated the project a bit better for those who aren't already familiar with the models. Oh well.
    - sampo 3y ago
      > Maybe that's a new change. Yes, I think about 3 months ago.
  - throwaway290 3y ago
    > Twitter has a limit of 280 characters per tweet I have seen much longer posts on there, like a full screen of text... how come?
    - Iwan-Zotow 3y ago
      I think if you pay, you'll get more
      - throwaway290 3y ago
        Classic!
- sdfghswe 3y ago
  It's not an "X post", it's a tweet.
  - deleted 3y ago
    [deleted]
lordofgibbons 3y ago
Could anyone please explain what this would be used for? Does this project define NNs that are designed for datasets with a known structure?
- gh02t 3y ago
  The things in SynJax aren't neural nets themselves as far as I see poking around, it's a bunch of different [complicated] probability distributions that you might encounter when modeling neural nets or related tasks stochastically. More of a toolbox of components implemented in JAX for advanced statistical modeling inside of a NN. Could also be useful apart from NNs, too.
  - nextos 3y ago
    Yes, that is confusing as the intro in the README says "SynJax is a neural network library for JAX structured probability distributions". However, contrary to my expectations after reading that, they do not have lots primitives to mix neural and probabilistic models. They seem to be heading in that direction, with e.g. connectionist temporal classification (CTC). Still very interesting library, seems to provide massive efficiency gains for difficult generative models such as PCFGs.
- Q6T46nT668w6i3m 3y ago
  It’s useful when you’re modeling a structure, e.g., P(x, y) or P(x | y), rather than a distribution. This is helpful in situations where generating samples is bad (CRFs) or your data imposes dependent features (HMMs). It appears from the README (and the authors’ previous work) this is useful for arbitrary structures too.
- 6gvONxR4sf7o 3y ago
  This is probably a good start https://en.wikipedia.org/wiki/Graphical_model https://en.wikipedia.org/wiki/Graphical_model It's traditionally fairly challenging to implement a relatively straightforward graphical model, but automatic differentiation changed that. Some people are working on getting the best of graphical models and black box models (like neural networks), which is where things like this come in. Because it's jax, you get autograd and gpu/tpu acceleration for free, which further lets you stick these things in with other models, including black box models like NNs. An example application is in sentence part of speech tagging (noun, adjective, etc). You could model each word's part of speech as a separate prediction, but you know that "noun-verb-noun" is more common than "noun-noun-noun" so you know your predictions should influence each other. Stuff like this makes that easier. edit: The authors of this library also put a paper on arxiv describing it https://arxiv.org/pdf/2308.03291.pdf https://arxiv.org/pdf/2308.03291.pdf The abstract gives a good sense of what they're going after: > The development of deep learning software libraries enabled significant progress in the field by allowing users to focus on modeling, while letting the library to take care of the tedious and time-consuming task of optimizing execution for modern hardware accelerators. However, this has benefited only particular types of deep learning models, such as Transformers, whose primitives map easily to the vectorized computation. The models that explicitly account for structured objects, such as trees and segmentations, did not benefit equally because they require custom algorithms that are difficult to implement in a vectorized form. SynJax directly addresses this problem by providing an efficient vectorized implementation of inference algorithms for structured distributions covering alignment, tagging, segmentation, constituency trees and spanning trees. > With SynJax we can build large-scale differentiable models that explicitly model structure in the data. The code is available at https://github.com/deepmind/synjax https://github.com/deepmind/synjax.

59 points by pizza 3y ago

Link should go to: https://github.com/deepmind/synjax https://github.com/deepmind/synjax The repo isn't even linked in the X post, which is strange.

sampo 3y ago
> The repo isn't even linked in the X post It's a thread of 4 tweets. Twitter has a limit of 280 characters per tweet, so usually people split their text into multiple tweets. The link is in the 4th one, as you might expect. https://nitter.net/milosstanojevic/status/1688896558790520832 https://nitter.net/milosstanojevic/status/168889655879052083...
- uoaei 3y ago
  I'm logged out and have adblockers on. I used to be able to see the posts that come after, but now I only see the first one. Maybe that's a new change. Thanks for the nitter link.
  - zote 3y ago
    For anyone else not in the know, Twitter(X?) now limits your view to only the tweet a link points to, if you're not logged in.
    - pizza 3y ago
      Annoying.. I posted the thread since I felt it motivated the project a bit better for those who aren't already familiar with the models. Oh well.
  - sampo 3y ago
    > Maybe that's a new change. Yes, I think about 3 months ago.
- throwaway290 3y ago
  > Twitter has a limit of 280 characters per tweet I have seen much longer posts on there, like a full screen of text... how come?
  - Iwan-Zotow 3y ago
    I think if you pay, you'll get more
    - throwaway290 3y ago
      Classic!
sdfghswe 3y ago
It's not an "X post", it's a tweet.
- deleted 3y ago
  [deleted]

lordofgibbons 3y ago

Could anyone please explain what this would be used for? Does this project define NNs that are designed for datasets with a known structure?

gh02t 3y ago
The things in SynJax aren't neural nets themselves as far as I see poking around, it's a bunch of different [complicated] probability distributions that you might encounter when modeling neural nets or related tasks stochastically. More of a toolbox of components implemented in JAX for advanced statistical modeling inside of a NN. Could also be useful apart from NNs, too.
- nextos 3y ago
  Yes, that is confusing as the intro in the README says "SynJax is a neural network library for JAX structured probability distributions". However, contrary to my expectations after reading that, they do not have lots primitives to mix neural and probabilistic models. They seem to be heading in that direction, with e.g. connectionist temporal classification (CTC). Still very interesting library, seems to provide massive efficiency gains for difficult generative models such as PCFGs.
Q6T46nT668w6i3m 3y ago
It’s useful when you’re modeling a structure, e.g., P(x, y) or P(x | y), rather than a distribution. This is helpful in situations where generating samples is bad (CRFs) or your data imposes dependent features (HMMs). It appears from the README (and the authors’ previous work) this is useful for arbitrary structures too.
6gvONxR4sf7o 3y ago
This is probably a good start https://en.wikipedia.org/wiki/Graphical_model https://en.wikipedia.org/wiki/Graphical_model It's traditionally fairly challenging to implement a relatively straightforward graphical model, but automatic differentiation changed that. Some people are working on getting the best of graphical models and black box models (like neural networks), which is where things like this come in. Because it's jax, you get autograd and gpu/tpu acceleration for free, which further lets you stick these things in with other models, including black box models like NNs. An example application is in sentence part of speech tagging (noun, adjective, etc). You could model each word's part of speech as a separate prediction, but you know that "noun-verb-noun" is more common than "noun-noun-noun" so you know your predictions should influence each other. Stuff like this makes that easier. edit: The authors of this library also put a paper on arxiv describing it https://arxiv.org/pdf/2308.03291.pdf https://arxiv.org/pdf/2308.03291.pdf The abstract gives a good sense of what they're going after: > The development of deep learning software libraries enabled significant progress in the field by allowing users to focus on modeling, while letting the library to take care of the tedious and time-consuming task of optimizing execution for modern hardware accelerators. However, this has benefited only particular types of deep learning models, such as Transformers, whose primitives map easily to the vectorized computation. The models that explicitly account for structured objects, such as trees and segmentations, did not benefit equally because they require custom algorithms that are difficult to implement in a vectorized form. SynJax directly addresses this problem by providing an efficient vectorized implementation of inference algorithms for structured distributions covering alignment, tagging, segmentation, constituency trees and spanning trees. > With SynJax we can build large-scale differentiable models that explicitly model structure in the data. The code is available at https://github.com/deepmind/synjax https://github.com/deepmind/synjax.