9 ms·

Open sourcing Sonnet – a new library for constructing neural networks

262 points by lopespm 10y ago

drej 10y ago
Looks cool, but: "This installation is compatible with Linux/Mac OS X and Python 2.7." Sad face.
- peakai_grandma 10y ago
  Are you suggesting they add compatibility for Win?
  - baq 10y ago
    My guess is Python 3. Not supporting Python 3.5+ in a newly released project is disrespectful at this point.
    - ColanR 10y ago
      The problem could be one of their dependencies only supports 2.7. I know that's been an issue for me in the past.
      - inputcoffee 10y ago
        Possible, but this is the first year that I have more 3 only dependencies than 2 only dependencies. I would be curious what they are.
      - mijoharas 10y ago
        I agree, but the right thing to do is to update the dependency to support python 3.
        ColanR 10y ago
        In my case it was cmu Sphinx, which is a little too big for a simple update.
        mijoharas 10y ago
        Sorry, I kinda meant "for a large company that is releasing a library" (e.g. deepmind), different if it's a single dev.
      - aw3c2 10y ago
        Looking at the setup.py they only depend on tensorflow and some testing thing which both seem to support Python 3 just fine.
- wdroz 10y ago
  Python 2.7 only is a no-go for me.
  - mark_l_watson 10y ago
    I am not a Python developer, but I do use Python occasionally. Is Python 2.7 requirement really a big deal for mostly using other people's code (adding a bit of your own application layer code)? I use Anaconda and just have different environments for 2.7 and 3.x, TensorFlow or spaCy, or NLTK, etc. Really easy to switch to the environment I need.
    - ganfortran 10y ago
      It is more about ideology. The 3.x is catching up, but still not fast enough as the community wants it to. But 2.7 is here for a reason, and shaming people that used 2.7 to push 3.x adoption, in my opinion is a bad practice.
- WeDontNeedH1Bs 10y ago
  Yep, we're Python 3 only now. The clear distinction between strings and bytes is more important to us than the latest iteration of Neural Network hype.
- tiff_seattle 10y ago
  If this is text only, maybe it will work on Bash on Windows?
- ganfortran 10y ago
  Which is fine. It is DL library, not for string processing.
- ylem 10y ago
  Another possibility is this: https://docs.python.org/3/howto/cporting.html https://docs.python.org/3/howto/cporting.html If either their library (or a dependency) depends on something which binds to C, it could take some effort to make it python 3 compatible. (FWIW, I use 2 or 3 depending on the OS or libraries that I am dealing with)
dbcurtis 10y ago
For us NN newbies, could some one do a comparison to Keras?
- Cacti 10y ago
  Not really, but from a look at the documentation, this seems to be geared towards recurrent nets (and their internal organization) and keras is a little more focused on layer abstraction. Hopefully they improve the docs as they go along; I could follow along because I work in the area, but they're pretty researchy, and I would imagine most would have difficulty parsing them unless they're regularly reading papers in this area.
vmsp 10y ago
Where does this leave Keras? Wasn't it supposed to be the new high level interface to TF? EDIT: Also, what about TF-Slim?
- maccam912 10y ago
  Keras did make it into tf.contrib now (with the release candidate for 1.1). I don't see it going away any time soon. TF-Slim though I'm not sure about. Never really used it myself.
- eanzenberg 10y ago
  That's the big long-term question. The nice thing of this is really just having Google backing. Keras, though, still really incredible for usability.
- dontreact 10y ago
  Generally all of these high level interfaces are somewhat interoperable because of the graph nature of Tensorflow. It might make your code harder to read but you can use some layers from slim (for inception-like nets this is a decent reference https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim/python/slim/nets https://github.com/tensorflow/tensorflow/tree/master/tensorf...), and some layers from Keras. I don't know as much about sonnet, but it looks like a similar story.
- sbbq 10y ago
  Not sure. I think Keras is more meant for education and off-the-shelf models, not so much for research due to its abstraction. Or in other words, its less customizable. I think the probably best trade-off between low-level code and coding productivity would be tf.layers, which is also in tf.core now.
  - nl 10y ago
    Keras is heavily used in research.
mrdrozdov 10y ago
There's definitely some examples of useful ops here: # 1. Define our computation as some op def useful_op(input_a, input_b, use_clipping=True, remove_nans=False, solve_agi='maybe'): # ...
singhrac 10y ago
> Making Sonnet public allows other models created within DeepMind to be easily shared with the community I'm looking forward to this - DeepMind has a frustrating track record of not open sourcing their code. While it's their prerogative and I know of many valid reasons for doing so, the many other researchers who publish their (sometimes terribly hacked together) models have been incredibly helpful in verifying that their work, well, works.
qeternity 10y ago
As a startup that's in the midst of hiring, I can't wait to see how quickly CVs get updated to include this.
- ravenstine 10y ago
  Do you often find a significant number of CVs listing technologies that are too new for one to have gained any mastery over?
  - qeternity 10y ago
    No, but we've only been interviewing for a few weeks. However we see an unbelievable number of CVs that have a full paragraph dedicated to buzzwords and python libraries.
gcr 10y ago
This isn't "yet another completely different neural network library." This library just has some new layer types for TensorFlow. Looks like there are some new layers for special kinds of attention RNNs, word embeddings, alternate implementations of spatial transformers, and so on. They also have another Batch Norm implementation that of course requires tons of fiddling to work properly, a classic tf staple :-) As a machine learning environment, tf is so complex that different research groups have to define their own best practices for it. Using tf is like learning C++ where everyone learns a slightly different, mutually incompatible, but broadly overlapping subset of the language. We're just seeing a glimpse into DeepMind's specialized tooling along with reference implementations of the operations they use in their work. This will be really useful for researchers who want to mess with deepmind's ideas/papers, but I'm a bit relieved that there isn't anything claimed to be fundamentally paradigm-shifting about this release.
- Cacti 10y ago
  It's nice to see people building on top of TF rather than continuing to develop completely new frameworks. I've been working with TF in detail for several months now and it is a very nicely designed framework. It is quite low-level, as you pointed out, (and so can be a challenge to get used to, and why you see helper wrappers/libs like this and keras popping up) but ML models and methods are becoming increasingly complex and TF provides exactly the foundation and flexibility needed to implement this stuff while still being able to slap together some basic layers if you need to. It also helps that the engineering is very solid (e.g., distributed models and datasets) and most of the performance kinks by now have been worked out. For those maybe putting off working with TF because of its steepish learning curve, I'd strongly suggest you dive in. I learned last year in the usual method (docs + google), but the new OReilly books are the first good ones I've seen if that's your style. The analogy with C++ is a good one, and historically appropriate given how scarce GPU resources are now.
  - eanzenberg 10y ago
    If you're already using TF, have you looked into keras? Does it not have the complexity or control you are looking for?
likelynew 10y ago
Anyone knowledgable with sonnet here mind detailing how is it different from keras.
hla19 10y ago
So many implementations of the same thing reminds me the JavaScript/Widget library rush several years ago, the majority of them will be forgotten very soon, long live the projects from the giants
waleedka 10y ago
Tip for the project lead: A quick way to lose half of the early adopters is to release a library without Python 3 support. Luckily, fixing this is easy and shouldn't take more than a day or two. And the ROI is high if your goal is wider adoption.
- rch 10y ago
  It's also an easy way for an outsider to contribute.
ReeSilva 10y ago
It looks nice and I will certainly try it. But, no support to Py3? Common, guys, it's 2k17

262 points by lopespm 10y ago

drej 10y ago

Looks cool, but: "This installation is compatible with Linux/Mac OS X and Python 2.7." Sad face.

peakai_grandma 10y ago
Are you suggesting they add compatibility for Win?
- baq 10y ago
  My guess is Python 3. Not supporting Python 3.5+ in a newly released project is disrespectful at this point.
  - ColanR 10y ago
    The problem could be one of their dependencies only supports 2.7. I know that's been an issue for me in the past.
    - inputcoffee 10y ago
      Possible, but this is the first year that I have more 3 only dependencies than 2 only dependencies. I would be curious what they are.
    - mijoharas 10y ago
      I agree, but the right thing to do is to update the dependency to support python 3.
      - ColanR 10y ago
        In my case it was cmu Sphinx, which is a little too big for a simple update.
        mijoharas 10y ago
        Sorry, I kinda meant "for a large company that is releasing a library" (e.g. deepmind), different if it's a single dev.
    - aw3c2 10y ago
      Looking at the setup.py they only depend on tensorflow and some testing thing which both seem to support Python 3 just fine.
wdroz 10y ago
Python 2.7 only is a no-go for me.
- mark_l_watson 10y ago
  I am not a Python developer, but I do use Python occasionally. Is Python 2.7 requirement really a big deal for mostly using other people's code (adding a bit of your own application layer code)? I use Anaconda and just have different environments for 2.7 and 3.x, TensorFlow or spaCy, or NLTK, etc. Really easy to switch to the environment I need.
  - ganfortran 10y ago
    It is more about ideology. The 3.x is catching up, but still not fast enough as the community wants it to. But 2.7 is here for a reason, and shaming people that used 2.7 to push 3.x adoption, in my opinion is a bad practice.
WeDontNeedH1Bs 10y ago
Yep, we're Python 3 only now. The clear distinction between strings and bytes is more important to us than the latest iteration of Neural Network hype.
tiff_seattle 10y ago
If this is text only, maybe it will work on Bash on Windows?
ganfortran 10y ago
Which is fine. It is DL library, not for string processing.
ylem 10y ago
Another possibility is this: https://docs.python.org/3/howto/cporting.html https://docs.python.org/3/howto/cporting.html If either their library (or a dependency) depends on something which binds to C, it could take some effort to make it python 3 compatible. (FWIW, I use 2 or 3 depending on the OS or libraries that I am dealing with)

dbcurtis 10y ago

For us NN newbies, could some one do a comparison to Keras?

Cacti 10y ago
Not really, but from a look at the documentation, this seems to be geared towards recurrent nets (and their internal organization) and keras is a little more focused on layer abstraction. Hopefully they improve the docs as they go along; I could follow along because I work in the area, but they're pretty researchy, and I would imagine most would have difficulty parsing them unless they're regularly reading papers in this area.

vmsp 10y ago

Where does this leave Keras? Wasn't it supposed to be the new high level interface to TF? EDIT: Also, what about TF-Slim?

maccam912 10y ago
Keras did make it into tf.contrib now (with the release candidate for 1.1). I don't see it going away any time soon. TF-Slim though I'm not sure about. Never really used it myself.
eanzenberg 10y ago
That's the big long-term question. The nice thing of this is really just having Google backing. Keras, though, still really incredible for usability.
dontreact 10y ago
Generally all of these high level interfaces are somewhat interoperable because of the graph nature of Tensorflow. It might make your code harder to read but you can use some layers from slim (for inception-like nets this is a decent reference https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim/python/slim/nets https://github.com/tensorflow/tensorflow/tree/master/tensorf...), and some layers from Keras. I don't know as much about sonnet, but it looks like a similar story.
sbbq 10y ago
Not sure. I think Keras is more meant for education and off-the-shelf models, not so much for research due to its abstraction. Or in other words, its less customizable. I think the probably best trade-off between low-level code and coding productivity would be tf.layers, which is also in tf.core now.
- nl 10y ago
  Keras is heavily used in research.

mrdrozdov 10y ago

There's definitely some examples of useful ops here: # 1. Define our computation as some op def useful_op(input_a, input_b, use_clipping=True, remove_nans=False, solve_agi='maybe'): # ...

singhrac 10y ago

> Making Sonnet public allows other models created within DeepMind to be easily shared with the community I'm looking forward to this - DeepMind has a frustrating track record of not open sourcing their code. While it's their prerogative and I know of many valid reasons for doing so, the many other researchers who publish their (sometimes terribly hacked together) models have been incredibly helpful in verifying that their work, well, works.

qeternity 10y ago

As a startup that's in the midst of hiring, I can't wait to see how quickly CVs get updated to include this.

ravenstine 10y ago
Do you often find a significant number of CVs listing technologies that are too new for one to have gained any mastery over?
- qeternity 10y ago
  No, but we've only been interviewing for a few weeks. However we see an unbelievable number of CVs that have a full paragraph dedicated to buzzwords and python libraries.

gcr 10y ago

This isn't "yet another completely different neural network library." This library just has some new layer types for TensorFlow. Looks like there are some new layers for special kinds of attention RNNs, word embeddings, alternate implementations of spatial transformers, and so on. They also have another Batch Norm implementation that of course requires tons of fiddling to work properly, a classic tf staple :-) As a machine learning environment, tf is so complex that different research groups have to define their own best practices for it. Using tf is like learning C++ where everyone learns a slightly different, mutually incompatible, but broadly overlapping subset of the language. We're just seeing a glimpse into DeepMind's specialized tooling along with reference implementations of the operations they use in their work. This will be really useful for researchers who want to mess with deepmind's ideas/papers, but I'm a bit relieved that there isn't anything claimed to be fundamentally paradigm-shifting about this release.

Cacti 10y ago
It's nice to see people building on top of TF rather than continuing to develop completely new frameworks. I've been working with TF in detail for several months now and it is a very nicely designed framework. It is quite low-level, as you pointed out, (and so can be a challenge to get used to, and why you see helper wrappers/libs like this and keras popping up) but ML models and methods are becoming increasingly complex and TF provides exactly the foundation and flexibility needed to implement this stuff while still being able to slap together some basic layers if you need to. It also helps that the engineering is very solid (e.g., distributed models and datasets) and most of the performance kinks by now have been worked out. For those maybe putting off working with TF because of its steepish learning curve, I'd strongly suggest you dive in. I learned last year in the usual method (docs + google), but the new OReilly books are the first good ones I've seen if that's your style. The analogy with C++ is a good one, and historically appropriate given how scarce GPU resources are now.
- eanzenberg 10y ago
  If you're already using TF, have you looked into keras? Does it not have the complexity or control you are looking for?

likelynew 10y ago

Anyone knowledgable with sonnet here mind detailing how is it different from keras.

hla19 10y ago

So many implementations of the same thing reminds me the JavaScript/Widget library rush several years ago, the majority of them will be forgotten very soon, long live the projects from the giants

waleedka 10y ago

Tip for the project lead: A quick way to lose half of the early adopters is to release a library without Python 3 support. Luckily, fixing this is easy and shouldn't take more than a day or two. And the ROI is high if your goal is wider adoption.

rch 10y ago
It's also an easy way for an outsider to contribute.

ReeSilva 10y ago

It looks nice and I will certainly try it. But, no support to Py3? Common, guys, it's 2k17