11 ms·
Tensorflow 2.0
- minimaxir 7y agoThe most important change in terms of usability, IMO, is the use of tf.keras as the recommended interface to TensorFlow. There hasn't been a case yet where I've needed to dip outside of Keras into raw TensorFlow, but the option is there and is easy to do. That said, TF 2.0 changes a lot. Many repos might break, so expect to see lots of tensorflow==1.14 in requirement.txt files from now on.
- 317070 7y agoAs someone who uses tensorflow a lot, I predict an enormous clusterfuck of a transition. Tensorflow has turned into a multiheaded monster, supporting many things and approaches but none of them very well. I mean, when relying on third party code, things like `tf.enable_control_flow_v2() and tf.disable_control_flow_v2()` can and will go horribly wrong. It looks like some operations are changing behaviour depending on a global flag being set. And not just some operations, but control flow operations! That will lead to some very hard to figure out bugs. In my opinion there are some architectural problems with TF, which have not been adressed in this update. There is still global state in TF2. There is still a difference in behaviour between eager and non-eager mode. There is still the control flow as a second class citizen. If you need to transition from TF1 to TF2, consider doing the TF1 to pytorch transition instead.
- akhilcacharya 7y agoI'm still not sure what the difference between tf.keras and the other keras repo is! And that's before Sonnet, the Estimator API (?) and TFLearn (I'm probably forgetting a bunch).
- minimaxir 7y agoThe intent of having separate repos for tf.keras and keras was to support third-party platforms in the latter. But since the supported third-party platforms (CNTK and Theano) have stopped development due to TensorFlow, well...
- ghop02 7y ago> This is also the last major release of multi-backend Keras. Going forward, we recommend that users consider switching their Keras code to tf.keras in TensorFlow 2.0 https://github.com/keras-team/keras/releases/tag/2.3.0 https://github.com/keras-team/keras/releases/tag/2.3.0
- jlebar 7y ago> If you need to transition from TF1 to TF2, consider doing the TF1 to pytorch transition instead. Or consider https://github.com/google/jax https://github.com/google/jax !
- applecrazy 7y agoJust looked at Jax. At first glance, it seems to be a GPU/TPU based NumPy? The thing is, TF has more than tensor ops. It has pre-defined NN layers, data loading/serialization, distributed training, metrics, and model serving. It seems like a bit of a step backwards, that's all. Edit: "matrix ops" -> "tensor ops"
- jlebar 7y ago> At first glance, it seems to be a GPU/TPU based NumPy? Yes, with a compiler to make this fast. > The thing is, TF has more than tensor ops. It has pre-defined NN layers, data loading/serialization, distributed training, metrics, and model serving. Yes, it is a simpler and smaller API. For things like data loading, you can use the tool of your choice -- TF, pytorch, whatever. For pre-defined NN layers, there are libraries that build this as a very thin wrapper around JAX's low-level API, see e.g. lax, which is include in JAX.
- nmca 7y agoHave you ever used any of those features of tensorflow though? They're all, er, idiosyncratic. If you're a decent software engineer and are following the mafs well from a book, I couldn't reccomend Jax highly enough. (I work on big tf RL projects every day).
- option 7y ago»If you need to transition from TF1 to TF2, consider doing the TF1 to pytorch transition instead.» - that’s exactly what we did and we don’t regret that decision
- applecrazy 7y agoI'm actually looking to learn PyTorch. Where's the best place to start?
- travisporter 7y agoPytorch website has a “blitz” tutorial that was fantastic.
- rm_-rf_slash 7y agoIf you have any existing ML/DL experience, picking up PyTorch is a breeze. You could get a pretty solid understanding of the framework with an MNIST handwritten digit recognition model over the course of an afternoon, so don’t sweat looking for the “right” tutorial.
- applecrazy 7y agoAwesome, I've done some ML before. I'll just dive into the docs.
- mitchellgoffpc 7y agoHighly recommend it! I love pytorch so much, it's basically numpy with automatic backprop and CUDA support. It evaluates eagerly by default, which makes debugging a lot easier since you can just print your tensors, and IMO it's much simpler to jump between high-level and low-level details in pytorch than in tensorflow+keras. Just as one example, activation functions in pytorch are applied by calling a python function on your layer, instead of passing a string argument with the function name to the layer constructor, so you write layer = F.relu(Linear()(input)) instead of layer = Dense(activation_fn='relu')(input) As a result, it's a lot more straightforward to try out custom activation functions in PyTorch.
- mitchellgoffpc 7y agoAgreed. Tensorflow kinda reminds me of OpenGL in that its dependence on global flags causes some really annoying bugs, especially when you're using third-party libraries or pre-trained models. `enable_eager_execution`, `enable_tensor_equality`, and `enable_v2_tensorshape` have all completely broken my code at one point or another.
- StreamBright 7y agoSorry maybe a stupid question but would not be Mxnet better for what you do in TF?
- alfalfasprout 7y agoAgreed, MxNet is faster and considerably easier to use than TF.
- solidasparagus 7y agoThere's absolutely no evidence that MXNet is faster than TF. At the high end, all three (TF/PyTorch/MXNet) are similarly performant. The reality is that implementation matters more than framework when you are talking about performance.
- StreamBright 7y agoMy question was more about usability or interchangeability. Could TF get replaced by Mxnet in a typical deep learning project?
- solidasparagus 7y agoIn terms of features and functionality, they are very much interchangeable. A new project could be written in any of the three major frameworks and be equally good. The only standout feature I'm aware of is that TF has the best support for doing inference on devices, but that won't be true forever. In terms of actually migrating a codebase from one to the other, the APIs are different enough in small ways that it would be a large amount of effort.
- XCSme 7y agoNot only upgrading is hard, but also installation (on Windows at least). For each Tensorflow version you need a specific python version, a specific CUDA version, specific tensorflow-gpu version, and many other easy to get wrong things. The problem is not the requirements, but that it's very hard to know what versions are compatible. There are endless threads on Github of people trying to use Tensorflow but failing after spending days trying to install it.
- nicklarsennz 7y agoI've been using containers for this, and mounting the source code.
- jacquesm 7y agoI've been using Anaconda and it finally made Python work without endless rounds of 'incompatible libary bingo'. It also works well for machine learning projects.
- tudelo 7y ago+1 for Anaconda, worth trying for anyone who has a problem with package versions.
- echelon 7y agoAs a software engineer and non-data scientist, I hate Anaconda because it feels like it's a tool that tries to be the be-all, end-all package management tool for everyone in the data science field, yet it feels like a sloppily built, bloated whale. It's even managed to overwrite PATH on some of my Linux machines, which is where I drew the line. I vastly prefer creating hermetic environments with either venv or Docker. They're much cleaner and easier to work with. I wish data scientists would adopt these tools instead. Sadly, many of the ML models I investigate on Github don't even have their package requirements frozen. It's an uphill battle...
- 7y ago
- rch 7y agoIs the friction a result of trying to support eager execution?
- vl 7y agoThe problem is that tensorflow is an umbrella name for bunch of related technologies: it's a matrix calculation engine, graph definition language, distributed graph calculation engine, ML algorithmic libraries, ML training libraries. On top of that it's extremely poorly documented. At the end of the day when you use it anything beyond most trivial stuff turns out to be incompatible with each other (this operation is not implemented for TPUs or GPUs, this API doesn't work with this API) and most of development cut-n-paste trial and error. Then you go to read it at source, but creative Python renaming and importing leads you to multi-hour wild goose chase. If you switch to PyTorch, what are you going to use for prod deployment? Is there any way to use TPUs?
- smhx 7y ago> If you switch to PyTorch, what are you going to use for prod deployment? Is there any way to use TPUs? PyTorch has an optional XLA device, that let's you use TPUs: https://github.com/pytorch/xla https://github.com/pytorch/xla
- tastyminerals 7y agoCannot but agree. Above that 1.14 documentation was simply deleted from the tensorflow website and now we are scratching our heads at what to do when it comes to model maintenance. We serve our models via TF Java API since our system is Java/Scala based. We can't even update existing TF Java API because it is incompatible with anything prior to 1.15. It's an utter mess.
- m0zg 7y agoStill pretty terrible compared to PyTorch _unless_ you need deployment to device, in which case it's basically the only game in town. Or at least the only _viable_ game. Case in point: people are still trying to figure out on their github how to apply global weight decay when training a model, and to get a "correct" resize for segmentation you have to fall back to the legacy 1.x API and specify align_corners=true there. These bugs existed for many years, and nobody gives a damn. That said if your choice is between 1.x and 2.0, 2.0 is much easier to work with, especially if you use something other than TF (e.g. PyTorch) for data pipeline and augmentation. You can hook that up pretty seamlessly if you train in eager mode.
- akhilcacharya 7y agoHas ONNX not caught fire on mobile yet?
- sairahul82 7y agoIt has but not as good as tensorflow support. This is one thing i miss truly in pytorch. Otherwise pytorch is wonderful.
- applecrazy 7y agoDoesn't CoreML abstract 90% of the actual model format/training source away? Last time I played with it in Xcode, it was painless to pull a pretrained TF model and use as-is.
- Q6T46nT668w6i3m 7y agoThe number of operations supported by CoreML or Onyx is limited.
- akhilcacharya 7y agoWhat's an example of a missing operator?
- abakus 7y agoRecently I found that a lot of TF2.0 Keras' functionaly does not support eager execution. This makes Pytorch still significantly easier to prototype with than TF2.0. If you miss Keras' way of defining NNs, you can use PyWarm: https://github.com/blue-season/pywarm https://github.com/blue-season/pywarm which offers a fully functional NN building API for pytorch.
- choppaface 7y agoDoes pytorch have a Tensorboard equivalent? For rapid prototyping, I find Tensorboard a lot more useful than, say, print statements and such that you can get through eager execution. Tensorboard is also crucial for post-hoc analysis, and the Summary format is clean enough to use as a primary data artifact (e.g. use loss recorded in summaries versus some alternative hand-crafted text file).
- smhx 7y agoIt has TensorBoard integration itself (which can be installed independently). https://pytorch.org/docs/stable/tensorboard.html https://pytorch.org/docs/stable/tensorboard.html
- abakus 7y agoThere is also tensorboardX: https://github.com/lanpa/tensorboardX https://github.com/lanpa/tensorboardX
- wdroz 7y ago"Visdom" [0] isn't well know, but it's powerful and easy to use. You can even centralize remote multiple experiments in the same dashboard. Very useful for following in real-time what is happening to your networks. [0] - https://github.com/facebookresearch/visdom https://github.com/facebookresearch/visdom
- axegon_ 7y agoDisclosure: I'm a big BIG tensorflow fan. I've been using the rc's for a while now and I must admit, it's a big step up for projects you are starting from scratch. Migrating... Probably not as clean as I would like to admit but it does the job. Overall tf 2.0 removes a lot of the boilerplate code, which is awesome.
- mark_l_watson 7y agoI think that I take the effort to update most of my side projects to TF 2, or mark the github repos as no longer supported for really old code. Some may complain about big API changes but I think it is occasionally healthy to tag old stable versions and do massive code refactoring. I assume that TensorFlow.js is also getting updated - I find it almost equally nice to prototype with (the bundled examples are first rate).
- Havoc 7y agoSo which Cuda version and cudnn is this compatible with? Same as beta?
- SloopJon 7y agoThis documentation seems to describe 2.0: https://www.tensorflow.org/install/gpu https://www.tensorflow.org/install/gpu It says CUDA 10.0 and cuDNN >= 7.4.1.
- Havoc 7y agoYeah that's same as beta. Was incredibly fussy when I tried it & tends to silently fail over to CPU if it doesn't like something (and worse builds of versions. Eg 7.4.1 comes in 10.0 Cuda flavour and 10.1)
- amrrs 7y agoOn a different context, what're some good resources to get started with Deep learning using TF (besides the stuff Google put on YT)
- woadwarrior01 7y agoThe 2nd edition of Aurélien Géron's book[1] was written for Tensorflow 2. [1]: https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/ https://www.oreilly.com/library/view/hands-on-machine-learni...
- sireat 7y agoThe first edition of the book is fantastic! Aurelien's writing is clear and clean compared to most other books on ML.
- fareesh 7y agoIs there something which has opinionated defaults where you can hack on projects without getting into all the boilerplate? Like there is a mostly finite set of typical solved things you would want to use ML for like image classification, object detection, etc. I find myself spending my time copying an example verbatim and replacing the csv / images with my own.
- Uehreka 7y agoI think this is what TF's Estimators are about. The idea is that they're going to reduce boilerplate on the assumption that most users of Tensorflow actually only use the same handful of models (ResNet, VGG, Inception, etc). I've been learning Tensorflow recently for a side project, and the style transfer work I'm doing means I need to build my own Tensorflow graphs, so I haven't had much use for this kind of thing. But it sounds like it was made for you, not me.
- jcmontx 7y agoKeras maybe?
- mcintyre1994 7y agofast.ai might do the job - https://docs.fast.ai https://docs.fast.ai
- solidasparagus 7y agoAre you looking for advanced models that you can train on your data pretty much out-of-the-box or simple, easy-to-read models that help you learn the underlying concepts? If it's the first case, what you want to do is find the best GitHub repos for the task(s) you are trying to do. Make sure the GitHub repo has a model zoo and good support and start from there. In CV, if you are trying to do high-end work, the repos to check out are: - https://github.com/facebookresearch/maskrcnn-benchmark https://github.com/facebookresearch/maskrcnn-benchmark (don't be misled by the name, it has support for lots of the high-end modern CV models) - https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection - https://github.com/TuSimple/simpledet https://github.com/TuSimple/simpledet (haven't explored this one as much, but it looks very solid) If you want easy-to-read code for non-trivial tasks, I would suggest taking a look at Gluon (GluonCV - https://gluon-cv.mxnet.io/ https://gluon-cv.mxnet.io/ and GluonNLP - https://gluon-nlp.mxnet.io/ https://gluon-nlp.mxnet.io/). I haven't worked much with the fast.ai library, but that's probably also a good suggestion.
- manojlds 7y agoBlog post - https://medium.com/tensorflow/tensorflow-2-0-is-now-available-57d706c2a9ab https://medium.com/tensorflow/tensorflow-2-0-is-now-availabl...
- spicyramen 7y agoReality is that most production ML workflows still use 1.x,( not even the latest 1.14[5]. Migration to 2.x does not make sense in many existing use cases unless you want to build your new algorithms from scratch. From previous experience with TF, API changes so much that is impossible to keep up from an Enterprise perspective, I expect TF team do some type of LTS version for Enterprise and continue improving 2.x, which clearly was a response to Pytorch and is an evolution of TF.keras and eager execution plus all the cleanup.
- adamnemecek 7y agoJust use julia.
- whlr 7y agoI like julia a lot (use it everyday, it's my primary language right now), but this isn't really a reasonable recommendation, imho. Julia seems like it could be really great for ML, but I'm not sure if the current libraries are mature enough to wholeheartedly recommend. I'd love to be proven wrong about that, though.
- adamnemecek 7y agoWhat are the mature libraries? Use PyCall and whatever your heart desires.
- abakus 7y agoMy heart desires python then
- adamnemecek 7y agoJulia consumes python and give you so much more.
- samstave 7y agoIf I were to start literally at ground zero with zero knowledge of programming or tensorflow. Where should I go?
- mantap 7y agoInstall jupyter notebook and play around with numpy/scipy. Neural networks are not really for outright beginners, and tensorflow is doubly not.
- 0-_-0 7y agoLinear regression might be the simplest thing to do with Tensorflow's autodifferentiation, if you want to jump straight in...
- LimKruscal 7y agoI'm also new to programming and I think you can start with Github:) A lot of free tutorials are available in Github like https://github.com/30-seconds/30-seconds-of-python https://github.com/30-seconds/30-seconds-of-python https://github.com/fonnesbeck/statistical-analysis-python-tutorial https://github.com/fonnesbeck/statistical-analysis-python-tu...
- fuckyah 7y agoPython, really? Maybe start using a compiled lang
- visionscaper 7y agoI decided a few weeks ago to transition to PyTorch (was using Keras before) and I must say that I really love it! How PyTorch is structured gives me the right balance between ease of use and the ability to make customisations. Further, using DistributedDataParallel, dividing the work over multiple processes, where each process uses one GPU, is very fast and GPU memory efficient. Before my switch I tried out Keras for Tensorflow, and even got a lot of support from Google in my endeavours to resolve the issues I encountered (kudos to Google for that!). In the end I felt it was still not mature enough. Further, although I do believe TF and Keras are moving in the right direction, I still felt that in some cases the way the software was set up just didn't sit well with me. Maybe it is worth to try again in a year or so, or by then I will tryout Swift for Tensorflow, which I think has a great future ahead.
- mrfusion 7y agoSo if you want to work in deep learning should you learn pytorch and tensorflow? Or just one, which one?
- p1esk 7y agofast.ai
- Fr0styMatt88 7y agoMight be slightly off-topic, what’s the latest state-of-play for AMD/OpenCL? I hear in various places that AMD is fantastic for compute, but everyone seems to be using CUDA. I’ve got a very expensive Bitcoin mining rig paperweight at the moment with two Vega 64s (along with another Vega 64 in my main rig) — it’d be great to re-purpose them for something potentially useful.
- visionscaper 7y agoI don't know too much about it myself, but you could checkout https://rocm.github.io/dl.html https://rocm.github.io/dl.html Edit : directly linked to the ROCm deep learning page.
- pixelhorse 7y agoAMD is great when it comes to raw compute power for the money. However, AMD has a history of shipping poor OpenCL drivers, so everyone just went with what actually worked - NVIDIA.
- hn2017 7y agoAny online courses using TF 2.0, such as Coursera? Or books coming out soon? That's key for proper adoption.
- hn2017 7y agoAs far as production deployments of ML models, what percentage use TF vs PyTorch?