56 ms·
Swift: Google’s Bet on Differentiable Programming
- teekert 7y ago"To put it bluntly, Python is slow. Also, Python is not great for parallelism." Really? Doesn't that depend on the modules, the underlying code? The problem?
- echelon 7y agoThe problem is the GIL prevents threads from making progress simultaneously, even with synchronization and locking primitives available. But even so, Python is a little slow when you start thinking about threading. You'd be better off using Rust or Go or something rather than the half-baked support found in scripting languages.
- andybak 7y ago> the half-baked support found in scripting languages. It's a long time since I've heard that phrase used disparagingly. Didn't we all decide to use the term "dynamic languages" just to avoid the judgemental overtones associated with "scripting".
- echelon 7y ago> It's a long time since I've heard that phrase used disparagingly. Didn't we all decide to use the term "dynamic languages" just to avoid the judgemental overtones associated with "scripting". I didn't mean to disparage. While I did intend to scrutinize Python's multiprocessing (my experience with it was less than fun), my use of the term "scripting language" was entirely subconscious. But I do have an anecdote. A decade ago I was using Python and Flask for building web apps. Now I use Rust instead, and many of my collages are choosing to use Go. I still use Python for scripting and now also use it for ML. But I wouldn't use it for the web anymore. I think the landscapes and use cases are shifting since there are new tools available. Python is doing new things (pytorch, tensorflow), but less of the things I used to use it for (Flask, Django, ...)
- 72deluxe 7y agoI am not sure. If it runs through an interpreter, isn't it a script??? I think the "judgemental overtones" are in your head, not in the speaker of the word "script".
- komuher 7y agoLooking forward to S4TF 1.0 lets see how much progress they'll manage to make in next few months without Chris Lattner
- threw4234324 7y agoDoes anyone here use Swift for Deep Learning work ? I'm yet to see any real advantage, and even if there were or were to be added, I think Julia would almost always be a better choice here.
- gaogao 7y ago“Why didn’t they choose Julia"
- gaogao 7y agoI'm quoting the article on how the first HN comment will be this.
- threw4234324 7y agoWell, why though ? Julia has plenty of numerical libraries going for it, and all of them can be used in a differentiable manner (see, Neural ODE etc.). While Julia was built by a mix of people doing Numerical Linear Algebra and PLs (MIT, UCSD..), Swift seems to be entirely the output of the latter group.
- erwinh 7y agoHeard about this a while back but then it went off the radar (from my pov at least). What is the latest from Apple and Google on this?
- gaogao 7y agoTaichi, http://taichi.graphics/ http://taichi.graphics/, is another interesting stab in this field.
- hamanin 7y agowow this one looks more promising.
- gaogao 7y agoI'm excited to test it out deeper for some game dev using the differentialblity.
- rcxdude 7y agoIt's nice that they've actually demonstrated something in it. The differential physics engine is pretty freaking cool. Also, if you look at some of the details they talk about it's clear that just having a differentiable language isn't enough: it's still very easy to have gradients which don't work properly if you're not careful, especially in boundary conditions (the example they give is collision detection).
- xenonite 7y agoThe SIGGRAPH paper on Taichi: http://taichi.graphics/wp-content/uploads/2019/09/taichi_lang.pdf http://taichi.graphics/wp-content/uploads/2019/09/taichi_lan... Supplementals: http://taichi.graphics/wp-content/uploads/2019/10/taichi_lang_supp.pdf http://taichi.graphics/wp-content/uploads/2019/10/taichi_lan...
- SeekingMeaning 7y agoTwo Minute Papers made a fascinating video on Taichi: https://youtube.com/watch?v=T7w7QuYa4SQ https://youtube.com/watch?v=T7w7QuYa4SQ (“Finally, Differentiable Physics is Here!”)
- mjlee 7y agoAccidental Tech Podcast Episode 371 is essentially two and a half hours of Chris Lattner: https://atp.fm/episodes/371 https://atp.fm/episodes/371 It's a bit of an outlier, most ATP episodes are Apple focussed news with adjecent tech interests. Episode 371 is not that.
- cerberusss 7y agoThe height for me was when Chris talked about the Python integration, somewhere after 0:24:30. Marco asks jokingly: "can you also do PHP integration?" Chris is silent for two seconds, then continues as though nobody said anything :-) To be fair, he does answer the question a minute later.
- yreg 7y agoDid ATP ever have a guest on the show before?
- ChrisMarshallNY 7y agoI’ve been writing Swift since the day it was announced. I believe in the language as a system programming language (and I’ve done systems programming in the past). That said, like C#, it is strongly associated with Apple as a “private” language (despite being open-source). That’s probably a big reason it’s slow to be adopted in a wider fashion, but I’m sure there’s other issues. It would be nice to see it be more widely adopted, but I’m OK with it being confined to the Apple ecosystem, as that is my domain.
- oaiey 7y agoThere is more in that rationale than being privately owned. In comparison to C#, Swift is not natively adopted by any of the three big cloud providers (AWS, Azure and GCS) despite being the go-to language for half of all mobile apps (which is a huge market). C# is supported despite being Microsoft controlled. I believe that Swift does not have the confidence of the community due to Apple's radical technology choices. IBM may have counter-balanced this. What do other think is the reason for the slow backend adoption of Swift?
- Bigpet 7y ago> What do other think is the reason for the slow backend adoption of Swift? It's not different enough. All the demos I've seen of it, felt like it was mostly an "also ran" language that modernised it to be a significant improvement over Obj-C. But I've never seen any argument for it being a huge enough improvement over C#, Kotlin or others to be worth it to switch ecosystems.
- andybak 7y agoI feel differently. I read the docs when it was first released and felt it was potentially hitting the sweet spot for "static yet expressive" and "familiar yet modern". I was mainly Python at the time and the readability and clarity impressed me and those are areas where I have very high standards. More radical languages all tend to look a bit "read only" to me but I am aware that might just be unfamiliarity on my part. C# and Kotlin both have nice features but they are more conservative Java-likes than Swift. Swift had enough modern features and syntax to feel like it would be reward the effort learning. But I don't own an iOS device so I'm still waiting for a use case!
- pjmlp 7y agoUntil Google actually offers Swift tooling comparable to Julia, Pytorch and ML.NET on Windows or Android, I will keep ignoring it. Note that the recent Tensorflow Summit had zero annoucements related to S4TF, including post event blog posts. From the outside it looks like there is a small group trying to push S4TF, without regards for usability outside Google Cloud, while everyone else is doing JavaScript, Python and C++ as usual.
- lalord 7y agoYeah I agree that the tooling at the moment is inadequate, specially on linux, but both Apple and Google are working towards improving it. Regarding the tensorflow dev summit, it was supposed to be 2 days long initially, with the S4TF talk taking place on the second day. One week before the summit, the whole day 2 got scraped due to covid19 though. So the intention was there at least.
- pjmlp 7y agoSo not enough stuff to at very least write a blog post?!
- gonzus 7y agoIn all honesty, this sounds to me like a whole lot of BS and hype. The name is pretentious, the quotes are ridiculous ("Deep Learning est mort. Vive Differentiable Programming."; "there will be a need for the creation of a whole new set of new tools, such as a new Git, new IDEs, and of course new programming languages"). Maybe I am just ignorant and fail to grasp the importance of such capabilities in a machine learning context, but I have to say, the grandstanding is a bit grating. Who knows, perhaps this will become the greatest thing since Lisp... What do I know.
- pjc50 7y ago> need for the creation of a whole new set of new tools, such as a new Git, new IDEs, and of course new programming languages The greatest possible barrier to adoption.
- baybal2 7y agoDeep learning is the new Object Oriented Programming and Service Oriented Architecture
- DeathArrow 7y agoWhat about SPA and serverless?
- arethuza 7y agoWhy would a language where it is possible to manipulate certain functions to get their derivatives require a new revision control system or IDE? And speaking of Lisp - wasn't symbolic differentiation a fairly common thing in Lisp? (basically as a neat example of what you can do once your code is easy to manipulate as data).
- joaogui1 7y agoFor the IDE part they have written some cool debuggers that help you understand the differentiation part and catch bugs on it. But I'm not sure why you couldn't just use the debugger, instead of a whole new IDE, much less why you would need a new RCS
- m12k 7y ago"So, what is differentiable programming? In a nutshell, differentiable programming is a programming paradigm in which your program itself can be differentiated. This allows you to set a certain objective you want to optimize, have your program automatically calculate the gradient of itself with regards to this objective, and then fine-tune itself in the direction of this gradient. This is exactly what you do when you train a neural network." Isn't this just declarative programming as we know it from e.g. Prolog, SQL or other places where the programmer declares what their objective is, and it's left up to the interpreter, compiler or scheduler to figure out the best way to achieve that? And now that's being applied to ML (which probably makes sense, since it involves a lot of manual tweaking). Sounds like a great use case for a library, but hardly worthy of being called a new programming paradigm.
- hope-striker 7y ago• Sure, it could come under the umbrella of "declarative programming", but that's an enormous umbrella, so that doesn't really say much. • I fail to see how differentiable programming (the idea of expressing the desired computation in terms of differentiable objective functions) is any less of a "paradigm" than logic programming (the idea of expressing the desired computation in terms of logical predicates). • Depending on the expressiveness of your programming language, every paradigm can seem like it's "a great use case of a library, but hardly worthy of being called a new programming paradigm."
- itronitron 7y agohole.fit(peg)
- deleted 7y ago[deleted]
- autonoshitbox 7y agoIt's the result of programmers too high on undergraduate math to see through their own smoke clouds.
- deleted 7y ago[deleted]
- fluffything 7y agoCan't you do this in Rust by writing a #[derivative] proc macro without having to modify the language? #[derivative] fn cube(x: f32) -> f32 { x * x * x } // expands to // impl Derivative for cube { // type Args = (f32,); // type Return = f32; // fn derivative(x: Self::Args) -> Self::Return { // 3 * x * x // } // } let cubeGrad = gradient(cube); assert_eq!(cubeGrad(2), 12); where `gradient` is just a normal Rust function: fn gradient<F: Derivative>(_: F) -> impl Fn(F::Args) -> F::Return { |x: F::Args| Derivative::derivative(x) } My original attempt was accidentally more powerful, in that it allowed `cube` to be a stateful function like a closure. To implement this, the only thing one needs to do in the `#[derivative]` macro is: * parse the function into an AST * fold the AST using symbolic differentiation * quasi quote the new AST Swift can probably differentiate functions in other translation units because it keeps the bytecode of all functions around. A proc-macro based approach wouldn't be able to achieve that, at least, not in general, but a Rust plugin could, since plugins can drive compilation to, e.g., compile whatever function is being derived, and differentiate its bytecode instead of Rust, and if the function happens to be in a different translation unit, access their byte code.
- stefan_ 7y agoMaybe we can also make a #[returns] macro. This macro parses the function into an AST, invokes magic, and determines if the function will return on any input. I feel there is an impedance mismatch here.
- fluffything 7y agoWhat magic does need to be invoked by the #[derivative] proc macro ? For parsing Rust into an AST you just use syn, which also supports doing AST folds, and for semi-quoting you just use the quote crate. For the symbolic differentiation part, you can use any of the hundred symbolic differentiation libraries out there to obtain '3 * x * x' from 'x * x * x'. Pretty much any machine learning framework out there supports AD and symbolic differentiation in some form, by using some libraries already. Swift people are standardizing standard practice, not discovering the americas. If any of that looks like magic to you, then you might just be lacking the background.
- nmca 7y agoDoes Chris Lattner still work at Google?
- komuher 7y agoNope
- pjmlp 7y agoNo, and Richard Wei has left as well.
- geodel 7y agoAnd he is working on Swift at Apple now which does not look bad to me.
- pjmlp 7y agoDepends to which extent working on Swift at Apple actually means Swift on Android, Linux and Windows, at an ecosystem and tooling level comparable to other programming languages.
- favorited 7y agoAnd then Google hired Dave Abrahams from Apple to work on S4TF.
- helsinkiandrew 7y ago> A data scientist could use Swift in much the same way as they use Python Is this really true? Many data scientists don't come from a development background. Low level swift code (what's likely to be in TF) can be as obscure as C++. Will the 'industry lag' without users having to dig into the library and understand the code?
- MiroF 7y agoNot all data scientists are the same. Sure, this probably isn't useful for someone without much coding experience who ya experience with the Kerala API, but for people who want to code custom architectures to solve their problem (which is also very common), the static type checking enabled by Swift, among other things, is very appealing.
- dep_b 7y ago> Low level swift code (what's likely to be in TF) can be as obscure as C++. True, but I didn't encounter it in the wild yet. Usually Swift used in "Python mode" is fast enough.
- garyclarke27 7y agoNice article, learned a lot about Swift, love the named parameters and argument labels, makes calling functions with many parameters much more reliable and with labels reads like English, which is great for immediate comprehension. Lack of Named parameters in Javascript, is for me, a critical glaring missing capability, that should be fixed urgently.
- enlyth 7y agoJust use an object with keys as the parameter names, this pattern is actually very common so I don't see the issue. doStuff({ foo: 3, bar: 7 })
- tomp 7y agoIndeed. What that pattern also solves, and AFAIK most static languages lack, is argument forwarding (though Python solves this even more elegantly than JS): def f(x, **kwargs): return g(x, foo=1, **kwargs)
- zeroxfe 7y agoIsn't the JS version just: function f(x, ...args) { return g(x, ...args); } Or did you mean forwarding object fields? function f(x, params) { return g(x, {foo: 1, ...params}); }
- tomp 7y agoAFAIK the first only works for positional (non-keyword) arguments. The second works, I guess? Not an expert in JS, especially not the "new parts", splats are definitely an improvement!
- garyclarke27 7y agoJS parameters are named to some extent, as in you can refer to them by name within the function body - BUT (a critical shortcoming in my view) you can’t assign values by name when calling functions, you can only assign values by position.
- elevenoh 7y agoSwiftUI for iOS got me in to swift. I find optional?s & protocol-oriented programming enable clear & concise mental models. I like swift's syntax better than Java/Kotlin, C++ & Rust imo. And it's damn fast.
- bla3 7y agoSpeculation time! Chris Lattner designed Swift while he was at Apple. Apple seems committed to it, and Chris seems to spend his time making it a language that isn't Apple-only. His stint at Tesla was maybe short because Tesla didn't want to adopt Swift. At Google, he got to start a project with it, but him leaving suggests it didn't catch on. SiFive is small and hardware ramps are slow, so maybe he's given up on the Swift-world-domination plan. Again, this is all speculation.
- weiming 7y agoMy gripe with Swift (and I've written a lot of it) for purposes requiring a lot of number crunching is the performance issues associated with default use of copy-on-write types/ease of accidentally writing O(N^2) code. In Objective C the behavior was more obvious with NSMutable* types being front and center.
- valuearb 7y agoPremature optimizations.
- lovetocode 7y agoHonest question. Why are languages like C# and Java not more heavily considered?
- ryuukk_ 7y agobecause they are: - slow (cold startup) - heavy (require heavy runtime) - their syntax is way to verbose
- BiteCode_dev 7y agoThere are some things really neat in the switf example that they compare to python, like the literal range similar to ruby's, the fact that "+" is a function like in lisp, although I'm really not fan of the CFAbsoluteTimeGetCurrent(). But the python example doesn't make me trust the rest of the article. It is clearly a swift example, translated verbatim to python. Idiomatic Python would be this: import time for it in range(15): start = time.time() total = sum((it,) * 3000) end = time.time() print(end - start, total) Which is shorter, and way faster. Now of course, Python is slower than swift (although a numpy version would not be, but I get it's not relevant in general purpose machine learning). But misrepresenting a language is not a good way to make a point.
- lalord 7y agoHi, author here, thanks for taking the time to read the article! The objective of the demo was not to see which language could sum up a bunch of numbers the fastest. You could keep optimizing that until you are left with just `print(<the resulting number>)`. The objective was to have a simple example of looping over an array a bunch of times. The only reason I ended up summing the numbers in the array and printing them was so that LLVM wouldn't optimize it away and be unfair towards python. I actually wrote it first in Python tbh.
- BiteCode_dev 7y agoMake sense, thanks.
- favorited 7y agoThe `CFAbsoluteTimeGetCurrent()` is just a call to a system library. There are libraries which implement APIs like: let clock = Clock.system let now = clock.thisInstant() there just isn't a nice core library interface yet.
- awwaiid 7y ago"If you’ve ever used PyTorch, TensorFlow, or any of the other big machine learning libraries, they all support this feature, but only if you’re using their particular library specific operations. What’s more, working with gradients in these Python libraries is not as lightweight, transparent, or well integrated as it is in plain Swift." This reminds me of concurrency and transactional memory in Clojure. You can have all of those things at a library level, but building it into the language... Well it kinda FORCES you to deal with them, for good and ill.
- erwincoumans 7y agoNo mention about Google JAX is odd, both in the article and in the comments here. It was discussed yesterday: https://news.ycombinator.com/item?id=22812312 https://news.ycombinator.com/item?id=22812312
- antpls 7y agoAgreed, this is the most clickbait title of the week. From my understanding, the author is someone from outside Google who doesn't know anything about Google bets more than what is publicly available, and the author is biased toward the Swift language. JAX makes more sense, as it uses Python, but more importantly, Google works at a lower level of the stack with TPU, MLIR and XLA. It doesn't really matter what language is on top of that stack.
- cutler 7y agoOne thing to bear in mind with Swift is how difficult it is to stay up to date with the latest version. Swift versions are tied to Xcode versions and Apple has a history of leaving older versions of OS X in the dust. I'm on OS X Mojave, which is only one version behind Catalina, and I'm already denied access to Swift 5.2. Talk about building a community.
- ken 7y agoEven worse, macOS is tied to hardware years, so even though I've got 3 working Macintoshes here, only 1 of them can run the current version of Swift (and just barely). My quad-Xeon can't run the latest minor upgrade, even though the compiler is faster than the previous one. I'm sick of this pointless and never-ending upgrade cycle. I started a new project this week, and it's in Clojure. I find it a much more productive language, and I'm confident I can run it on nearly any computer from the past 20 years. (This is probably not my most productive comment ever, but I'm pretty frustrated right now with having chosen Swift in the past.)
- 72deluxe 7y agoYes I too am getting tired of it. I have to get a new SSD every time Apple releases an OS because I don't want to run their latest OS and break all my audio hardware, but need something to build + release on for the app stores. I have stuck with C++ but I did buy a book on Swift, which is now hopelessly out of date. People can say bad things about C++ (plenty of material!) but at least it is mostly backwards compatible.
- john_alan 7y agoLove Swift. Really elegant language.
- scottlocklin 7y agoI had to google "differentiable programming" to see if they were really touting automatic differentiation as some futuristic language feature. Honestly the only real response to this should be :trollface: I've been doing software development in data science, large scale optimization and machine learning for over 15 years... I've needed automatic differentiation in my language .... exactly never. I mean, most of the languages I use regularly are capable of it, and it is a neat trick; it's just not that useful. The best part of this article is Yann and Soumith twittering they need Lush back (and not because of automatic differentiation). I agree; it's still my all time favorite programming language, and I don't even fool around with Deep Learning. https://twitter.com/jeremyphoward/status/1097799892167122944 https://twitter.com/jeremyphoward/status/1097799892167122944
- abra1010 7y agoIf you're doing "deep learning" using frameworks like Tensorflow, PyTorch or JAX you're using autodiff all the time.
- scottlocklin 7y agoSure: great -it can be used with other sorts of models as well, but it's really not that big a deal, and can be implemented in a lazy afternoon. I'm pretty sure lack of this as a first class language feature is holding back exactly no DL frameworks or programming languages, and glueing it onto Swift isn't going to make DL people use it. I've never used Swift's repl; I suppose that might actually be something that gets people on board.
- stormtroper1721 7y ago> can be implemented in a lazy afternoon Hi, I'm working on making a tensor lib in Rust (think numpy + autodiff) to learn about these topics. There isn't much information online about how projects like numpy and autograd work under the hood. Do you have any ideas/tips/resources about how it could be done?
- 7y ago
- yahyaheee 7y agoGoogle really messed up here, they had an unprecedented opportunity to create a new language for numeric computation and rally the scientific community around that. They hired Chris Lattner and he basically steamrolled that dream by forking Swift. I don’t see people running over here to write numerical libraries like you see in Julia, that’s largely because of the crowd around Swift. The language is also a bit verbose and heavy handed for what data scientists would prefer. Latner was too close to Swift to understand this. The blame really falls on google project management.
- deleted 7y ago[deleted]
- nevi-me 7y agoIn hindsight, I wonder if using Swift would seem like a good idea in retrospect. AFAIK NVIDIA has dropped CUDA support on Mac, and outside Mac and a bit of Linux; how much support does Swift have? Even though the article talks about the "why not Julia", which is the highest comment at time of typing ... Choosing a cross-compatible language would have kept more people interested in the long-run. Why should I as a Windows user want to learn a language just to use it with Tensorflow; when I'm not sure if such language support and other tooling will generally come to Windows?
- yahyaheee 7y agoYeah its a very closed ecosystem, that is heavily Apple centric. The Apple war with NVidia is ridiculous, I guess they are just betting on TPUs or AMD to make up the difference. It is also strange that its so tied to Tensorflow rather than being just a better numeric language. Julia is definitely a better option, but I still feel we can do better than that.
- valuearb 7y agoSwift is only 6 years old and is growing rapidly on Linux. Despite IBM bowing out it's server frameworks (vapor and Kitura) continue their development. There are Swift compilers for many platforms.
- nimmer 7y agoNim would have been a better choice. It's often as fast as C, similar to a static typed Python and completely programmable.
- yahyaheee 7y agoNim is maybe the best alternative right now next to Julia, still think there may be a better numeric language yet to show up
- goatlover 7y agoSo what about Fortran if the modern languages in this space fail to satisfy? It's fast, statically typed, and often used for heavy duty number crunching on supercomputing architectures, so obviously it handles parallelism well.
- nurettin 7y ago> Also, Python is not great for parallelism. I agree that cpython threads are not parallelism, but python still comes with built-in support for multiprocessing and I've been earning my bread using that within the past two years, so unless you have to use in-process parallelism for some weird reason, python and your OS scheduler of choice has you covered there.
- logicchains 7y ago> have to use in-process parallelism for some weird reason Needing shared memory paralellism is a weird reason now? Pretty much any parallel algorithm that's not embarassingly parallel is going to perform better with threads able to share memory than with message passing between processes.
- nurettin 7y agoExactly, pretty much any parallel algorithm is embarrassingly parallel (gather or calculate a bunch of data, process it and merge it together) so I question the need for continuously needing in-process parallelism instead of solving it trivially.
- lmeyerov 7y agoPython is not great for fine-grained data parallelism (SIMD, GPU), which is increasingly the lion's share: non-starter for direct inline and pretty bad for DSLs. The result is runtime heroics for embedded ~dataframe DSLs (pyspark, rapids.ai) with high overhead. OTOH, those heroics do happen, and been OK so far. Accelerating differentiable programming is basically an extra transform layer on accelerating data parallel programming. Thankfully, our team writes zero raw OpenCL/CUDA nowadays and instead fairly dense dataframes code. Similar to async/await being added to doing a lot for web programming on Python, curious what it'll take for data parallel fragments (incl. differentiable.) If it wasn't for language resistance for UDF + overhead, and legacy libs around blocking, we'd be happy.
- mratsim 7y agoI'm not sure what's your domain but for machine learning that will be slow: - Parallel matrix multiplication may be embarassingly parallel, you have a reduction step that is not trivial to parallelize across processes. Also you need to take care of register tiling, L1 cache tiling and L2 cache tiling. It is way easier to do this in OpenMP - Parallel Monte Carlo Tree-Search: it's much easier and more efficient to spawn/collect trees with a proper spawn/sync librairie.
- sriku 7y agoif you want nice DP, try Julia with Flux and Zygote. Zygote perhaps has a bit of a distance to go for performance, but it just feels totally natural to code with these libraries .. if you assume performance gaps will be closed (its good enough).
- mratsim 7y agoCan Julia build static binaries now? Because for mobile devices you don't want to ship LLVM. Plus it's easier to package binaries than scripts which usually require Docker to deal with dependencies in a sane way.
- eigenspace 7y agoA lot of great progress is happening in PackageCompiler.jl (large binaries, very useable) and StaticCompiler.jl (small binaries, still immature). I'd argue that Julia is much closer to having a good static compilation story than Swift or Nim are to having vibrant scientific ecosystems. > Plus it's easier to package binaries than scripts which usually require Docker to deal with dependencies in a sane way. Julia does have a really good story for reproducible dependency management. We learned a lot of valuable lessons by looking at Python's dumpsterfire.
- Eridrus 7y agoI'm surprised by all the negativity in this thread. The need to write C++ components for models is pretty real when you need things to be fast and run on CPU. God forbid you ever want to deploy that on some sort of accelerator, necessitating a rewrite to CUDA or whatever language that accelerator speaks. Maybe that's too niche of a subject for people to really care about, and maybe it's going to fail now that Lattner is gone (though there still seemed to be a bunch of activity last I checked), but the problem it is trying to solve is real.
- gigatexal 7y agoPerhaps it could be this: if my value comes from running some crazy interesting high level Math derived machine learning model on data and I’m highly paid the opportunity cost to invest the time and effort to both become good at and write C++ is higher than if I can do the same thing in a few lines of python and just throw some more compute at the problem since that’s a fixed cost.
- 867-5309 7y agoparsing this single sentence would constitute passing an EFL exam in my book
- elliekelly 7y agoPerhaps the most valuable writing advice I've ever received: “When you think one sentence needs 35+ words what you really need is two sentences.”
- gigatexal 7y agoYes, yes, I tend to ramble.
- 867-5309 7y agoit wasn't meant as a dig - it seems perfectly valid and a good example of how much can actually be written in one sentence
- kelvin0 7y agoIsn't Swift an Apple 'invention'? Surprised Google is going for such a 'niche' language ... This reminds me of the initial 'Dart' hype and promises.
- yters 7y agoWill differential programming be able to solve the halting problem?
- joshmarlow 7y agoNo - a differentiable program would still be running on a real computer (which is a finite approximation of a Turing machine). So for a differentiable program to solve the halting problem, that would imply that the underlying machine could solve the halting problem (which it can't).
- yters 7y agoSo how is this more powerful than normal programming?
- JoeCamel 7y agoObviously "more powerful" doesn't imply the capability of solving the halting problem. What exactly do you want to say? You are not impressed with "differentiable programming"?
- yters 7y agoIt is misnamed. Makes it sound like some kind of Turing complete approach, like when DeepMind invented differentiable Turing machines. Tgis post just describes differentiating mathematical functions, which has been around forever, and you can find in matlab, mathematica, python libs, explained in CISCP, etc. Why doesn't Google just buy a Matlab license and be done with it? Why bother embedding in a whole new programming language?
- ddragon 7y agoTechnically, RNNs, including LSTMs, are already turing complete, neural turing machines mostly decoupled memory from the hidden layers so it can grow the memory size without a square increase on the number of parameters, which helps with the unbounded part of turing machines. it also helped inspire many attention based models that came after. Also matlab isn't really related here, and automatic differentiation is different from symbolic differentiation, the direct comparison are those python libs like tensorflow, pytorch and jax. Differentiable programming in the end is just a way of making something that you can already do better (just like you could create neural network long before theano/tensorflow/torch but it was not as streamlined). With a differentiable programming approach you can get something as dynamic as pytorch, with the performance optimizations and deploy capabilities of a tensorflow graph and with an easy way to plug any new operation and it's gradient by writing in the same host language (so no need to learn or restrict yourself the tensorflow/pytorch defined methods/DSL). You don't even need to change the compiler or define a new language for it. Julia's Zygote [1] is just a 100% Julia library you can import, for which you can add at any point any custom gradient even if the library creators never added them, and them run either on CPU or GPU (for which you can also fully extend using just pure Julia [2]). And of course, you can also use a higher level framework like Flux [3] which is also high level Julia code. I think the heart of differentiable programming is just another step in the evolution, from early (lua) torch-like libraries that gave you the high level blocks to compose, to autodiff libraries that gave you easy access to low level math operators to build the blocks to the point where you can easily create your own operators to create the high level blocks. [1] https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl [2] https://github.com/JuliaGPU/CUDAnative.jl https://github.com/JuliaGPU/CUDAnative.jl [3] https://github.com/FluxML/Flux.jl https://github.com/FluxML/Flux.jl
- dzonga 7y agothe growth of an high performing language for data will probably come from NIM. it has expressiveness like Python and speeds like C. Though the community needs to develop. and drop efforts like deploy to JS etc
- fastball 7y agoSomething of an aside: closures don't need to be "unnamed functions which capture their context", right? A function that "closes over" its context but is named is still a closure, unless I'm wrong.
- wool_gather 7y agoYou're correct, but there's a bit of "linguistic drift" at play, I guess. The use of the term "closure" in the Swift ecosystem has come to refer almost entirely to the anonymous kind. (Side note, named functions in Swift are indeed also closures: let x = 10 func f() { print(x) } f() // Okay; prints 10 as you'd expect )
- fastball 7y agoYeah, that seems to be the case in the JS community as well – I see a lot of people referring to the arrow function syntax as "closures", even though named functions are closures in JS too.
- gtycomb 7y agoOP likes Swift and he compared it to other languages Python, Go, C++ & Rust, Julia. In the enterprise sphere that I am in, I haven't seen Swift and its hard to move a team in this direction at this point. Python already drives a significant market and captured talent in numerical differentiable programming, particularly because of the ease with which prototyping and tuning is done with it. Obviously when we want to scale we have the conversation on some other option.Personally its a bit frustrating that Go support for TensorFlow or one of the competitors is not quite satisfactory and this is surprising, I can't explain it. Instead of inventing any new language why don't one of you -- Python, Go, C++ & Rust, Julia, or Swift -- complete the job with end-to-end differentiable programming. 'Complete' to me means a language level seamless GPU (or related distributed/parallel architecture) and language level deployment ease (the kind of thing done with Kubernetes) and integration with embedded hardware.
- eugenekolo 7y agoGoogle loves entering legal battles over programming languages huh.... Just use something actually free, or create your own.
- deleted 7y ago[deleted]
- mkchoi212 7y agoWhile I’m a fan of Swift and am glad that it’s getting a lot more attention outside of the iOS world, I’m not sure if it’s the best language to be used in the data/ML world. Yes, it’s faster than Python, yes it’s almost as easy to read, and yes its standard library is powerful. But when interacting with Swift as a framework, I can’t imagine how much of a pain in the ass the strict type system will be for researchers. I get it. Types help people developers write less error prone code and blah blah. But the main target here are ML researchers. They aren’t developers. They just want to write down their theory in code and just run to train their models. They aren’t writing complex iOS apps here.
- nestorD 7y agoTo me this seem attractive for those that want to put ML into production. Having a solid type system and an easily interoperable language could lead to a lot less bad surprises than using python. For research this seem indeed like a bad fit.
- sparrc 7y agoIMO the first three lines of the program basically explain why academics and data programmers are never going to use Swift: Python: import time for it in range(15): start = time.time() Swift: import Foundation for it in 0..<15 { let start = CFAbsoluteTimeGetCurrent() This is why people like Python: - import time: clearly we are importing a 'time' library and then we clearly see where we use it two lines later - range(15): clearly this is referring to a range of numbers up to 15 - start = time.time(): doesnt need any explanation This is why academics and non-software engineers will never use Swift: - import Foundation: huh? Foundation? - for it in 0..<15 {: okay, not bad, I'm guessing '..<' creates a range of numbers? - let start = CFAbsoluteTimeGetCurrent(): okay i guess we need to prepend variables with 'let'? TimeGetCurrent makes sense but wtf is CFAbsolute? Also where does this function even come from? (probably Foundation? but how to know that without a specially-configured IDE?) EDIT: Yes everyone, I understand the difference between exclusive and inclusive ranges. The point is that some people (maybe most data programmers?) don't care. The index variable you assign it to will index into an array of length 15 the way you would expect. Also in this example the actual value of 'it' doesn't even matter, the only purpose of range(15) is to do something 15 times.
- pjc50 7y agoMaybe, but by that metric nobody would ever be using Java.
- metroholografix 7y agoThe improved Swift code shown in the original post takes 0.5s to compile with optimizations on my system. It runs in 4-5us per inner loop. The following Common Lisp code: (defun bench-array () (declare (optimize (speed 3) (debug 0) (safety 1))) (let ((arr (make-array 3000))) (declare (dynamic-extent arr)) ; stack allocate (loop :with sum fixnum = 0 :for i fixnum :from 0 :repeat 15 :do (time (setf sum (loop :for k fixnum :from 0 :repeat 3000 :do (setf (aref arr k) i) :finally (return (loop :for l fixnum :across arr :sum l fixnum))))) (print sum)))) compiles in 0.01s and runs in 5us per inner loop on my system (SBCL). It has array bounds checking enabled (safety 1 declaration). If I remove it (safety 0), runtime improves to 2-3us per inner loop. I'll take Common Lisp over Swift any day of the week :-]
- pkulak 7y ago> Google’s team had more expertise in Swift, given that Swift’s creator Chris Lattner started the project Way to bury the lead there, haha.
- cgarciae 7y agoOne thing not mentioned in the original "Why Swift for Tensorflow" document and was a mayor source of conflict when the differentiable programming feature was formally proposed by the S4TF as a standard Swift feature: Swift has no mechanisms for metaprogramming. The reason is that Automatic Differentiation can be implemented 100% using metaprogramming, instead the S4TF team had to create internally certain features for this, that is probably one of the reasons why it took so long to get the most basic stuff working. In retrospective you can really say Swift was a bad choice for the project because the time to market was much slower than it could be vs e.g choosing Julia. The other thing they didn't take into account was the actual market, that is, the Data Science ecosystem in Swift is non-existente, you have an excellent Deep Learning library standing alone without a numpy, a pandas, a scipy, a opencv, a pillow, ect, which makes doing real application with it nearly impossible. That said, Swift as a language is amazing, doing parallel computation is so easy, not having a garbage collector makes it super efficient. Its the kind of thing we need, but the language right now is not in the right state.
- flipgimble 7y agoReading through the top level comments, they are all a form of surface level aversion to the unfamiliar. It really highlights that many industry trends are based on hot takes and not any sort of deep analysis. This makes sense why Javascript and C++ will remain as entrenched favorites despite their technical flaws. For those who actually spent time with Swift, and realize its value and potential, consider yourself lucky that large portions of the industry have an ill informed aversion to it. That creates opportunity that can be taken advantage of in the next 5 years. Developers who invest in Swift early can become market leaders, or run circles around teams struggling with slowness of python, over-complexity of c++. Top three comments paraphrased: > 1) "huh? Foundation?" but you have no qualms with `if __name__ == "__main__"` ? > 2) "The name is pretentious" is that an example of the well-substantiated and deep technical analysis HN is famous for? > 3) Swift is "a bit verbose and heavy handed" so Google FAILED by not making yet another language.
- BubRoss 7y agoI think you misunderstood criticism about the name differential programming and the idea that building in a gradient operator in to a language is somehow a breakthrough that warrants the label "software 2.0". This is not really about swift. Swift seems to have been chosen because the creator was there when they picked the language, even though he left.
- flipgimble 7y agoI think my point stands that the criticism on this thread is mostly a surface level reaction and hung up on meaningless slogans like "software 2.0" or "breakthrough". You use of the word "seems" is very apt here. Have you considered that Google might have hired Lattner precisely because he is the founder of LLVM and Swift, and they hoped to leverage his organizational skills to jump start next generation tooling? We know google is heavily invested in llvm and C++, but dissatisfied with the direction C++ is heading [0]. They also are designing custom hardware like TPUs that isn't supported well by any current language. To me it seems like they are thinking a generation or two ahead with their tooling while the outside observers can't imagine anything beyond 80s era language design. [0] https://www.infoworld.com/article/3535795/c-plus-plus-proposal-dismisses-backward-compatibility.html https://www.infoworld.com/article/3535795/c-plus-plus-propos...
- nnq 7y agoWhy the heck would anyone try to implement differentiable programming in a language with no meta-programming features whatsoever?! It's like trying to put a car into orbit or smth...
- deleted 7y ago[deleted]
- id_ris 7y agoI'm an iOS developer deeply familiar with Swift and on the path of learning ML. The irritation with "import Foundation" is understandable. Foundation is, wait for it, the Foundation of Objective-C programming and development for the iPhone. It's a library Swift makes use of and needs to inter-op with due to legacy concerns for iOS and Mac developers. It's not an inherent part of Swift the language, and efforts like those at Google and the open source development of Swift can develop more modern and suitable replacements for those libraries. I like python, but man I really don't want to make large systems in it. Swift is a great language, and imo the biggest thing holding it back is that it's intertwined with a lot of Apple code. But it doesn't need to stay that way, and for that reason I applaud the efforts to move it beyond just an "app creation" language.
- altaaf_baatli 7y agoWhy not kotlin?
- chombier 7y agoAlso, the swift ABI [1] should make it fairly easy to distribute compiled networks, right? [1] https://gankra.github.io/blah/swift-abi/ https://gankra.github.io/blah/swift-abi/
- red_admiral 7y agoThere are several things that make me give this a thumbs down. First: "All these usability problems aren’t just making it more difficult to write code, they are unnecessarily causing the industry to lag behind academia." Industry lag behind academia? Seriously, have you any idea of the amount of data that industry crunches these days, while some in academia still think half a GB is "big data"? Or the amount of money there is in the kind of ML that industry does (which is the main reason why anyone cares about this field at all). Also, your whole post is about innovation going on at ... google. Then apple. Not academia. Secondly: "swift is fast", point taken. Then you go off on a drool about your favourite syntactic sugar. My own experience is that this is exactly the kind of thing you do not need in an enterprise-grade product.
- deleted 7y ago[deleted]