13 ms·

JuliaCon2020: Julia Is Production Ready

194 points by ViralBShah 6y ago

ViralBShah 6y ago
People often ask me for examples of non-scientific Julia codebases. My favourite one is Franklin.jl (https://franklinjl.org https://franklinjl.org), a static site generator. This blog discusses quite a bit more, and it doesn't even talk about all the improvements in integrating with databases and such.
- StefanKarpinski 6y ago
  Another example is that all the infrastructure for serving Julia packages to Julia users in the new 1.5 release is implemented in Julia. This is a content distribution network served by highly concurrent servers over HTTPS. The systems interacts with git servers, GitHub APIs and AWS S3 among many other moving pieces. Code can be found here: https://github.com/JuliaPackaging/PkgServer.jl https://github.com/JuliaPackaging/PkgServer.jl. Writing servers in Julia is really pleasant thanks to the clean coroutine-based task model. Under the hood, Julia uses libuv to get efficient high-concurrency I/O with an event loop. But from the user's perspective, you just write simple blocking code and use `@sync` and `@async` to spawn concurrent tasks. If you want to use multiple threads, just use `@spawn` instead of `@async`. This is very similar to Go's highly successful I/O and concurrency model.
  - ViralBShah 6y ago
    That's a great example. Also, JuliaBox.com (discontinued service) ran on Julia in production for 3 years, and has now been replaced by JuliaHub.com, which is also largely written in Julia. Several Julia Computing customers also run long running Julia jobs in production, and some are discussed in our Case Studies (https://juliacomputing.com/case-studies/ https://juliacomputing.com/case-studies/).
- bransonf 6y ago
  My favorite is Genie. [0] https://genieframework.com/ https://genieframework.com/
dnautics 6y ago
Having followed it since the 0.4 era I'm excited that Julia is production-ready (though I have professionally moved to elixir). Since I'm working in ml infra, I'm interested in orchestrating Julia jobs (via Elixir), what I'm less clear on is if there are good resources on how to deploy Julia, keep dependencies sane, pin memory usage. Or does one just go straight to containers?
- StefanKarpinski 6y ago
  For services we deploy at Julia Computing, we use docker containers since that's just the way devops is done these days, but we also use manifest files to pin all our dependencies which almost makes dockerizing things redundant. It doesn't hurt to have a couple of layers ensuring reproducibility.
  - dnautics 6y ago
    Thanks! I'm also very excited by pluto.jl, looks fantastic. I'll probably be offering public machine learning vms with Pluto builtin by the end of the year.
    - ScottPJones 6y ago
      I'm having a lot of fun with Pluto.jl and PlutoUI.jl this past week, it's so easy to use and add interactivity. There are still some rough edges, but that's what PRs are made for!
- staticfloat 6y ago
  Yep, to build on Stefan's sibling comment, we deploy a few services through docker that follow the flow of <merge on github> -> <build on dockerhub> -> <deploy through watchtower> that has been working for us really well. Docker has some nice infrastructure that we like, such as the ability to set memory limits, automatically restart processes on crash/if a healthcheck goes sour/on reboot, easily spin up different versions of the same service (based on which image you tag to a name), etc... Containerization has done a lot for our ability to bring servers up onto heterogenous hardware easily and quickly. As an example, this repository [0] is deployed on 10 machines around the world, providing the pkg servers that users connect to in geographically-disparate regions. The README on that repository is a small rant that I wrote a while back that walks through some of the decisions made in that repository and why I feel they're good ones for deployment. To answer your specific questions: * How to deploy Julia: we use docker containers and watchtower [1] to automatically deploy new versions of Julia. * Keep dependencies sane: it's all in the Pkg Manifests. We never do `Pkg.update()` on our deployments, we only ever `Pkg.instantiate()`. * Pin memory usage: We use docker to apply cgroup limits to kill processes with runaway memory usage [2]. This is only triggered by bugs (normal program execution does not expect to ever hit this) but bugs happen, so it's good to not have to restart your VM because you can't start a new ssh session without your shell triggering the OOM killer. ;) [0] https://github.com/JuliaPackaging/PkgServerS3Mirror https://github.com/JuliaPackaging/PkgServerS3Mirror [1] https://github.com/containrrr/watchtower https://github.com/containrrr/watchtower [2] https://github.com/JuliaPackaging/PkgServerS3Mirror/blob/c6ae2c8e9a7b228cebbcf48c25fd2381be97b565/docker-compose.yml#L18 https://github.com/JuliaPackaging/PkgServerS3Mirror/blob/c6a...
oxinabox 6y ago
Invenia had run Julia for our primary production system for over a year (nearly 2). The size of our Julia codebase is about 400,000 lines of code, spread over numerous internal and open source packages. You can cite the "we're hiring" slide from my Juliacon talk for that https://raw.githack.com/oxinabox/ChainRulesJuliaCon2020/main/out/build/index.html#2 https://raw.githack.com/oxinabox/ChainRulesJuliaCon2020/main...
- 7thaccount 6y ago
  As someone working in power systems (electrical engineer) and who programs in Julia for fun a bit. I'm curious about y'all's business model. I've asked before and only gotten something about load forecasting which I can't see supporting 30 people. Can you shed any light on the business model? Contract work? Do y'all have an actual software product? Please note that I'm not being critical, I am legitimately curious as the work y'all do seems to be very academic which I tend to see more coming out of universities. 400k loc is a lot btw, so it seems like y'all are pretty busy!
  - oxinabox 6y ago
    The short answer is that you are underestimating the scale. Invenia operates over most of grids in the continental US. Think less "helping one powerplant how much to generate, and thus saving them money on wasted fuel if not needed." And more "helping thousands...", we're more linked to the system operators than the individual plants. And the systems move huge amount of power each day, so small percentage efficiencies is still a big number dollar wise
    - 7thaccount 6y ago
      No, I know what a system operator is which is why I'm asking the question. It was a fair assumption though to assume I was talking about the utility level of granularity though. As far as I know, all ISO's in the USA use either ABB, Siemens, or GE for their market optimization software (huge systems) or basically those same vendors plus a few more for their EMS systems. There are also plenty of other amazing products for transmission planning, short term studies, and transient stability studies. More specifically, how are you linked to the system operators? When you say you optimize the grid, that could mean any one of a thousand different things. Are you providing various inputs (forecasts, risk factors, etc) that are used by some markets? Please note that I'm not questioning that y'all work with some of these entities in some fashion, I'm just trying to get a better handle on the specifics of where y'all are plugged in.
adamnemecek 6y ago
Julia is next gen. The whole experience is so much more pleasant than anything in the space. The interop is also crazy. How is it possible that you can call Matlab, Python, cpp, fortran, Rust from one language? I legit think that even if you are using say pytorch. You are better off writing your code in julia and using the python interop.
- amkkma 6y ago
  Using pytorch/keras/TF from Julia is possible, and even pleasant: https://twitter.com/aureliengeron/status/1277751121440698368 https://twitter.com/aureliengeron/status/1277751121440698368 Though I much prefer Flux.jl
  - UncleOxidant 6y ago
    In my experience Flux.jl has a ways to go before it reaches Pytorch performance. Apparently this has something to do with some very efficient GPU kernels that Pytorch uses - I wonder if it's possible for Flux to "borrow" those GPU kernels? It's possible to write GPU kernels in Julia so hopefully this will help close the gap.
    - amkkma 6y ago
      The short answer is yes, it's possible and easy. The longer answer is, that's a bit of an apples to oranges comparison. Pytorch is calling optimized c++ kernels, whereas Flux is completely generic and can do GPU and CPU codegen (including AMD). Flux can do things Pytorch just doesn't have CUDA code for, It's more generic and composable and it's getting there with a much smaller team/budget. That means it has more potential, with the drawback that it isn't quite as fast, for now, in some of the common use cases the pytorch has been hard coded/ hyperoptimized for. In other more complex cases beyond big fat matmuls and such, it's much faster, even comparing to torchscript (sciml, neural ODEs etc). Still, the codegen is getting there with the funded work on the GPU compiler and memory allocation and there eventually won't be any sort of tradeoff. In the meantime, Yes, there is a package at allows calling pytorch kernels without modifying your code. This is also only possible with such finesse due to Julia's power and composability. See here for the package: https://fluxml.ai/2020/06/29/acclerating-flux-torch.html https://fluxml.ai/2020/06/29/acclerating-flux-torch.html
sadfev 6y ago
Is it actually backwards compatible? I don’t think it is, they continually introduce breaking syntax changes. Please somebody enlighten me
- amkkma 6y ago
  It's been backwards compatible since the 1.0 release two years ago.
  - StefanKarpinski 6y ago
    Julia follows semantic versioning [1] and is now on version 1.5, which means there been five non-breaking releases introducing new features (and dozens of bugfix only patch releases). Code that worked on 1.0 also works on 1.1, 1.2, 1.3, 1.4 and 1.5 and will keep working in 1.6, etc. until such a time as a breaking 2.0 release is made. Even then breaking changes will be kept minimal and only made to enable worthwhile language features. [1] https://semver.org https://semver.org
vasili111 6y ago
What is the state of data frame support in Julia?
- 3JPLW 6y ago
  Exceptional — and its lead author is the author of this blog post. http://juliadata.github.io/DataFrames.jl/stable/ http://juliadata.github.io/DataFrames.jl/stable/
  - chubot 6y ago
    Is that what you meant to link? It looks like the official docs, and not a blog post.
    - LolWolf 6y ago
      I think “this blog post” refers to the OP :)
    - rsfern 6y ago
      This post is also relevant: https://bkamins.github.io/julialang/2020/08/02/post_juliacon_1.html https://bkamins.github.io/julialang/2020/08/02/post_juliacon... Edit: it’s a bit more into the weeds than I thought. Honestly the docs give a better picture of DataFrames.jl
- nextos 6y ago
  DataFrames.jl provides very similar functionality to the R DataFrame, plus some functions similar to those from the R reshape package. Other packages build upon this to provide e.g. LINQ-like operators [2]. [1] https://juliadata.github.io/DataFrames.jl https://juliadata.github.io/DataFrames.jl [2] https://github.com/queryverse/Query.jl https://github.com/queryverse/Query.jl
- smabie 6y ago
  It's fine, but I think row indices would add a lot. Right now there's also a TimeSeries package that could be obviate if Julia had a Pandas-like implementation. And the TimeSeries functionality is very primitive, not even supporting resampling and forward or backfilling data Row indices are super powerful and things are annoying without them. I've actually been thinking about programming a new and improved data frame library inspired by Pandas (but not so ugly and complicated), but the amount of work to write and then maintain it would be prohibitive.
noahamar 6y ago
Highly disappointed that this isn't a convention for Julia Roberts fans.
deleted 6y ago
[deleted]
lordgroff 6y ago
I've been thinking about jumping on the Julia ship for a while. I like the LISP-like nature of R and miss its flexibility very much while working in Python (a language I also enjoy, but not particularly for data science). Julia, specifically its macro story and speed, fascinate me. One of the things that still keeps me at bay is the JIT and the pre-compilation in general. It seems like it's still not very easy to actually compile a Julia library or an executable. There is PackageCompiler.jl, but my understanding is that it necessitates pulling in sys.so, which is some 130 MB in size, making it prohibitive for many projects.
- ViralBShah 6y ago
  We are always working on streamlining PackageCompiler.jl and building binaries. Bug reports always appreciated. I'm curious to understand why the 130MB is prohibitive. Are you looking at embedded or resource constrained environments?
  - lordgroff 6y ago
    It's more generic than that. For basic general purpose computing with some calculation and where performance is of some importance, I can design and ship the whole thing via something like Lazarus in 5-10 MB, or -- from my understanding -- 130+ MB in Julia. I understand that this might be just asking Julia to break out a bit too far out of its intended use case, but it would be nice.
    - deleted 6y ago
      [deleted]
  - peatmoss 6y ago
    I know for me the documentation and general process of using PackageCompiler.jl is a little daunting for me in a way that other parts of the ecosystem and docs haven’t been. I was just trying to do a quick test app the other day to see if I could build a Hello World app, and (this may have been my fault) but I hit the threshold of cognitive overhead before deciding to revisit later. This isn’t a criticism so much as a recognition that compiling binaries may take a bit more care / attention / learning in Julia than something like Go, Rust, or even Racket. As for binary sizes, for some serverless deployments smaller binaries can open up a little easier path. I’ll try and take another run at PackageCompiler and give it a more fair shake. If in that process I find anything constructive to report or submit documentation PRs, I will do so. Thank you for your work on Julia. Julia was on my radar a few years ago and then dropped off again. I’m excited to see where it has come in that time.
jeanl 6y ago
Do you guys know if it's possible to generate self-standing C++ code from Julia code? (i.e., code that you can then compile into another project, and will not require any interpreter to run or anything like that)? And if it's possible do you know how good the generated code is (in terms of speed)?
- ellisv 6y ago
  You can create binaries from Julia and there are APIs to call Julia code from C/C++ https://julialang.github.io/PackageCompiler.jl/dev/devdocs/binaries_part_2/ https://julialang.github.io/PackageCompiler.jl/dev/devdocs/b... https://docs.julialang.org/en/v1/manual/embedding/ https://docs.julialang.org/en/v1/manual/embedding/
- ViralBShah 6y ago
  We did have this once upon a time, going from Julia->LLVM->C using the llvm-cbe project (https://github.com/JuliaComputing/llvm-cbe https://github.com/JuliaComputing/llvm-cbe). It is not supported any more, since there are easier ways to achieve what you are asking for. If your code does eval, then there is no way out. If you can ensure your code and dependencies does not have dynamic things that need compilation at runtime, it is fairly easy to remove the dependency on bundling LLVM. Speed is the same, since you are then just running cached native code.
- deleted 6y ago
  [deleted]
amacbride 6y ago
I really enjoyed the workshops and the talks; I thought the SciML and MLJ talks were particularly good. Everything is available on their YouTube channel: https://www.youtube.com/playlist?list=PLP8iPy9hna6Tl2UHTrm4jnIYrLkIcAROR https://www.youtube.com/playlist?list=PLP8iPy9hna6Tl2UHTrm4j...
andi999 6y ago
I can currently see Julia only as a replacement for Python, with the biggest advantage, that fast modules can be written in-language, and you do not default back to C. So in a sense it could solve the two language problem. But then for my applicationsspace there are problems: - compilation seems possible now, but one has to be careful of GPL (like in the fft) - no way to turn off GC (so byebye real-time possibilities). - what about GUI design? Heard some mixed messages about it
- eigenspace 6y ago
  > - no way to turn off GC (so byebye real-time possibilities). While Julia does not offer semantic guarantees that you can avoid the GC, it;s actually quite possible and easy to write code where you're manually managing all your memory and the GC is never invoked. > - what about GUI design? Heard some mixed messages about it There's a lot of promising things happening with Makie.jl, Stipple.jl, Dash.jl, Pluto.jl and others. This space is still a little immature in Julia, but it's progressing fast.
  - andi999 6y ago
    I know there are workarounds, but when you do not have guarantees the phrase 'production ready'is not the first that comes to my mind. (Of course in the problem spaces of matlab and python, and probably R, Julia is probably on par)
    - DNF2 6y ago
      Well, how high is the bar for calling a language 'production ready' in that case? How many production ready languages are there?
      - andi999 6y ago
        What I meant: production ready depends on the feature. So for substituting Python (and beeing able to write fast in-language extensions), the language is most likely production ready; so in a sense it is a replacement for Python + C; but not a replacement for C. And this is a bit sad, because it could have been (building linked executables should be possible, since the language is compiled). But I like your question, lets call it 'which languages are all purpose production ready'; probably only C, C++, Java (there is a real-time version), LabView.
usgroup 6y ago
If you were trying to motivate someone embedded in Python / scipy / pandas / etc to move to Julia at this time , what would you say?
- eigenspace 6y ago
  I think this post: https://julialang.org/blog/2012/02/why-we-created-julia/ https://julialang.org/blog/2012/02/why-we-created-julia/ presents the aims of the Julia project at a high level quite well. That post was partly aspirational at the time, and partly already achieved. Now, 8 years later, I'd say essentially everything there has been achieved. The most important difference between Julia now and Julia then is the gigantic, vibrant and interacting ecosystem that's sprung up (though share are other important technical differences too!) _______________________________________________ Here's the pitch I'd give: Julia's value proposition is mostly geared towards people trying to do things that are just too difficult, awkward or slow in other languages. It's not just about speed, dynamism and friendly syntax. It's also about solving the expression problem and providing unprecedented composability. So while it may not necessarily have great appeal to 'end users' (who are people just calling numpy/scipy/pandas/whatever functions the way they were meant to be called by the library writers), it does appeal to people who make the sort of things end users want so the community is currently swelling with very talented people making state of the art research libraries. The language is designed for 'composability', so that it's very easy to combine packages together in ways the authors didn't necessarily intend or forsee. This approach is really starting to bear fruit. Our package ecosystem has some real, state of the art stuff not easily available in other languages, and it's often built in less time and with less resources. For 'end users', whether or not Julia makes sense for them really just depends on if the things they need are already available. Another huge advantage is that even if you're not some super researcher developer, Julia substantially lowers the bar for mere mortals such as myself to do package development. The composability I mentioned above means that I can stand on the shoulders of giants as I implement my own stuff, plus the fact that the majority of libraries are written in pure julia means that I can easily "peek under the hood" to figure out how they work and learn from them. This is really special and can't be replicated in python, because 'peeking under the hood' invariably means reading very specialized C code, rather than Python code. I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. In the meantime, we have an excellent package for calling and running Python code from withing Julia, called PyCall.jl [1], so that can make the burden of switching even easier. [1] https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl
sgillen 6y ago
Is the debugger in Vs code or anywhere else better now? I remember trying Julia just a few months ago and the debugger frustrated me enough that I went back to print statement debugging.
- Eugeleo 6y ago
  I’m not using Julia actively, but I’ve been following the news and I there was a lot of improvements made to the VS Code extension, and better debugging experience as one of them. You should try it out again.
malshe 6y ago
Thanks for sharing this. Are there instructor resources available to teach Julia in a data science program? I use R mostly but I am porting one course on ML applications to Python. The communities in both R and Python are vibrant for instructors to create structured learning experiences for data science. How about Julia?
- amkkma 6y ago
  https://juliaacademy.com/ https://juliaacademy.com/ https://github.com/mitmath/6S083 https://github.com/mitmath/6S083 https://www.youtube.com/watch?v=R84L-BQcjHw&t=905s https://www.youtube.com/watch?v=R84L-BQcjHw&t=905s
  - malshe 6y ago
    That's great, thanks!
- ViralBShah 6y ago
  See the data science course by Huda Nassar on JuliaAcademy.com.
  - malshe 6y ago
    Thanks! Edit: This looks like a fantastic resource! Full link: https://juliaacademy.com https://juliaacademy.com
ablekh 6y ago
While being a big fan of Julia in general and recognizing a very significant progress in both language and ecosystem development [kudos to Julia's core team as well as numerous contributors to the language and its growing ecosystem!], I respectfully disagree with the post's author conclusion in its entirety. For scientific computing and advanced machine learning domains, Julia is certainly a breath of fresh air and a no-brainer decision. However, as a general-purpose language as well as technology stack, I think Julia has still a long way to go. This is mostly due to relative immaturity of that part of the ecosystem, spotty - and sometimes even non-existent! - documentation (of course, except for the language itself and quite a limited number of core and popular packages) as well as some other issues, including tooling, development/compilation performance and limited pool of skilled developers. So, from a startup founder perspective [who has to select the optimal, risk-minimizing, platform stack], despite Julia's many attractive features (including powerful meta-programming facilities - in my case, for potential DSL development), I'm now leaning toward Python and .NET ecosystems. Both offer very mature and large package ecosystems along with comprehensive tooling support and incomparably larger pool of skilled developers. Additionally, .NET offers stability of consistent development improvements, backing of a major corporation [no acquisition risk] along with support for modern enterprise-focused architectural practices and patterns, including DDD and multi-tenancy.
- avasthe 6y ago
  This. Not trolling. I don't really know why people like to shoehorn a purpose-oriented language into all domains. Eg: Julia for scientific computing, Rust for low level programming. They would be great if they focused on those "strong zones".
  - ddragon 6y ago
    I read this article not as a "Julia is ready to replace the all-purpose languages used in business", but as "Julia is ready to deploy it's scientific computing into production environments" (as opposed to just local/hobby tasks and academic environments which were the early focus). No one is recommending people to do their small e-commerce in Julia (or perhaps Rust) unless you're in for the fun of the languages (or your e-commerce happens to be a special case that will play with either language's strengths). But having a solid web deployment story is important to both regardless, be it in Julia to have live dashboards or integrating your scientific models in your microservice architecture and to serve them directly to clients or Rust for embedded web servers for IoT and other devices. For a language to be great in one domain, it has to be at least good in everything around it for when you need to connect that domain to the real world. And Julia creators do focus on it's strong zone, with all the works on TPU, HPC and parallel computing, and so does the community with the stuff presented in juliacon like interactive reproducible notebooks (Pluto.jl) or data dashboards (Stipple.jl), both using the web domain to improve the scientific computing domain.
xiaodai 6y ago
Does Julia webstack have middle-ware like authenticators etc etc?
minetest2048 6y ago
Julia feels like a good replacement for Matlab, but then Matlab have Simulink, is there any project to create something like Simulink but for Julia?
- ChrisRackauckas 6y ago
  Something ModelingToolkit.jl, JuSDL.jl, NetworkDynamics.jl, Modia.jl, etc. all are extensions of Simulink in some form. Supporting stochastic equations, delays, automatic parallelization, acausal modeling, etc.
vasili111 6y ago
Does the Julia matrix has named columns as R or unnamed as numpy?
- ddragon 6y ago
  The built-in array (which is multidimensional) is unnamed, but Julia was made in a way that anyone can create as a library their own types that are as high performance as the native ones (most of which are written in Julia) so you can just get a named array package like [1]. [1] https://github.com/davidavdav/NamedArrays.jl https://github.com/davidavdav/NamedArrays.jl

194 points by ViralBShah 6y ago

People often ask me for examples of non-scientific Julia codebases. My favourite one is Franklin.jl (https://franklinjl.org https://franklinjl.org), a static site generator. This blog discusses quite a bit more, and it doesn't even talk about all the improvements in integrating with databases and such.

StefanKarpinski 6y ago
Another example is that all the infrastructure for serving Julia packages to Julia users in the new 1.5 release is implemented in Julia. This is a content distribution network served by highly concurrent servers over HTTPS. The systems interacts with git servers, GitHub APIs and AWS S3 among many other moving pieces. Code can be found here: https://github.com/JuliaPackaging/PkgServer.jl https://github.com/JuliaPackaging/PkgServer.jl. Writing servers in Julia is really pleasant thanks to the clean coroutine-based task model. Under the hood, Julia uses libuv to get efficient high-concurrency I/O with an event loop. But from the user's perspective, you just write simple blocking code and use `@sync` and `@async` to spawn concurrent tasks. If you want to use multiple threads, just use `@spawn` instead of `@async`. This is very similar to Go's highly successful I/O and concurrency model.
- ViralBShah 6y ago
  That's a great example. Also, JuliaBox.com (discontinued service) ran on Julia in production for 3 years, and has now been replaced by JuliaHub.com, which is also largely written in Julia. Several Julia Computing customers also run long running Julia jobs in production, and some are discussed in our Case Studies (https://juliacomputing.com/case-studies/ https://juliacomputing.com/case-studies/).
bransonf 6y ago
My favorite is Genie. [0] https://genieframework.com/ https://genieframework.com/

dnautics 6y ago

Having followed it since the 0.4 era I'm excited that Julia is production-ready (though I have professionally moved to elixir). Since I'm working in ml infra, I'm interested in orchestrating Julia jobs (via Elixir), what I'm less clear on is if there are good resources on how to deploy Julia, keep dependencies sane, pin memory usage. Or does one just go straight to containers?

StefanKarpinski 6y ago
For services we deploy at Julia Computing, we use docker containers since that's just the way devops is done these days, but we also use manifest files to pin all our dependencies which almost makes dockerizing things redundant. It doesn't hurt to have a couple of layers ensuring reproducibility.
- dnautics 6y ago
  Thanks! I'm also very excited by pluto.jl, looks fantastic. I'll probably be offering public machine learning vms with Pluto builtin by the end of the year.
  - ScottPJones 6y ago
    I'm having a lot of fun with Pluto.jl and PlutoUI.jl this past week, it's so easy to use and add interactivity. There are still some rough edges, but that's what PRs are made for!
staticfloat 6y ago
Yep, to build on Stefan's sibling comment, we deploy a few services through docker that follow the flow of <merge on github> -> <build on dockerhub> -> <deploy through watchtower> that has been working for us really well. Docker has some nice infrastructure that we like, such as the ability to set memory limits, automatically restart processes on crash/if a healthcheck goes sour/on reboot, easily spin up different versions of the same service (based on which image you tag to a name), etc... Containerization has done a lot for our ability to bring servers up onto heterogenous hardware easily and quickly. As an example, this repository [0] is deployed on 10 machines around the world, providing the pkg servers that users connect to in geographically-disparate regions. The README on that repository is a small rant that I wrote a while back that walks through some of the decisions made in that repository and why I feel they're good ones for deployment. To answer your specific questions: * How to deploy Julia: we use docker containers and watchtower [1] to automatically deploy new versions of Julia. * Keep dependencies sane: it's all in the Pkg Manifests. We never do `Pkg.update()` on our deployments, we only ever `Pkg.instantiate()`. * Pin memory usage: We use docker to apply cgroup limits to kill processes with runaway memory usage [2]. This is only triggered by bugs (normal program execution does not expect to ever hit this) but bugs happen, so it's good to not have to restart your VM because you can't start a new ssh session without your shell triggering the OOM killer. ;) [0] https://github.com/JuliaPackaging/PkgServerS3Mirror https://github.com/JuliaPackaging/PkgServerS3Mirror [1] https://github.com/containrrr/watchtower https://github.com/containrrr/watchtower [2] https://github.com/JuliaPackaging/PkgServerS3Mirror/blob/c6ae2c8e9a7b228cebbcf48c25fd2381be97b565/docker-compose.yml#L18 https://github.com/JuliaPackaging/PkgServerS3Mirror/blob/c6a...

oxinabox 6y ago

Invenia had run Julia for our primary production system for over a year (nearly 2). The size of our Julia codebase is about 400,000 lines of code, spread over numerous internal and open source packages. You can cite the "we're hiring" slide from my Juliacon talk for that https://raw.githack.com/oxinabox/ChainRulesJuliaCon2020/main/out/build/index.html#2 https://raw.githack.com/oxinabox/ChainRulesJuliaCon2020/main...

7thaccount 6y ago
As someone working in power systems (electrical engineer) and who programs in Julia for fun a bit. I'm curious about y'all's business model. I've asked before and only gotten something about load forecasting which I can't see supporting 30 people. Can you shed any light on the business model? Contract work? Do y'all have an actual software product? Please note that I'm not being critical, I am legitimately curious as the work y'all do seems to be very academic which I tend to see more coming out of universities. 400k loc is a lot btw, so it seems like y'all are pretty busy!
- oxinabox 6y ago
  The short answer is that you are underestimating the scale. Invenia operates over most of grids in the continental US. Think less "helping one powerplant how much to generate, and thus saving them money on wasted fuel if not needed." And more "helping thousands...", we're more linked to the system operators than the individual plants. And the systems move huge amount of power each day, so small percentage efficiencies is still a big number dollar wise
  - 7thaccount 6y ago
    No, I know what a system operator is which is why I'm asking the question. It was a fair assumption though to assume I was talking about the utility level of granularity though. As far as I know, all ISO's in the USA use either ABB, Siemens, or GE for their market optimization software (huge systems) or basically those same vendors plus a few more for their EMS systems. There are also plenty of other amazing products for transmission planning, short term studies, and transient stability studies. More specifically, how are you linked to the system operators? When you say you optimize the grid, that could mean any one of a thousand different things. Are you providing various inputs (forecasts, risk factors, etc) that are used by some markets? Please note that I'm not questioning that y'all work with some of these entities in some fashion, I'm just trying to get a better handle on the specifics of where y'all are plugged in.

adamnemecek 6y ago

Julia is next gen. The whole experience is so much more pleasant than anything in the space. The interop is also crazy. How is it possible that you can call Matlab, Python, cpp, fortran, Rust from one language? I legit think that even if you are using say pytorch. You are better off writing your code in julia and using the python interop.

amkkma 6y ago
Using pytorch/keras/TF from Julia is possible, and even pleasant: https://twitter.com/aureliengeron/status/1277751121440698368 https://twitter.com/aureliengeron/status/1277751121440698368 Though I much prefer Flux.jl
- UncleOxidant 6y ago
  In my experience Flux.jl has a ways to go before it reaches Pytorch performance. Apparently this has something to do with some very efficient GPU kernels that Pytorch uses - I wonder if it's possible for Flux to "borrow" those GPU kernels? It's possible to write GPU kernels in Julia so hopefully this will help close the gap.
  - amkkma 6y ago
    The short answer is yes, it's possible and easy. The longer answer is, that's a bit of an apples to oranges comparison. Pytorch is calling optimized c++ kernels, whereas Flux is completely generic and can do GPU and CPU codegen (including AMD). Flux can do things Pytorch just doesn't have CUDA code for, It's more generic and composable and it's getting there with a much smaller team/budget. That means it has more potential, with the drawback that it isn't quite as fast, for now, in some of the common use cases the pytorch has been hard coded/ hyperoptimized for. In other more complex cases beyond big fat matmuls and such, it's much faster, even comparing to torchscript (sciml, neural ODEs etc). Still, the codegen is getting there with the funded work on the GPU compiler and memory allocation and there eventually won't be any sort of tradeoff. In the meantime, Yes, there is a package at allows calling pytorch kernels without modifying your code. This is also only possible with such finesse due to Julia's power and composability. See here for the package: https://fluxml.ai/2020/06/29/acclerating-flux-torch.html https://fluxml.ai/2020/06/29/acclerating-flux-torch.html

sadfev 6y ago

Is it actually backwards compatible? I don’t think it is, they continually introduce breaking syntax changes. Please somebody enlighten me

amkkma 6y ago
It's been backwards compatible since the 1.0 release two years ago.
- StefanKarpinski 6y ago
  Julia follows semantic versioning [1] and is now on version 1.5, which means there been five non-breaking releases introducing new features (and dozens of bugfix only patch releases). Code that worked on 1.0 also works on 1.1, 1.2, 1.3, 1.4 and 1.5 and will keep working in 1.6, etc. until such a time as a breaking 2.0 release is made. Even then breaking changes will be kept minimal and only made to enable worthwhile language features. [1] https://semver.org https://semver.org

vasili111 6y ago

What is the state of data frame support in Julia?

3JPLW 6y ago
Exceptional — and its lead author is the author of this blog post. http://juliadata.github.io/DataFrames.jl/stable/ http://juliadata.github.io/DataFrames.jl/stable/
- chubot 6y ago
  Is that what you meant to link? It looks like the official docs, and not a blog post.
  - LolWolf 6y ago
    I think “this blog post” refers to the OP :)
  - rsfern 6y ago
    This post is also relevant: https://bkamins.github.io/julialang/2020/08/02/post_juliacon_1.html https://bkamins.github.io/julialang/2020/08/02/post_juliacon... Edit: it’s a bit more into the weeds than I thought. Honestly the docs give a better picture of DataFrames.jl
nextos 6y ago
DataFrames.jl provides very similar functionality to the R DataFrame, plus some functions similar to those from the R reshape package. Other packages build upon this to provide e.g. LINQ-like operators [2]. [1] https://juliadata.github.io/DataFrames.jl https://juliadata.github.io/DataFrames.jl [2] https://github.com/queryverse/Query.jl https://github.com/queryverse/Query.jl
smabie 6y ago
It's fine, but I think row indices would add a lot. Right now there's also a TimeSeries package that could be obviate if Julia had a Pandas-like implementation. And the TimeSeries functionality is very primitive, not even supporting resampling and forward or backfilling data Row indices are super powerful and things are annoying without them. I've actually been thinking about programming a new and improved data frame library inspired by Pandas (but not so ugly and complicated), but the amount of work to write and then maintain it would be prohibitive.

noahamar 6y ago

Highly disappointed that this isn't a convention for Julia Roberts fans.

deleted 6y ago

[deleted]

lordgroff 6y ago

I've been thinking about jumping on the Julia ship for a while. I like the LISP-like nature of R and miss its flexibility very much while working in Python (a language I also enjoy, but not particularly for data science). Julia, specifically its macro story and speed, fascinate me. One of the things that still keeps me at bay is the JIT and the pre-compilation in general. It seems like it's still not very easy to actually compile a Julia library or an executable. There is PackageCompiler.jl, but my understanding is that it necessitates pulling in sys.so, which is some 130 MB in size, making it prohibitive for many projects.

ViralBShah 6y ago
We are always working on streamlining PackageCompiler.jl and building binaries. Bug reports always appreciated. I'm curious to understand why the 130MB is prohibitive. Are you looking at embedded or resource constrained environments?
- lordgroff 6y ago
  It's more generic than that. For basic general purpose computing with some calculation and where performance is of some importance, I can design and ship the whole thing via something like Lazarus in 5-10 MB, or -- from my understanding -- 130+ MB in Julia. I understand that this might be just asking Julia to break out a bit too far out of its intended use case, but it would be nice.
  - deleted 6y ago
    [deleted]
- peatmoss 6y ago
  I know for me the documentation and general process of using PackageCompiler.jl is a little daunting for me in a way that other parts of the ecosystem and docs haven’t been. I was just trying to do a quick test app the other day to see if I could build a Hello World app, and (this may have been my fault) but I hit the threshold of cognitive overhead before deciding to revisit later. This isn’t a criticism so much as a recognition that compiling binaries may take a bit more care / attention / learning in Julia than something like Go, Rust, or even Racket. As for binary sizes, for some serverless deployments smaller binaries can open up a little easier path. I’ll try and take another run at PackageCompiler and give it a more fair shake. If in that process I find anything constructive to report or submit documentation PRs, I will do so. Thank you for your work on Julia. Julia was on my radar a few years ago and then dropped off again. I’m excited to see where it has come in that time.

jeanl 6y ago

Do you guys know if it's possible to generate self-standing C++ code from Julia code? (i.e., code that you can then compile into another project, and will not require any interpreter to run or anything like that)? And if it's possible do you know how good the generated code is (in terms of speed)?

ellisv 6y ago
You can create binaries from Julia and there are APIs to call Julia code from C/C++ https://julialang.github.io/PackageCompiler.jl/dev/devdocs/binaries_part_2/ https://julialang.github.io/PackageCompiler.jl/dev/devdocs/b... https://docs.julialang.org/en/v1/manual/embedding/ https://docs.julialang.org/en/v1/manual/embedding/
ViralBShah 6y ago
We did have this once upon a time, going from Julia->LLVM->C using the llvm-cbe project (https://github.com/JuliaComputing/llvm-cbe https://github.com/JuliaComputing/llvm-cbe). It is not supported any more, since there are easier ways to achieve what you are asking for. If your code does eval, then there is no way out. If you can ensure your code and dependencies does not have dynamic things that need compilation at runtime, it is fairly easy to remove the dependency on bundling LLVM. Speed is the same, since you are then just running cached native code.
deleted 6y ago
[deleted]

amacbride 6y ago

I really enjoyed the workshops and the talks; I thought the SciML and MLJ talks were particularly good. Everything is available on their YouTube channel: https://www.youtube.com/playlist?list=PLP8iPy9hna6Tl2UHTrm4jnIYrLkIcAROR https://www.youtube.com/playlist?list=PLP8iPy9hna6Tl2UHTrm4j...

andi999 6y ago

I can currently see Julia only as a replacement for Python, with the biggest advantage, that fast modules can be written in-language, and you do not default back to C. So in a sense it could solve the two language problem. But then for my applicationsspace there are problems: - compilation seems possible now, but one has to be careful of GPL (like in the fft) - no way to turn off GC (so byebye real-time possibilities). - what about GUI design? Heard some mixed messages about it

eigenspace 6y ago
> - no way to turn off GC (so byebye real-time possibilities). While Julia does not offer semantic guarantees that you can avoid the GC, it;s actually quite possible and easy to write code where you're manually managing all your memory and the GC is never invoked. > - what about GUI design? Heard some mixed messages about it There's a lot of promising things happening with Makie.jl, Stipple.jl, Dash.jl, Pluto.jl and others. This space is still a little immature in Julia, but it's progressing fast.
- andi999 6y ago
  I know there are workarounds, but when you do not have guarantees the phrase 'production ready'is not the first that comes to my mind. (Of course in the problem spaces of matlab and python, and probably R, Julia is probably on par)
  - DNF2 6y ago
    Well, how high is the bar for calling a language 'production ready' in that case? How many production ready languages are there?
    - andi999 6y ago
      What I meant: production ready depends on the feature. So for substituting Python (and beeing able to write fast in-language extensions), the language is most likely production ready; so in a sense it is a replacement for Python + C; but not a replacement for C. And this is a bit sad, because it could have been (building linked executables should be possible, since the language is compiled). But I like your question, lets call it 'which languages are all purpose production ready'; probably only C, C++, Java (there is a real-time version), LabView.

usgroup 6y ago

If you were trying to motivate someone embedded in Python / scipy / pandas / etc to move to Julia at this time , what would you say?

eigenspace 6y ago
I think this post: https://julialang.org/blog/2012/02/why-we-created-julia/ https://julialang.org/blog/2012/02/why-we-created-julia/ presents the aims of the Julia project at a high level quite well. That post was partly aspirational at the time, and partly already achieved. Now, 8 years later, I'd say essentially everything there has been achieved. The most important difference between Julia now and Julia then is the gigantic, vibrant and interacting ecosystem that's sprung up (though share are other important technical differences too!) _______________________________________________ Here's the pitch I'd give: Julia's value proposition is mostly geared towards people trying to do things that are just too difficult, awkward or slow in other languages. It's not just about speed, dynamism and friendly syntax. It's also about solving the expression problem and providing unprecedented composability. So while it may not necessarily have great appeal to 'end users' (who are people just calling numpy/scipy/pandas/whatever functions the way they were meant to be called by the library writers), it does appeal to people who make the sort of things end users want so the community is currently swelling with very talented people making state of the art research libraries. The language is designed for 'composability', so that it's very easy to combine packages together in ways the authors didn't necessarily intend or forsee. This approach is really starting to bear fruit. Our package ecosystem has some real, state of the art stuff not easily available in other languages, and it's often built in less time and with less resources. For 'end users', whether or not Julia makes sense for them really just depends on if the things they need are already available. Another huge advantage is that even if you're not some super researcher developer, Julia substantially lowers the bar for mere mortals such as myself to do package development. The composability I mentioned above means that I can stand on the shoulders of giants as I implement my own stuff, plus the fact that the majority of libraries are written in pure julia means that I can easily "peek under the hood" to figure out how they work and learn from them. This is really special and can't be replicated in python, because 'peeking under the hood' invariably means reading very specialized C code, rather than Python code. I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. In the meantime, we have an excellent package for calling and running Python code from withing Julia, called PyCall.jl [1], so that can make the burden of switching even easier. [1] https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl

sgillen 6y ago

Is the debugger in Vs code or anywhere else better now? I remember trying Julia just a few months ago and the debugger frustrated me enough that I went back to print statement debugging.

Eugeleo 6y ago
I’m not using Julia actively, but I’ve been following the news and I there was a lot of improvements made to the VS Code extension, and better debugging experience as one of them. You should try it out again.

malshe 6y ago

Thanks for sharing this. Are there instructor resources available to teach Julia in a data science program? I use R mostly but I am porting one course on ML applications to Python. The communities in both R and Python are vibrant for instructors to create structured learning experiences for data science. How about Julia?

amkkma 6y ago
https://juliaacademy.com/ https://juliaacademy.com/ https://github.com/mitmath/6S083 https://github.com/mitmath/6S083 https://www.youtube.com/watch?v=R84L-BQcjHw&t=905s https://www.youtube.com/watch?v=R84L-BQcjHw&t=905s
- malshe 6y ago
  That's great, thanks!
ViralBShah 6y ago
See the data science course by Huda Nassar on JuliaAcademy.com.
- malshe 6y ago
  Thanks! Edit: This looks like a fantastic resource! Full link: https://juliaacademy.com https://juliaacademy.com

ablekh 6y ago

While being a big fan of Julia in general and recognizing a very significant progress in both language and ecosystem development [kudos to Julia's core team as well as numerous contributors to the language and its growing ecosystem!], I respectfully disagree with the post's author conclusion in its entirety. For scientific computing and advanced machine learning domains, Julia is certainly a breath of fresh air and a no-brainer decision. However, as a general-purpose language as well as technology stack, I think Julia has still a long way to go. This is mostly due to relative immaturity of that part of the ecosystem, spotty - and sometimes even non-existent! - documentation (of course, except for the language itself and quite a limited number of core and popular packages) as well as some other issues, including tooling, development/compilation performance and limited pool of skilled developers. So, from a startup founder perspective [who has to select the optimal, risk-minimizing, platform stack], despite Julia's many attractive features (including powerful meta-programming facilities - in my case, for potential DSL development), I'm now leaning toward Python and .NET ecosystems. Both offer very mature and large package ecosystems along with comprehensive tooling support and incomparably larger pool of skilled developers. Additionally, .NET offers stability of consistent development improvements, backing of a major corporation [no acquisition risk] along with support for modern enterprise-focused architectural practices and patterns, including DDD and multi-tenancy.

avasthe 6y ago
This. Not trolling. I don't really know why people like to shoehorn a purpose-oriented language into all domains. Eg: Julia for scientific computing, Rust for low level programming. They would be great if they focused on those "strong zones".
- ddragon 6y ago
  I read this article not as a "Julia is ready to replace the all-purpose languages used in business", but as "Julia is ready to deploy it's scientific computing into production environments" (as opposed to just local/hobby tasks and academic environments which were the early focus). No one is recommending people to do their small e-commerce in Julia (or perhaps Rust) unless you're in for the fun of the languages (or your e-commerce happens to be a special case that will play with either language's strengths). But having a solid web deployment story is important to both regardless, be it in Julia to have live dashboards or integrating your scientific models in your microservice architecture and to serve them directly to clients or Rust for embedded web servers for IoT and other devices. For a language to be great in one domain, it has to be at least good in everything around it for when you need to connect that domain to the real world. And Julia creators do focus on it's strong zone, with all the works on TPU, HPC and parallel computing, and so does the community with the stuff presented in juliacon like interactive reproducible notebooks (Pluto.jl) or data dashboards (Stipple.jl), both using the web domain to improve the scientific computing domain.

xiaodai 6y ago

Does Julia webstack have middle-ware like authenticators etc etc?

minetest2048 6y ago

Julia feels like a good replacement for Matlab, but then Matlab have Simulink, is there any project to create something like Simulink but for Julia?

ChrisRackauckas 6y ago
Something ModelingToolkit.jl, JuSDL.jl, NetworkDynamics.jl, Modia.jl, etc. all are extensions of Simulink in some form. Supporting stochastic equations, delays, automatic parallelization, acausal modeling, etc.

vasili111 6y ago

Does the Julia matrix has named columns as R or unnamed as numpy?

ddragon 6y ago
The built-in array (which is multidimensional) is unnamed, but Julia was made in a way that anyone can create as a library their own types that are as high performance as the native ones (most of which are written in Julia) so you can just get a named array package like [1]. [1] https://github.com/davidavdav/NamedArrays.jl https://github.com/davidavdav/NamedArrays.jl