21 ms·

The Accelerating Adoption of Julia

316 points by chmaynard 6y ago

deleted 6y ago
[deleted]
7thaccount 6y ago
Julia is pretty neat, but I'm still using Python at the moment for numerical computing work. Julia does have built-in sparse arrays, but I don't think it's on par with scypy.sparse yet.
- adamnemecek 6y ago
  The nice thing about julia is that its much simpler to roll your own things. With numpy, you might have to be writing C.
  - andi999 6y ago
    But then writing something in C is not really difficult isnt it? (The real difficult part is the algorithm not the implementation in a language)
    - adamnemecek 6y ago
      That really depends but on average it will be more difficult and there will be a lot of boiler plate.
    - stabbles 6y ago
      I think you're overestimating how many Python developers are willing to write and compile C. Debugging gets more difficult across two languages. Shipping binaries with a Python package correctly can be a hassle too (what about -march=native performance).
      - marmaduke 6y ago
        Numba compiles Python to LLVM as a backend, why is this even a discussion?
        krastanov 6y ago
        Because numba (and jax), although being wonderful pieces of engineering, work on a fairly limited subset of python, do not permit the same level of introspection as python, and do not play nice with other python libraries. Julia does not make that sacrifice.
        marmaduke 6y ago
        > do not play nice with other python libraries. Julia hasn't been around long enough to build ecosystem with multiple commerical giants building competing products. > Julia does not make that sacrifice. Sure, Julia sacrifices your sanity on the alter of stack traces, method resolution, time to first plot, among others. > fairly limited subset of python Fantastic comment, always brought up, except that this same subset of Python (really how to use CPU efficiently) is about the same that anyone writing about Julia performance is preaching.
        DNF2 6y ago
        I beg your pardon. What parts of Julia are not supported by its jit compiler?
        krastanov 6y ago
        I think a really good example of how Julian tools are composable and work with the whole of the language, while that is not the case with Pythonic tools, is autodifferentiation. The many different julian autodiff libraries have little trouble performing the autodiff through special functions, weird data structures, and 3rd party libraries. In python, none of the autodiff tools play nice with scipy or with mpmath or with sympy (or qutip, which is something I need). This is not about commercial competitors either - all of the libraries I mentioned are non-commercial. Julia provides the interoperability between such libraries, including custom datatypes, frequently with fast zero-cost abstractions.
    - StefanKarpinski 6y ago
      I'm a pretty competent C programmer, but doing even fairly simple matrix programming in C is fiendishly hard not to screw up. When I was porting this code [1] to this [2], it took an absurdly long time to iron out all the bugs. And just look at the difference in verbosity and difficulty. Worst of all, C has no guard rails: it's so easy to think you have it right, only to realize much later (or not at all, frankly) that you're computing complete nonsense. [1] https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3b604c47aa0da767cc97bbea9f64151/perf.jl#L128-L148 https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3... [2] https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3b604c47aa0da767cc97bbea9f64151/perf.c#L135-L209 https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3...
      - andi999 6y ago
        Thanks for the example. Could you also tell how much the speedup of this led to?
        StefanKarpinski 6y ago
        That C code is 39% faster than the Julia version, so faster but not massively so. You can optimize the Julia code to be as fast as the C code by using in-place operations, but it's probably not worth if for such a small speedup unless the code is a real bottleneck. By comparison, the Python version (using NumPy, of course) is 25x slower than C, which is a whole different situation. Most the speed advantage of the C here is due to reusing the same memory over and over, which you can do pretty easily in Julia as well. Here's the Julia code using in-place operations and it's still quite a bit more readable than the C version: https://gist.github.com/StefanKarpinski/e57f5a36b7890b261a0d42db4eb1324c https://gist.github.com/StefanKarpinski/e57f5a36b7890b261a0d.... I timed this and it's the same speed as the C version. When developing this, it follows the same general outline as the C version, but you have several benefits: (1) You can use asserts to compare to the easy version; (2) There are niceties like bounds checks and array indexing. In C you can't do (1) because there is on easy version to compare with. And doing the array index computations in C is kind of a nightmare—it's so easy to accidentally screw them up.
  - coliveira 6y ago
    Just the opposite, because more people know C than Julia.
    - adamnemecek 6y ago
      They do, what's your point.
      - coliveira 6y ago
        My point is that there are far more C programmers and C code in the world than there are Julia programmers and Julia code. This is the advantage of Python: it was able to leverage billions of lines of C code by simply acting as a glue language.
        eigenspace 6y ago
        Just fyi, Julia has a fantastic and very fast C foreign function interface, just like Python.
        DNF2 6y ago
        But I don't know C, and it's easier for me to be able to write fast code in a language I actually know. Having to switch to a language you don't know is not a good situation, it doesn't help that a lot of other people know that other language.
    - snicker7 6y ago
      A Julia user knows more about Julia than a Python user knows about C.
    - dunefox 6y ago
      So more people knowing C than Julia somehow makes it easier for me? Absolute nonsense.
  - marmaduke 6y ago
    > With numpy, you might have to be writing Python FTFY cf Numba http://numba.pydata.org http://numba.pydata.org
    - krastanov 6y ago
      Numba is great (as well as Jax, which is even better), but its limitations are drastic compared to Julia. So much of what is great about python (e.g., compossibility, introspection, simplicity) is lost with these libraries, while Julia still retains them.
      - marmaduke 6y ago
        The main thing missing from Numba is user defined structures, the jit classes are a performance penalty usually. That said, in terms of composability you can jit over the closure to achieve a lot of what you might want, e.g. def make_loop(f): @jit def fn(x): for i in range(x.shape[0]): x[i] = f(x[i]) return fn for any jit'd function f will be just as fast as if you had inlined the body of f. For introspection and simplicity, I think, in high performance, you simply have to choose two of fast, simple and generic. Julia clearly chooses fast and generic.
        krastanov 6y ago
        Could you elaborate on why you are saying julia is not as simple? Certainly, there are performance tips one needs to be aware of, but that is the case with python too. I have used python for relatively high-performance numerics for research work for the last decade, and now that I am exploring Julia, it certainly addresses all of the concerns I mentioned above with code which is at least as legible/simple and much more introspective. It has its warts, but those warts are on the roadmap and I have seen significant improvement between v1.3 and v1.6 (e.g. compilation latency and debuggability).
        marmaduke 6y ago
        Compiler errors can be on par with heavily templates C++, with similarly difficult to read code bases. It’s of course easier to work with than C++ but not the panacea that seems to be implied in a lot of the comments. In terms of deployment, you essentially have to have Julia installed wherever you want to run, which frequently ok, and if not there’s always PackageCompiler except it’s not that easy to get a working shared lib. There are other things I find complex but probably because I’ve used Python too long so Julia is different etc. I think Julia is a great choice for HPC generally except for challenging SIMD codes where it’s hard to vectorize without an explicit ILP model.
- stabbles 6y ago
  I would assume scipy uses MKL, and Julia has wrappers for that too: https://github.com/JuliaSparse/MKLSparse.jl https://github.com/JuliaSparse/MKLSparse.jl. For the GPU CUDA.jl has support for sparse matrices as well.
  - 7thaccount 6y ago
    My only problem too is I just want to write high level mathematical code and have it work. With Python's anaconda distribution, I just download it and the sparse stuff works. I don't want the hassle of dealing with wrappers and code that is not very mature. I know it's a chicken and the egg thing.
    - stabbles 6y ago
      Have you even read the readme in that link? Start a julia repl, hit ] for the package manager, paste `add MKLSparse`, write `using SparseArrays, MKLSparse`, run `sprand(100,100, 0.1) * rand(100)`
- spacedome 6y ago
  I think the functionality of Julia's sparse arrays is mostly on par with scipy.sparse, it's just a bit rough around some edges still, and spread out into non-base packages (e.g. into IterativeSolvers.jl, ARPACK.jl, MUMPS(*).jl etc) of various maturity. Having the sparse arrays built-in might (hopefully) lead to a large ecosystem of interoperable sparse packages, but as with much of the Julia ecosystem this is still a work in progress. Personally, working on sparse eigensolvers, my only complaint is that IterativeSolvers.jl is still somewhat lacking (though the ARPACK.jl bindings are more comparable to what scipy does), but otherwise cannot imagine going back to python for this kind of low-level numerical research. Having code run comparably to fortran speed (barring some memory annoyances) is huge for working on numerical algorithms.
marmaduke 6y ago
Julia looks great but in practice I can pretend it doesn’t exist without much worry. Python and MATLAB are good enough for interactive work and for the performance stuff we use CUDA or similar directly. Edit just to mention that Numba solves by and large the middle ground for the domain I work in, despite what Julia proponents have to say.
- bachmeier 6y ago
  > Julia looks great but in practice I can pretend it doesn’t exist without much worry. That's always the problem with a new programming language. The target for the language is the people that already have something that works. Then a decade passes and that language is no longer a shiny new object, so enthusiasm for it dies, it's just another old language with warts.
  - teorema 6y ago
    Not to sound dismissive but I remember when that was true of Python (vis-a-vis Perl and some other things). My recollection of discussions on forums when it was starting to gain mindshare were arguments largely about the aesthetics of the syntax. Numerical computing is something where there's always been something that works. Before Python it was C and Fortran. The problem people ran into is that when everything is wrapped around low-level libraries for speed, you eventually run into the catch 22 of using the slower language that you prefer for clarity, or the faster one that makes it acceptable performance-wise. In other words, with Python, to get the performance you would have got in C or Fortran, you have to code in C or Fortran. Then you're not using Python anymore. The idea (in theory, and a lot in practice) with Julia (or Nim, or other LLVM-targeting languages) is that you don't have this penalty. So in that sense Julia is providing something that isn't working in Python or Matlab. I think Julia's not quite what it's cracked up to be, but mostly it is, and I'd probably prefer working with it over Python or Matlab for numerical stuff.
    - bachmeier 6y ago
      > I remember when that was true of Python But Python didn't take off because of scientific computing. Google was a heavy user of Python from the start. Python seriously sucked for scientific computing in 2005. I was looking for a new language (the old one wasn't cutting it) and I'd been looking for a use for Python for years. It had some projects but it just was not in a useful state so I went with R, which was quite mature by that time, and it focused on statistical analysis. Years later Python became popular for scientific computing, likely due to the userbase it had built up in other areas.
outlace 6y ago
Julia as a whole still lags behind Python or Matlab (unsurprising given the age difference) but there are some packages in Julia that just steal the show compared to Python et al, such as the SciML packages and the probabilistic programming packages. I switched to Julia for Turing.jl when I was previously using Pyro. Much happier with Julia.
- adamnemecek 6y ago
  The best thing about julia is the interop with essentially any language you can name. Want to call numpy? using PyCall np = pyimport("numpy") res = np.fft.fft(rand(ComplexF64, 10)) The interop with matlab and cpp is similarly painless. https://github.com/JuliaInterop/Cxx.jl https://github.com/JuliaInterop/Cxx.jl
  - ed25519FUUU 6y ago
    Why interop with a language that already solves the problems? Just use that language.
    - adamnemecek 6y ago
      Because julia is a better language. Numpy is written in C, why use python when you can use C?
    - Enginerrrd 6y ago
      Because it's a PITA sometimes to use c++. It's easy to accidentally go out of bounds on stuff. I've arguably got a lot more experience doing scientific programming in c++ and FORTAN, than I do in python. Yet for quick prototyping where performance is almost never an issue, I still elect to use python because it lets me wrap my head around the problem faster. I spend less time fixing stupid errors and thinking about coding and more problem thinking about the problem I really need to solve. Julia might be even better, I've heard good things, I've just never used it though. In the scientific computing and data science world that I deal with, I'd say at least 95% of the time there's never a need to jump from a hacked together prototype to a more production level product.
tgb 6y ago
Can the DifferentialEquations package actually solve quaternion equations correctly? Quaternions are non-commutative and I'd hazard to guess that some solver methods would assume commutativity at some point.
- spacedome 6y ago
  My experience with various linear solvers (which a diffeq solver typicaly relies on) is that if they assume commutivity anywhere at all, which they often do, they almost definitely will not work for quaternions. Even if a derivation of the algorithm can be done with non-commutivity in mind, the implementation typically is not. Many NLA solvers are based on orthogonality transforms, which do not translate directly to quaternions, and even solvers which use only inner products and mat-vec multipication like BiCG do not really work as-is for quaternion matrices. Luckily dense linear solves with LU are fine! One can usually use the complex matrix expansion for quaternions and solve that instead without much difficulty though, and for computing eigenvalues this is perhaps even preferable, as it gives a canonical representation. I have not tried to use DifferentialEquations.jl for quaternion problems, so maybe they have figured some of it out, but it is non-trivial.
  - ChrisRackauckas 6y ago
    ODE solvers only use LU
    - spacedome 6y ago
      This is good to know, thanks Chris. I mostly solve sparse PDEs, being able to always use LU makes everything much simpler, especially with quaternions.
- LolWolf 6y ago
  I would say it's probably worth converting your problem to its linear-algebraic representation and then using the solvers on that. They're quite fast there, and it's probably fairly easy to make a representation map and its inverse work well. (I will say that I don't usually deal with quaternions other than in a linear-algebraic form when doing some robotics stuff, so maybe this is more of a pain than warranted!)
dragontamer 6y ago
The important part of Julia is its programming model. The implicitly parallel fork-join model is easy to program and incredibly general. And I'm glad to see a high-level language embrace it. ------- I probably should note some other languages of this model: CUDA, OpenCL, ISPC, OpenMP, OpenACC. Most of these other languages are low-level, where you manage memory directly (and GPUs have very little memory per thread. So manual memory management is still hugely important) But for speed of development, prototyping, and higher level reasons, a language like Julia that implements this "mindset" is going to be hugely important moving forward. ------ The fact that parallel fork-join scales from 2 CPU threads all the way up to thousands of GPU-threads... or CPU + SIMD (such as AVX512), is proof that this methodology is useful. I feel like people are sleeping on this model: its hugely useful for scaling on what I believe to be the computer of the future.
- marmaduke 6y ago
  Classing for example ISPC as low level invalidates this comment for me; ISPC among others provides a great interface to SIMD model for CPUs that simply isn’t available in Julia.
  - dragontamer 6y ago
    ISPC is higher level than its peers, but its still new/delete based manual memory allocation. So in the great scheme of programming languages, its still rather low level (since you're manually handling memory). Indeed: ISPC provides constructs to provide structure-of-arrays, and other low-level memory layout details. This is a good thing: these details have significant implications on the speed of your program. Nonetheless, any language which wrangles with manual details of memory layout, or new/delete based memory allocation, is inevitably going to be classified as low level in my books. > ISPC among others provides a great interface to SIMD model for CPUs that simply isn’t available in Julia. If Julia can compile into GPU-assembly (which is innately SIMD), I'm sure an AVX-based compile could work eventually. They may have to target AVX512 (since most GPU-assembly requires per-thread exec-masks), but the general concept is being proven as Julia can now compile down into PTX or AMDGPU assembly. Julia's compile down to GPU-assembly / SIMD code is not supported in the "general case", only in select circumstances. But that's still an incredible boon for a high-level language.
ed25519FUUU 6y ago
Julia is an interesting language. I see that there's a lot of excitement for it, but most excitement seems to come from data scientists who are heavier on the engineer side. For almost all of the rest of the data scientists I know, Python seems "more than good enough". Also to note, Julia has seen a rise over the last few years, but so has Python, its primary competitor in the data science field.
- qaq 6y ago
  I am pretty excited about Julia potential for web apps
  - dnautics 6y ago
    I wouldn't be. I love julia, but it doesn't have a spectacular error-handling story. So if you have to start running a web server, and also have to worry about people issuing a DDOS or finding sneaky ways to cause your system to consume a ton of resources on a handful of threads, and lower availability, it's not going to get pretty. Unless you mean compling julia or something like it to WASM and running it in the browser. That could get interesting, but julia is still "a bit too chonky" for that.
    - ddragon 6y ago
      To be fair you wouldn't just expose your server directly nowadays, you'd have them in a container orchestrator like k8s that would have a liveness probe (and container usage tracking) that would appropriately scale/restart pods as to not have the system going down, together with load balancers and DDoS protection systems like AWS Shield to detect such attempts. And of course, you can also use a more low latency high availability oriented language handling the frontend (like Elixir, which will have trouble doing the heavy data oriented stuff Julia can), with Julia as a microservice to handle the actual backend logic and analytics, which is probably the setup I'd go for (using 2 high level lispy-languages with very different qualities, each doing what it does best).
      - 6y ago
melling 6y ago
There’s a current Fall 2020 class at MIT that uses Julia. https://computationalthinking.mit.edu/Fall20/ https://computationalthinking.mit.edu/Fall20/ There’s a live video on ray tracing starting in about 20 minutes https://youtu.be/MkkZb5V6HqM https://youtu.be/MkkZb5V6HqM
- threatofrain 6y ago
  It's notable that the 3B1B guy is contributing to that class.
choeger 6y ago
Finally, Lisp wins ;).
- masijo 6y ago
  What do you mean?
  - ampdepolymerase 6y ago
    Julia has a heavy Lisp heritage. If you want a clearer comparison, see Dylan. On the topic of Lisp, Julia's parser is written in a custom version of Scheme called femtolisp.
    - gnufx 6y ago
      And Dylan is at https://opendylan.org/ https://opendylan.org/ I never really forgave it losing s-expressions, but it did pioneer hygienic, referentially transparent macros with infix syntax.
  - gnufx 6y ago
    Perhaps, as well as Dylan, it's worth referencing another Lisp system, T, that pioneered fast dynamic language implementations on the typical Unix workstation-type hardware of the time, and was of interest for research computing: https://en.wikipedia.org/wiki/T_(programming_language) https://en.wikipedia.org/wiki/T_(programming_language) (T has some historical interest just as the original implementation language of the Yale Haskell compiler.)
- nemoniac 6y ago
  It's not finally. It's more like, here we go again. Ideas from Lisp have been creeping into "mainstream" programming languages for decades. As the article points out, Common Lisp introduced multiple dispatch over 40 years ago and now it's finding its way into Julia. There's lots more in Lisp that can still find its way into a modern programming language. I'm particularly interested in the inclusion of a LispSyntax for Julia that might allow Lisp macros to be written at the LispSyntax level that can then be used at the surface Julia level.
datanecdote 6y ago
When I look at google trends or redmonk rankings, Julia appears stable, not accelerating.
- DNF2 6y ago
  I don't understand google trends. Julia's progress doesn't look like much there, but even python has had pretty modest progress there, which doesn't make sense.
aborsy 6y ago
Julia is better suited to numerical computation. Matlab is expensive and not really a language. Python is not designed for numerical computation and can be verbose. The issue for Julia is, if you stay in Numpy and Scipy in Python, and can use JIT when applicable, you are basically using C. You can make it fast using C libraries and there is not whole lot of room for Julia to shine. Even with C, I had to be lately a bit careful to beat Numpy.
- krastanov 6y ago
  I have quite the opposite experience. Numpy is terrible in terms of allocation overhead (even when you use the available, but limited, inplace operators and memory views). In Julia it is 𝚝̶𝚛̶𝚒̶𝚟̶𝚒̶𝚊̶𝚕̶ much easier to write allocation free code. And only so many operations are trivially broadcastable, after which numpy becomes very cumbersome and slow. Numpy is a great piece of engineering, but its limitations are very noticeable when compared to Julia. edit: not trivial, but much easier
  - spacedome 6y ago
    As much as I like Julia, I think "trivial to write allocation free code" is a bit of an overstatement. Depending on what you are doing, it can be difficult, for example iteratively calling any of the LinearAlgebra methods, since there is no interface for preallocating work arrays (doing the ccall on BLAS yourself is not a fun work-around). It is also not always clear why something is allocating, even with the debugging tools.
    - DNF2 6y ago
      Perhaps I misunderstand what a 'work array' is, but you can easily pre-allocate arrays, and then work on them in-place, including with LinearAlgebra functions like `mul!` et. al.
      - spacedome 6y ago
        Higher level BLAS operations, such as solving a linear system, or computing svd/eigen, cannot be done in-place the way matrix multiplication can, and require additional memory of a predetermined, fixed size, called the work array. This cannot be pre-allocated in Julia, as there is no interface to do so in LinearAlgebra, so these BLAS calls will always allocate memory.
zemo 6y ago
so has anyone tried making games with this yet or no
- eigenspace 6y ago
  Yes.
soVeryTired 6y ago
Anyone have thoughts on the ecosystem around Julia? Suppose you want to hook it into a database, or implement production-grade logging, or hook into e.g. Spark. Are those things do-able these days? I've heard from some colleagues that Julia as a language is pretty solid these days, but all the machinery that you might need to take a system to production _aside_ from the business logic is a little lacking. I'd be interested to hear if others agree!
- oxinabox 6y ago
  My employer (Invenia) runs Memento.jl (a logging package) in production, hooking in variously to SumoLogic and AWS Cloudwatch and from their into PagerDuty. I would say it is production grade logging. and it is integrated enough with standard lib logging that stuff logged using that also ends up at the right place. I don't think it would actually be that hard to get the stdlib logging to work directly with that kind of system either. (Probably if we were starding today we might have built ontop of the stdlib logging, but Memento is a bit older than that, and a bit different in philosophy) THere are a bunch of database drivers. We use LibPQ.jl in production. I hear good things also about OCDB.jl, and SQLite.jl also. and anything to do with data is really nice because of how interoperable all the data-frames like packges are. Everything just works together as long as the minimal API described by Tables.jl is implemented. (and everything implements that)
- quixoticelixer- 6y ago
  It depends. There are quite a lot of really good libraries for things like that but there are still a few important things missing.
diarrhea 6y ago
I don't have a CS background, so maybe that's how I'm missing the point here, but I don't see what is so supposed to be so special about Julia's multiple dispatch. In fact, reading more about it and knowing Python, single dispatch/polymorphism feels more natural to me. This would allow a `plot` function in Python, originally written for the float data type, to also work for float-like (quacks like a duck) data types, like `Decimal`. I struggle to see what about Julia improves on this situation or makes it conceptually different. Lastly, the article mentions that `plot` also works for uncertainty-carrying floats, because the `Measurements` package has implemented "a simple recipe" for this case. But doesn't this mean very tight coupling? The `Measurements` author very specifically (if I understand this right) implemented a `plot` functionality. For the end user, this allows things to just work, which is great. But under the hood, it seems this depends on package authors having to be aware of all possible use cases for their library code.
- pdeffebach 6y ago
  > This would allow a `plot` function in Python, originally written for the float data type, to also work for float-like (quacks like a duck) data types, like `Decimal`. This might come with sacrificed performance. If you want a different implementation for a different data type to get the best performance, writing one method makes that hard. > The `Measurements` author very specifically (if I understand this right) implemented a `plot` functionality. This is a good point. But in that example, note that `Measurements` also works with all of differential equations, and they didn't implement any differential equations specific code, unlike as you pointed out with `plot`. The fact that uncertainty propagates through a differential equation solver is pretty impressive, imo.
  - jschwartzi 6y ago
    Yeah as I was reading this it occurred to me that a startling number of lab reports I wrote as an undergrad would have been much easier to do in Julia. We normally used Excel for the calculations and had to check all of our propagation of errors by hand.
    - 6y ago
FranzFerdiNaN 6y ago
Julia is a wonderfully designed language and I fully anticipate making it my main language in a few years. Right now I will stick with R though, because R’s ecosystem is simply unparalleled right now. And for someone who is not terrible interested in programming, except as a way to get things done, that is what matters most. But i really think that in a few years time, when Julia’s ecosystem has had more time to grow and develop it can rival R.
- eigenspace 6y ago
  > And for someone who is not terrible interested in programming, except as a way to get things done, that is what matters most. For what it's worth, I found that taking the plunge into julia is what made me love programming. I'm constantly amazed by how easy it is to whip something up myself that before I would have relied on a built-in routine in another language. Doing this made me a better programmer and more capable of doing the things that 'matter most' as well. Plus, whenever I whip something up myself, I try to share it with the community either as it's own package or as a PR to an existing package, so I get to feel involved in the progress as we get to the sort of ecosystem you would find acceptable.
- dunefox 6y ago
  Why wait? https://github.com/JuliaInterop/RCall.jl https://github.com/JuliaInterop/RCall.jl
unnouinceput 6y ago
Searching for Julia related work on Upwork resulted in only 3 jobs. One of them was actually to port an existing application from Julia to C++
jrevels 6y ago
I think one of Julia's greatest indicators of long-term success is the variety of commercial users/companies from a diverse pool of technical domains/industries that are all excited, willing, and capable of contributing back to the ecosystem. We had a BoF at this year's JuliaCon revolving around this topic [1] and are now planning the first Annual Industry Julia Users Contributhon as a result. I especially think that well-configured Julia + K8s setups have the capacity to really tighten exploratory data science <-> operational data engineering feedback loops in industrial settings in a way that is much more ergonomic, generically useful, and portable/extensible than using pre-baked frameworks to achieve something similar. Julia-centric tooling for coarse-grained workflow orchestration, experiment tracking, data provenance, etc. would be nice, though I also think existing generic tools in this vein (e.g. Argo) could probably compose well too :) A few different companies have nice in-house implementations of these kinds of setups, and Julia Computing is building a nice looking commercial product suite in this vein that I look forward to exploring more in the future (especially JuliaHub and JuliaRun). [1] https://julialang.org/blog/2020/09/juliacon-2020-open-source-bof-follow-up/ https://julialang.org/blog/2020/09/juliacon-2020-open-source...
sida 6y ago
Honestly, I just wish that this was 0-indexed. 1-indexing annoys me enough that I haven't really started playing with Julia.
- throwaway889900 6y ago
  There's nothing preventing you from using a 0-base, it's just extra work to implement. You can even use the fabled 2-base index! https://docs.julialang.org/en/v1/devdocs/offset-arrays/ https://docs.julialang.org/en/v1/devdocs/offset-arrays/
  - henearkr 6y ago
    For people complaining about 1-indexing, it is precisely because they wish what they feel the "natural" way to count would come without overhead. So it's this "extra work" that is the problem. It would be nice to have it as an environment variable or a runtime option. Then what about portability issues... So let's just fork Julia and call it JuliB with 0-indexing =)
- socialdemocrat 6y ago
  I don't get why people care about this. How often does that hit you in normal code? It never bothered me in Lua, and certainly not in Julia. For me I just do a context switch. I know when I am in Julia I have to think about things like in my mathematics books. Just like you index a matrix. When I am in other languages like C/C++ I tell myself that I working with memory addresses. For memory addresses 0-index makes more sense. For tables, matrices etc I honestly think 1-based indexing makes more sense. Only time I dislike 1-based indexing in Julia is when I write code which is strongly related to how memory works. Like when I was implementing a CPU simulator and assembler.
  - henearkr 6y ago
    It depends on which kind of maths you refer to. After studying ordinal numbers, I feel like the normal way to start counting is from 0 (or the "empty set" ordinal). This is also how you prove something by transfinite induction, you need to start by 0. I wish I could easily recompile Julia so that it is 0-indexed. But then what about all the libraries...
ahurmazda 6y ago
Pet curiosity ... if the inventors would humour me. What was the thinking behind adopting the `end` notation for marking the end of blocks. I know that certain family of PLs (Erlang, Ruby) use them. But the dominant flavor is the `{}` syntax (c/c++/scala/rust etc) Was it an explicit choice or more based on your past familiarity with other languages?
- newen 6y ago
  It’s from Matlab. That’s really it. Well, and probably {} are perfectly good characters that shouldn’t be wasted on code blocks.
  - patal 6y ago
    So it's from Fortran ;)
- eigenspace 6y ago
  The reason is that they want to reserve the curly braces for other uses and the language devs call using `{}` to delimit code a 'waste of valuable ascii characters'. Julia currently uses {} mostly for type systemy things like parametric types. When I write Foo{T}, this says that T is a parameter in the type Foo. E.g. a vector of Float64s in julia is an Array{Float64, 1} (1 means one dimensional) whereas a matrix of vectors of complex Ints is an Array{Array{Complex{Int}, 1}, 2}.
lenkite 6y ago
Please allow Julia tooling to easily compile to a native binary/dynlib/dll that can be deployed on both desktop and mobile OS'es. Right now, it appears like a niche interpreted language for numeric geeks only.
- vsskanth 6y ago
  https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl Note that it doesn't do slim binaries yet for embedded. Don't know if that's in the roadmap.
- edsac_xyzw 6y ago
  Matlab and its Simulink extension, which is widely used for designing control system has a feature for generating C code that can be deployed without Matlab runtime. Someone with knowledge of code generation and compilers could take this Matlab feature as an inspiration and implement a C code generator for Julia functions. > Right now, it appears like a niche interpreted language for numeric geeks only. Yes, it was designed for numerical and scientific computing. In Julia, you can at least prototype the numerical algorithm before implementing in C++ or Fortran.

316 points by chmaynard 6y ago

deleted 6y ago

[deleted]

Julia is pretty neat, but I'm still using Python at the moment for numerical computing work. Julia does have built-in sparse arrays, but I don't think it's on par with scypy.sparse yet.

adamnemecek 6y ago
The nice thing about julia is that its much simpler to roll your own things. With numpy, you might have to be writing C.
- andi999 6y ago
  But then writing something in C is not really difficult isnt it? (The real difficult part is the algorithm not the implementation in a language)
  - adamnemecek 6y ago
    That really depends but on average it will be more difficult and there will be a lot of boiler plate.
  - stabbles 6y ago
    I think you're overestimating how many Python developers are willing to write and compile C. Debugging gets more difficult across two languages. Shipping binaries with a Python package correctly can be a hassle too (what about -march=native performance).
    - marmaduke 6y ago
      Numba compiles Python to LLVM as a backend, why is this even a discussion?
      - krastanov 6y ago
        Because numba (and jax), although being wonderful pieces of engineering, work on a fairly limited subset of python, do not permit the same level of introspection as python, and do not play nice with other python libraries. Julia does not make that sacrifice.
        marmaduke 6y ago
        > do not play nice with other python libraries. Julia hasn't been around long enough to build ecosystem with multiple commerical giants building competing products. > Julia does not make that sacrifice. Sure, Julia sacrifices your sanity on the alter of stack traces, method resolution, time to first plot, among others. > fairly limited subset of python Fantastic comment, always brought up, except that this same subset of Python (really how to use CPU efficiently) is about the same that anyone writing about Julia performance is preaching.
        DNF2 6y ago
        I beg your pardon. What parts of Julia are not supported by its jit compiler?
        krastanov 6y ago
        I think a really good example of how Julian tools are composable and work with the whole of the language, while that is not the case with Pythonic tools, is autodifferentiation. The many different julian autodiff libraries have little trouble performing the autodiff through special functions, weird data structures, and 3rd party libraries. In python, none of the autodiff tools play nice with scipy or with mpmath or with sympy (or qutip, which is something I need). This is not about commercial competitors either - all of the libraries I mentioned are non-commercial. Julia provides the interoperability between such libraries, including custom datatypes, frequently with fast zero-cost abstractions.
  - StefanKarpinski 6y ago
    I'm a pretty competent C programmer, but doing even fairly simple matrix programming in C is fiendishly hard not to screw up. When I was porting this code [1] to this [2], it took an absurdly long time to iron out all the bugs. And just look at the difference in verbosity and difficulty. Worst of all, C has no guard rails: it's so easy to think you have it right, only to realize much later (or not at all, frankly) that you're computing complete nonsense. [1] https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3b604c47aa0da767cc97bbea9f64151/perf.jl#L128-L148 https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3... [2] https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3b604c47aa0da767cc97bbea9f64151/perf.c#L135-L209 https://github.com/JuliaLang/Microbenchmarks/blob/af3d18f7b3...
    - andi999 6y ago
      Thanks for the example. Could you also tell how much the speedup of this led to?
      - StefanKarpinski 6y ago
        That C code is 39% faster than the Julia version, so faster but not massively so. You can optimize the Julia code to be as fast as the C code by using in-place operations, but it's probably not worth if for such a small speedup unless the code is a real bottleneck. By comparison, the Python version (using NumPy, of course) is 25x slower than C, which is a whole different situation. Most the speed advantage of the C here is due to reusing the same memory over and over, which you can do pretty easily in Julia as well. Here's the Julia code using in-place operations and it's still quite a bit more readable than the C version: https://gist.github.com/StefanKarpinski/e57f5a36b7890b261a0d42db4eb1324c https://gist.github.com/StefanKarpinski/e57f5a36b7890b261a0d.... I timed this and it's the same speed as the C version. When developing this, it follows the same general outline as the C version, but you have several benefits: (1) You can use asserts to compare to the easy version; (2) There are niceties like bounds checks and array indexing. In C you can't do (1) because there is on easy version to compare with. And doing the array index computations in C is kind of a nightmare—it's so easy to accidentally screw them up.
- coliveira 6y ago
  Just the opposite, because more people know C than Julia.
  - adamnemecek 6y ago
    They do, what's your point.
    - coliveira 6y ago
      My point is that there are far more C programmers and C code in the world than there are Julia programmers and Julia code. This is the advantage of Python: it was able to leverage billions of lines of C code by simply acting as a glue language.
      - eigenspace 6y ago
        Just fyi, Julia has a fantastic and very fast C foreign function interface, just like Python.
      - DNF2 6y ago
        But I don't know C, and it's easier for me to be able to write fast code in a language I actually know. Having to switch to a language you don't know is not a good situation, it doesn't help that a lot of other people know that other language.
  - snicker7 6y ago
    A Julia user knows more about Julia than a Python user knows about C.
  - dunefox 6y ago
    So more people knowing C than Julia somehow makes it easier for me? Absolute nonsense.
- marmaduke 6y ago
  > With numpy, you might have to be writing Python FTFY cf Numba http://numba.pydata.org http://numba.pydata.org
  - krastanov 6y ago
    Numba is great (as well as Jax, which is even better), but its limitations are drastic compared to Julia. So much of what is great about python (e.g., compossibility, introspection, simplicity) is lost with these libraries, while Julia still retains them.
    - marmaduke 6y ago
      The main thing missing from Numba is user defined structures, the jit classes are a performance penalty usually. That said, in terms of composability you can jit over the closure to achieve a lot of what you might want, e.g. def make_loop(f): @jit def fn(x): for i in range(x.shape[0]): x[i] = f(x[i]) return fn for any jit'd function f will be just as fast as if you had inlined the body of f. For introspection and simplicity, I think, in high performance, you simply have to choose two of fast, simple and generic. Julia clearly chooses fast and generic.
      - krastanov 6y ago
        Could you elaborate on why you are saying julia is not as simple? Certainly, there are performance tips one needs to be aware of, but that is the case with python too. I have used python for relatively high-performance numerics for research work for the last decade, and now that I am exploring Julia, it certainly addresses all of the concerns I mentioned above with code which is at least as legible/simple and much more introspective. It has its warts, but those warts are on the roadmap and I have seen significant improvement between v1.3 and v1.6 (e.g. compilation latency and debuggability).
        marmaduke 6y ago
        Compiler errors can be on par with heavily templates C++, with similarly difficult to read code bases. It’s of course easier to work with than C++ but not the panacea that seems to be implied in a lot of the comments. In terms of deployment, you essentially have to have Julia installed wherever you want to run, which frequently ok, and if not there’s always PackageCompiler except it’s not that easy to get a working shared lib. There are other things I find complex but probably because I’ve used Python too long so Julia is different etc. I think Julia is a great choice for HPC generally except for challenging SIMD codes where it’s hard to vectorize without an explicit ILP model.
stabbles 6y ago
I would assume scipy uses MKL, and Julia has wrappers for that too: https://github.com/JuliaSparse/MKLSparse.jl https://github.com/JuliaSparse/MKLSparse.jl. For the GPU CUDA.jl has support for sparse matrices as well.
- 7thaccount 6y ago
  My only problem too is I just want to write high level mathematical code and have it work. With Python's anaconda distribution, I just download it and the sparse stuff works. I don't want the hassle of dealing with wrappers and code that is not very mature. I know it's a chicken and the egg thing.
  - stabbles 6y ago
    Have you even read the readme in that link? Start a julia repl, hit ] for the package manager, paste `add MKLSparse`, write `using SparseArrays, MKLSparse`, run `sprand(100,100, 0.1) * rand(100)`
spacedome 6y ago
I think the functionality of Julia's sparse arrays is mostly on par with scipy.sparse, it's just a bit rough around some edges still, and spread out into non-base packages (e.g. into IterativeSolvers.jl, ARPACK.jl, MUMPS(*).jl etc) of various maturity. Having the sparse arrays built-in might (hopefully) lead to a large ecosystem of interoperable sparse packages, but as with much of the Julia ecosystem this is still a work in progress. Personally, working on sparse eigensolvers, my only complaint is that IterativeSolvers.jl is still somewhat lacking (though the ARPACK.jl bindings are more comparable to what scipy does), but otherwise cannot imagine going back to python for this kind of low-level numerical research. Having code run comparably to fortran speed (barring some memory annoyances) is huge for working on numerical algorithms.

marmaduke 6y ago

Julia looks great but in practice I can pretend it doesn’t exist without much worry. Python and MATLAB are good enough for interactive work and for the performance stuff we use CUDA or similar directly. Edit just to mention that Numba solves by and large the middle ground for the domain I work in, despite what Julia proponents have to say.

bachmeier 6y ago
> Julia looks great but in practice I can pretend it doesn’t exist without much worry. That's always the problem with a new programming language. The target for the language is the people that already have something that works. Then a decade passes and that language is no longer a shiny new object, so enthusiasm for it dies, it's just another old language with warts.
- teorema 6y ago
  Not to sound dismissive but I remember when that was true of Python (vis-a-vis Perl and some other things). My recollection of discussions on forums when it was starting to gain mindshare were arguments largely about the aesthetics of the syntax. Numerical computing is something where there's always been something that works. Before Python it was C and Fortran. The problem people ran into is that when everything is wrapped around low-level libraries for speed, you eventually run into the catch 22 of using the slower language that you prefer for clarity, or the faster one that makes it acceptable performance-wise. In other words, with Python, to get the performance you would have got in C or Fortran, you have to code in C or Fortran. Then you're not using Python anymore. The idea (in theory, and a lot in practice) with Julia (or Nim, or other LLVM-targeting languages) is that you don't have this penalty. So in that sense Julia is providing something that isn't working in Python or Matlab. I think Julia's not quite what it's cracked up to be, but mostly it is, and I'd probably prefer working with it over Python or Matlab for numerical stuff.
  - bachmeier 6y ago
    > I remember when that was true of Python But Python didn't take off because of scientific computing. Google was a heavy user of Python from the start. Python seriously sucked for scientific computing in 2005. I was looking for a new language (the old one wasn't cutting it) and I'd been looking for a use for Python for years. It had some projects but it just was not in a useful state so I went with R, which was quite mature by that time, and it focused on statistical analysis. Years later Python became popular for scientific computing, likely due to the userbase it had built up in other areas.

outlace 6y ago

Julia as a whole still lags behind Python or Matlab (unsurprising given the age difference) but there are some packages in Julia that just steal the show compared to Python et al, such as the SciML packages and the probabilistic programming packages. I switched to Julia for Turing.jl when I was previously using Pyro. Much happier with Julia.

adamnemecek 6y ago
The best thing about julia is the interop with essentially any language you can name. Want to call numpy? using PyCall np = pyimport("numpy") res = np.fft.fft(rand(ComplexF64, 10)) The interop with matlab and cpp is similarly painless. https://github.com/JuliaInterop/Cxx.jl https://github.com/JuliaInterop/Cxx.jl
- ed25519FUUU 6y ago
  Why interop with a language that already solves the problems? Just use that language.
  - adamnemecek 6y ago
    Because julia is a better language. Numpy is written in C, why use python when you can use C?
  - Enginerrrd 6y ago
    Because it's a PITA sometimes to use c++. It's easy to accidentally go out of bounds on stuff. I've arguably got a lot more experience doing scientific programming in c++ and FORTAN, than I do in python. Yet for quick prototyping where performance is almost never an issue, I still elect to use python because it lets me wrap my head around the problem faster. I spend less time fixing stupid errors and thinking about coding and more problem thinking about the problem I really need to solve. Julia might be even better, I've heard good things, I've just never used it though. In the scientific computing and data science world that I deal with, I'd say at least 95% of the time there's never a need to jump from a hacked together prototype to a more production level product.

tgb 6y ago

Can the DifferentialEquations package actually solve quaternion equations correctly? Quaternions are non-commutative and I'd hazard to guess that some solver methods would assume commutativity at some point.

spacedome 6y ago
My experience with various linear solvers (which a diffeq solver typicaly relies on) is that if they assume commutivity anywhere at all, which they often do, they almost definitely will not work for quaternions. Even if a derivation of the algorithm can be done with non-commutivity in mind, the implementation typically is not. Many NLA solvers are based on orthogonality transforms, which do not translate directly to quaternions, and even solvers which use only inner products and mat-vec multipication like BiCG do not really work as-is for quaternion matrices. Luckily dense linear solves with LU are fine! One can usually use the complex matrix expansion for quaternions and solve that instead without much difficulty though, and for computing eigenvalues this is perhaps even preferable, as it gives a canonical representation. I have not tried to use DifferentialEquations.jl for quaternion problems, so maybe they have figured some of it out, but it is non-trivial.
- ChrisRackauckas 6y ago
  ODE solvers only use LU
  - spacedome 6y ago
    This is good to know, thanks Chris. I mostly solve sparse PDEs, being able to always use LU makes everything much simpler, especially with quaternions.
LolWolf 6y ago
I would say it's probably worth converting your problem to its linear-algebraic representation and then using the solvers on that. They're quite fast there, and it's probably fairly easy to make a representation map and its inverse work well. (I will say that I don't usually deal with quaternions other than in a linear-algebraic form when doing some robotics stuff, so maybe this is more of a pain than warranted!)

dragontamer 6y ago

The important part of Julia is its programming model. The implicitly parallel fork-join model is easy to program and incredibly general. And I'm glad to see a high-level language embrace it. ------- I probably should note some other languages of this model: CUDA, OpenCL, ISPC, OpenMP, OpenACC. Most of these other languages are low-level, where you manage memory directly (and GPUs have very little memory per thread. So manual memory management is still hugely important) But for speed of development, prototyping, and higher level reasons, a language like Julia that implements this "mindset" is going to be hugely important moving forward. ------ The fact that parallel fork-join scales from 2 CPU threads all the way up to thousands of GPU-threads... or CPU + SIMD (such as AVX512), is proof that this methodology is useful. I feel like people are sleeping on this model: its hugely useful for scaling on what I believe to be the computer of the future.

marmaduke 6y ago
Classing for example ISPC as low level invalidates this comment for me; ISPC among others provides a great interface to SIMD model for CPUs that simply isn’t available in Julia.
- dragontamer 6y ago
  ISPC is higher level than its peers, but its still new/delete based manual memory allocation. So in the great scheme of programming languages, its still rather low level (since you're manually handling memory). Indeed: ISPC provides constructs to provide structure-of-arrays, and other low-level memory layout details. This is a good thing: these details have significant implications on the speed of your program. Nonetheless, any language which wrangles with manual details of memory layout, or new/delete based memory allocation, is inevitably going to be classified as low level in my books. > ISPC among others provides a great interface to SIMD model for CPUs that simply isn’t available in Julia. If Julia can compile into GPU-assembly (which is innately SIMD), I'm sure an AVX-based compile could work eventually. They may have to target AVX512 (since most GPU-assembly requires per-thread exec-masks), but the general concept is being proven as Julia can now compile down into PTX or AMDGPU assembly. Julia's compile down to GPU-assembly / SIMD code is not supported in the "general case", only in select circumstances. But that's still an incredible boon for a high-level language.

ed25519FUUU 6y ago

Julia is an interesting language. I see that there's a lot of excitement for it, but most excitement seems to come from data scientists who are heavier on the engineer side. For almost all of the rest of the data scientists I know, Python seems "more than good enough". Also to note, Julia has seen a rise over the last few years, but so has Python, its primary competitor in the data science field.

qaq 6y ago
I am pretty excited about Julia potential for web apps
- dnautics 6y ago
  I wouldn't be. I love julia, but it doesn't have a spectacular error-handling story. So if you have to start running a web server, and also have to worry about people issuing a DDOS or finding sneaky ways to cause your system to consume a ton of resources on a handful of threads, and lower availability, it's not going to get pretty. Unless you mean compling julia or something like it to WASM and running it in the browser. That could get interesting, but julia is still "a bit too chonky" for that.
  - ddragon 6y ago
    To be fair you wouldn't just expose your server directly nowadays, you'd have them in a container orchestrator like k8s that would have a liveness probe (and container usage tracking) that would appropriately scale/restart pods as to not have the system going down, together with load balancers and DDoS protection systems like AWS Shield to detect such attempts. And of course, you can also use a more low latency high availability oriented language handling the frontend (like Elixir, which will have trouble doing the heavy data oriented stuff Julia can), with Julia as a microservice to handle the actual backend logic and analytics, which is probably the setup I'd go for (using 2 high level lispy-languages with very different qualities, each doing what it does best).
    - 6y ago

melling 6y ago

There’s a current Fall 2020 class at MIT that uses Julia. https://computationalthinking.mit.edu/Fall20/ https://computationalthinking.mit.edu/Fall20/ There’s a live video on ray tracing starting in about 20 minutes https://youtu.be/MkkZb5V6HqM https://youtu.be/MkkZb5V6HqM

threatofrain 6y ago
It's notable that the 3B1B guy is contributing to that class.

choeger 6y ago

Finally, Lisp wins ;).

masijo 6y ago
What do you mean?
- ampdepolymerase 6y ago
  Julia has a heavy Lisp heritage. If you want a clearer comparison, see Dylan. On the topic of Lisp, Julia's parser is written in a custom version of Scheme called femtolisp.
  - gnufx 6y ago
    And Dylan is at https://opendylan.org/ https://opendylan.org/ I never really forgave it losing s-expressions, but it did pioneer hygienic, referentially transparent macros with infix syntax.
- gnufx 6y ago
  Perhaps, as well as Dylan, it's worth referencing another Lisp system, T, that pioneered fast dynamic language implementations on the typical Unix workstation-type hardware of the time, and was of interest for research computing: https://en.wikipedia.org/wiki/T_(programming_language) https://en.wikipedia.org/wiki/T_(programming_language) (T has some historical interest just as the original implementation language of the Yale Haskell compiler.)
nemoniac 6y ago
It's not finally. It's more like, here we go again. Ideas from Lisp have been creeping into "mainstream" programming languages for decades. As the article points out, Common Lisp introduced multiple dispatch over 40 years ago and now it's finding its way into Julia. There's lots more in Lisp that can still find its way into a modern programming language. I'm particularly interested in the inclusion of a LispSyntax for Julia that might allow Lisp macros to be written at the LispSyntax level that can then be used at the surface Julia level.

datanecdote 6y ago

When I look at google trends or redmonk rankings, Julia appears stable, not accelerating.

DNF2 6y ago
I don't understand google trends. Julia's progress doesn't look like much there, but even python has had pretty modest progress there, which doesn't make sense.

aborsy 6y ago

Julia is better suited to numerical computation. Matlab is expensive and not really a language. Python is not designed for numerical computation and can be verbose. The issue for Julia is, if you stay in Numpy and Scipy in Python, and can use JIT when applicable, you are basically using C. You can make it fast using C libraries and there is not whole lot of room for Julia to shine. Even with C, I had to be lately a bit careful to beat Numpy.

krastanov 6y ago
I have quite the opposite experience. Numpy is terrible in terms of allocation overhead (even when you use the available, but limited, inplace operators and memory views). In Julia it is 𝚝̶𝚛̶𝚒̶𝚟̶𝚒̶𝚊̶𝚕̶ much easier to write allocation free code. And only so many operations are trivially broadcastable, after which numpy becomes very cumbersome and slow. Numpy is a great piece of engineering, but its limitations are very noticeable when compared to Julia. edit: not trivial, but much easier
- spacedome 6y ago
  As much as I like Julia, I think "trivial to write allocation free code" is a bit of an overstatement. Depending on what you are doing, it can be difficult, for example iteratively calling any of the LinearAlgebra methods, since there is no interface for preallocating work arrays (doing the ccall on BLAS yourself is not a fun work-around). It is also not always clear why something is allocating, even with the debugging tools.
  - DNF2 6y ago
    Perhaps I misunderstand what a 'work array' is, but you can easily pre-allocate arrays, and then work on them in-place, including with LinearAlgebra functions like `mul!` et. al.
    - spacedome 6y ago
      Higher level BLAS operations, such as solving a linear system, or computing svd/eigen, cannot be done in-place the way matrix multiplication can, and require additional memory of a predetermined, fixed size, called the work array. This cannot be pre-allocated in Julia, as there is no interface to do so in LinearAlgebra, so these BLAS calls will always allocate memory.

zemo 6y ago

so has anyone tried making games with this yet or no

eigenspace 6y ago
Yes.

soVeryTired 6y ago

Anyone have thoughts on the ecosystem around Julia? Suppose you want to hook it into a database, or implement production-grade logging, or hook into e.g. Spark. Are those things do-able these days? I've heard from some colleagues that Julia as a language is pretty solid these days, but all the machinery that you might need to take a system to production _aside_ from the business logic is a little lacking. I'd be interested to hear if others agree!

oxinabox 6y ago
My employer (Invenia) runs Memento.jl (a logging package) in production, hooking in variously to SumoLogic and AWS Cloudwatch and from their into PagerDuty. I would say it is production grade logging. and it is integrated enough with standard lib logging that stuff logged using that also ends up at the right place. I don't think it would actually be that hard to get the stdlib logging to work directly with that kind of system either. (Probably if we were starding today we might have built ontop of the stdlib logging, but Memento is a bit older than that, and a bit different in philosophy) THere are a bunch of database drivers. We use LibPQ.jl in production. I hear good things also about OCDB.jl, and SQLite.jl also. and anything to do with data is really nice because of how interoperable all the data-frames like packges are. Everything just works together as long as the minimal API described by Tables.jl is implemented. (and everything implements that)
quixoticelixer- 6y ago
It depends. There are quite a lot of really good libraries for things like that but there are still a few important things missing.

diarrhea 6y ago

I don't have a CS background, so maybe that's how I'm missing the point here, but I don't see what is so supposed to be so special about Julia's multiple dispatch. In fact, reading more about it and knowing Python, single dispatch/polymorphism feels more natural to me. This would allow a `plot` function in Python, originally written for the float data type, to also work for float-like (quacks like a duck) data types, like `Decimal`. I struggle to see what about Julia improves on this situation or makes it conceptually different. Lastly, the article mentions that `plot` also works for uncertainty-carrying floats, because the `Measurements` package has implemented "a simple recipe" for this case. But doesn't this mean very tight coupling? The `Measurements` author very specifically (if I understand this right) implemented a `plot` functionality. For the end user, this allows things to just work, which is great. But under the hood, it seems this depends on package authors having to be aware of all possible use cases for their library code.

pdeffebach 6y ago
> This would allow a `plot` function in Python, originally written for the float data type, to also work for float-like (quacks like a duck) data types, like `Decimal`. This might come with sacrificed performance. If you want a different implementation for a different data type to get the best performance, writing one method makes that hard. > The `Measurements` author very specifically (if I understand this right) implemented a `plot` functionality. This is a good point. But in that example, note that `Measurements` also works with all of differential equations, and they didn't implement any differential equations specific code, unlike as you pointed out with `plot`. The fact that uncertainty propagates through a differential equation solver is pretty impressive, imo.
- jschwartzi 6y ago
  Yeah as I was reading this it occurred to me that a startling number of lab reports I wrote as an undergrad would have been much easier to do in Julia. We normally used Excel for the calculations and had to check all of our propagation of errors by hand.
  - 6y ago

FranzFerdiNaN 6y ago

Julia is a wonderfully designed language and I fully anticipate making it my main language in a few years. Right now I will stick with R though, because R’s ecosystem is simply unparalleled right now. And for someone who is not terrible interested in programming, except as a way to get things done, that is what matters most. But i really think that in a few years time, when Julia’s ecosystem has had more time to grow and develop it can rival R.

eigenspace 6y ago
> And for someone who is not terrible interested in programming, except as a way to get things done, that is what matters most. For what it's worth, I found that taking the plunge into julia is what made me love programming. I'm constantly amazed by how easy it is to whip something up myself that before I would have relied on a built-in routine in another language. Doing this made me a better programmer and more capable of doing the things that 'matter most' as well. Plus, whenever I whip something up myself, I try to share it with the community either as it's own package or as a PR to an existing package, so I get to feel involved in the progress as we get to the sort of ecosystem you would find acceptable.
dunefox 6y ago
Why wait? https://github.com/JuliaInterop/RCall.jl https://github.com/JuliaInterop/RCall.jl

unnouinceput 6y ago

Searching for Julia related work on Upwork resulted in only 3 jobs. One of them was actually to port an existing application from Julia to C++

jrevels 6y ago

I think one of Julia's greatest indicators of long-term success is the variety of commercial users/companies from a diverse pool of technical domains/industries that are all excited, willing, and capable of contributing back to the ecosystem. We had a BoF at this year's JuliaCon revolving around this topic [1] and are now planning the first Annual Industry Julia Users Contributhon as a result. I especially think that well-configured Julia + K8s setups have the capacity to really tighten exploratory data science <-> operational data engineering feedback loops in industrial settings in a way that is much more ergonomic, generically useful, and portable/extensible than using pre-baked frameworks to achieve something similar. Julia-centric tooling for coarse-grained workflow orchestration, experiment tracking, data provenance, etc. would be nice, though I also think existing generic tools in this vein (e.g. Argo) could probably compose well too :) A few different companies have nice in-house implementations of these kinds of setups, and Julia Computing is building a nice looking commercial product suite in this vein that I look forward to exploring more in the future (especially JuliaHub and JuliaRun). [1] https://julialang.org/blog/2020/09/juliacon-2020-open-source-bof-follow-up/ https://julialang.org/blog/2020/09/juliacon-2020-open-source...

sida 6y ago

Honestly, I just wish that this was 0-indexed. 1-indexing annoys me enough that I haven't really started playing with Julia.

throwaway889900 6y ago
There's nothing preventing you from using a 0-base, it's just extra work to implement. You can even use the fabled 2-base index! https://docs.julialang.org/en/v1/devdocs/offset-arrays/ https://docs.julialang.org/en/v1/devdocs/offset-arrays/
- henearkr 6y ago
  For people complaining about 1-indexing, it is precisely because they wish what they feel the "natural" way to count would come without overhead. So it's this "extra work" that is the problem. It would be nice to have it as an environment variable or a runtime option. Then what about portability issues... So let's just fork Julia and call it JuliB with 0-indexing =)
socialdemocrat 6y ago
I don't get why people care about this. How often does that hit you in normal code? It never bothered me in Lua, and certainly not in Julia. For me I just do a context switch. I know when I am in Julia I have to think about things like in my mathematics books. Just like you index a matrix. When I am in other languages like C/C++ I tell myself that I working with memory addresses. For memory addresses 0-index makes more sense. For tables, matrices etc I honestly think 1-based indexing makes more sense. Only time I dislike 1-based indexing in Julia is when I write code which is strongly related to how memory works. Like when I was implementing a CPU simulator and assembler.
- henearkr 6y ago
  It depends on which kind of maths you refer to. After studying ordinal numbers, I feel like the normal way to start counting is from 0 (or the "empty set" ordinal). This is also how you prove something by transfinite induction, you need to start by 0. I wish I could easily recompile Julia so that it is 0-indexed. But then what about all the libraries...

ahurmazda 6y ago

Pet curiosity ... if the inventors would humour me. What was the thinking behind adopting the `end` notation for marking the end of blocks. I know that certain family of PLs (Erlang, Ruby) use them. But the dominant flavor is the `{}` syntax (c/c++/scala/rust etc) Was it an explicit choice or more based on your past familiarity with other languages?

newen 6y ago
It’s from Matlab. That’s really it. Well, and probably {} are perfectly good characters that shouldn’t be wasted on code blocks.
- patal 6y ago
  So it's from Fortran ;)
eigenspace 6y ago
The reason is that they want to reserve the curly braces for other uses and the language devs call using `{}` to delimit code a 'waste of valuable ascii characters'. Julia currently uses {} mostly for type systemy things like parametric types. When I write Foo{T}, this says that T is a parameter in the type Foo. E.g. a vector of Float64s in julia is an Array{Float64, 1} (1 means one dimensional) whereas a matrix of vectors of complex Ints is an Array{Array{Complex{Int}, 1}, 2}.

lenkite 6y ago

Please allow Julia tooling to easily compile to a native binary/dynlib/dll that can be deployed on both desktop and mobile OS'es. Right now, it appears like a niche interpreted language for numeric geeks only.

vsskanth 6y ago
https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl Note that it doesn't do slim binaries yet for embedded. Don't know if that's in the roadmap.
edsac_xyzw 6y ago
Matlab and its Simulink extension, which is widely used for designing control system has a feature for generating C code that can be deployed without Matlab runtime. Someone with knowledge of code generation and compilers could take this Matlab feature as an inspiration and implement a C code generator for Julia functions. > Right now, it appears like a niche interpreted language for numeric geeks only. Yes, it was designed for numerical and scientific computing. In Julia, you can at least prototype the numerical algorithm before implementing in C++ or Fortran.