16 ms·

The unreasonable effectiveness of the Julia programming language

284 points by nikbackm 6y ago

tasubotadas 6y ago
Since everybody is so hyped about Julia, I took on learning it and using it in one of my pet projects. The use case seemed to be perfect for Julia: financial portfolio optimization. But honestly, it is a royal pain in the ass to use. Mostly because tooling is so crappy (no decent IDE). For example, I've made these notes on my journey with Julia: - Juno and VSCode julia is a pain - Debugging is pain - no watch expressions (but debug> works) - Works like with matlab (workspaces, designed for the single script use-cases) - Long registry lookup times (2-5mins) when starting a new session - No hints from IDE (Juno) on what methods are available for which types - Ctrl-click navigation does not work to lookup definitations in libraries - Debugger does not show the full list of elements - there are triple dots in the middle - Filters and maps are not lazy - Arrays start at one - Precompilation times can take 1-2min - Package management is done as a part of script execution - Delay (0.5-1s) when processing commands (compilation?) - Editors (Juno) do not check types at runtime when the information is readily available - Include() includes (is this PHP all over again?) a file into the script! - Reuses same REPL between different files - DataFrames.jl: Transform does nonsense (use map instead) - DataFrames.jl: No transpose Obviously, there is a bunch of good things like freakin insane speed and vectorized method calls are awesome, but I struggle seeing myself becoming more producting than Python + Pandas. It's really a shame that language is strongly and semi-statically typed yet no tooling is making a use of that.
- jackcviers3 6y ago
  To be fair, most of these are ide complaints, not language complaints. Much like all development work, language designers are now required to know much more than how to write an effective static compiler, an effective language syntax, and an effective standard library. A working IDE that integrates easily with all mainstream editors, extensions for all mainstream editors, cross-platform compilation for all major environments, a secure and effective dependency manager, a secure and effective build tool, and incremental compilation are all requirements for a language to get off the ground. Honestly, that seems like a big hurdle for one person or even a dedicated team to manage effectively without major funding outside of what is normally present in research grants. Hosting and curation alone for the dependency package system is a costly problem. Are we as developers painting ourselves into a corner of stagnating pl technology with our expectations that all these things exist in order for us to call a language "good?"
  - tasubotadas 6y ago
    Yes. But only computer scientists are interested only in the language itself. I am an engineer so I have to deal with practical aspects of it.
    - jackcviers3 6y ago
      I'm an engineer. I primarily care about how good the language is at expressing and executing my intent. I've been around long enough to see several tooling fads come and go. I've also been around enough to know that the barrier for delivering quality software is becoming insurmountable, and the additional tooling and infrastructure isn't resulting in projects being more successful, nor being delivered more quickly, nor being less costly than when I started. Teams are enormous now. Software projects are enormous. Deployment requires deep knowledge of multiple hosting platforms and various cloud provider apis at the enterprise level. Available libraries, increasing reliance upon testing, type safety, and better development life cycle practices have had a far larger impact than the ide's I've seen come and go. In my experience, a particular tool has a popularity lifecycle of about ten years. Vscode and lsp will probably be the same. Give me types, good compiler error output, and access to a large universe of libraries with good documentation over relying upon ide api discovery any day of the week and twice on Sundays. The rest is an ever-changing tide of requirements fashion.
      - kuzuman 6y ago
        "I am an engineer" Did you pass the Fundamentals of Engineering exam, are you a member in order of your local association of engineers?, do you have an engineer seal? If you answer no to the questions above please don't call yourself engineer
    - KenoFischer 6y ago
      Give us some more time, there's just so many things to do and the reliance issues favor getting the fundamentals right first (much easier to switch the debugger UI than the names of functions that'll get put all over the place). We certainly understand the value of good tooling, so just stay tuned and maybe try it again every once in a while to see if it's good enough for you yet. Too much to do, too little time :)
- bieganek 6y ago
  > Long registry lookup times (2-5mins) when starting a new session I'm not sure what you are referring to here. Yes, sometimes after you've installed new packages, the Julia VS Code Language Server has to do some re-indexing which can take some time, but they've improved this and you can still program and use the integrated REPL while this is occurring. > Ctrl-click navigation does not work to lookup definitations in libraries In VS Code, F12 (Go to Definition) does work. You can also use `@edit foo(x)` in the REPL. > Filters and maps are not lazy Use Iterators.filter or use generator expressions. > Package management is done as a part of script execution I'm also not sure what this refers to, but the package manager in Julia is one of the best things about Julia. > Editors (Juno) do not check types at runtime when the information is readily available The VS Code Julia extension has a linter that is pretty helpful. It's not perfect, but they're actively working on developing it. > DataFrames.jl: Transform does nonsense (use map instead) The new select and transform functions in DataFrames.jl are actually quite powerful and useful. > DataFrames.jl: No transpose Transpose is not a generic concept for a table. Sure, it might make sense in specific cases, but in general it doesn't make sense to transpose a table.
  - setr 6y ago
    >Sure, it might make sense in specific cases, but in general it doesn't make sense to transpose a table. From my intuition, I don't see anything stopping the function from existing -- it can be applied to any arbitrary table. You probably don't want to transpose your table, except when you want to, but that's true of any function -- I'm can't imagine any scenario where transpose(table)->table would as an algorithm fail (unless I suppose if julia tables include header rows, in which case there's probably no generally correct definition)
    - dragonwriter 6y ago
      > From my intuition, I don't see anything stopping the function from existing A datatable is (or is isomorphic to and can be analyzed as) a mapping from row numbers to tuples of a given shape[0]. The transpose of table will only be table (a mapping from rows numbers to tuples of a common shape) if the tuples of the starting table were homogenous (every field of the same type.) This works with, say, matrices where all the elements are numbers; but it fails in the general case. [0] Yes, I know, Julia defines them as columns of arrays, and columnar organization is ideal for all kinds of processing tasks. For me, thinking about and explaining the the problem with transpose works easier thinking about it in row-oriented form (which is logically equivalent). In column-oriented description, a datatable is an ordered set of columns, each of which is a homogenous array, but if you try to transpose it, each of the columns of the result would be a heterogenous array unless the columns were all of the same type to start with. So, again, it fails to be a table->table function except in the case where the starting table consists of columns of identical type.
- ogogmad 6y ago
  - Arrays start at one Why is this such a big deal? You just change the range of your for-loops from 0<=i<n to 1<=i<=n. Are you using cyclic buffers?
  - adolgert 6y ago
    Dijkstra on the matter: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/E...
    - ogogmad 6y ago
      The points he makes are so extremely minor... You could just as easily say that natural language and mathematical convention are index-1.
      - e12e 6y ago
        I never understood the fascination with preferring "the element at zero offset from the start" to "the first element" - and I don't think djiskarta makes a compelling argument.
    - goto11 6y ago
      Dijkstra has a certain tone. He can make his opinions sound almost like a mathematical proof. But when you dig into it, he just says 0-based is "nicer".
  - Athas 6y ago
    It matters when doing multidimensional index arithmetic. The formulae for ranking and unranking multi-dimensional to flat indexes are neater with 0-indexing than with 1-indexing.
    - ogogmad 6y ago
      Fair enough. But is this a common enough occurrence in the life of a programmer that it's worth being a religious extremist about?
- JHonaker 6y ago
  > Editors (Juno) do not check types at runtime when the information is readily available This complaint is really common with people that see types and expect them to do something like TypeScript. Julia is not a statically typed language. The types are not there for a type checker to check your code for correctness. Instead the types allow Julia to dispatch your method call to the correct implementation for the type of your variable at runtime. This is Julia's secret sauce, and it's the main reason you'll hear about Julia programmers declaring about how composable Julia packages are. I can "reach into your package", and define the behaviour of your functions on my own custom types and then whenever anyone tries to call the function/method with one of my types, it'll just work. I don't need to author a pull request to your package or futz around with anything that you've wrote. Novice Julia programmers often come in thinking that the type annotations are there for ensuring correctness, but that's not it at all. In actuality, you want to be as general with your types as you can, and by default most parameters will probably be untyped. If you need certain behaviour for the function, then you should probably annotate it, but it's definitely not required. I'm guilty of this misstep as well, one of the biggest things I struggled with when I was new was using ::Array everywhere, when what I wanted was ::AbstractArray. > Include() includes (is this PHP all over again?) a file into the script! Use Modules. Don't fault Julia for you being a beginner and not reading the necessary parts of the documentation.
  - systemvoltage 6y ago
    > Instead the types allow Julia to dispatch your method call to the correct implementation for the type of your variable at runtime. This is Julia's secret sauce, and it's the main reason you'll hear about Julia programmers declaring about how composable Julia packages are. I can "reach into your package", and define the behaviour of your functions on my own custom types and then whenever anyone tries to call the function/method with one of my types, it'll just work. I don't need to author a pull request to your package or futz around with anything that you've wrote. That's not a bug, that's a feature. Libraries are abstractions. I get an API and it encapsulates the behavior of the library. That is a nice thing. Being able to modify runtime behavior of internal libraries can lead to insane jumblygoo of code spathetti that would bring the finest programmers to their knees. I don't want implicit behavior at runtime. I want explicit behavior in case of non-contractual externalities (wrong user input for e.g. at runtime). I want the program to fail so it can be patched.
- nalimilan 6y ago
  > - DataFrames.jl: Transform does nonsense (use map instead) What is this supposed to mean? > - DataFrames.jl: No transpose This is actually going to be fixed in the next few days (https://github.com/JuliaData/DataFrames.jl/pull/2447 https://github.com/JuliaData/DataFrames.jl/pull/2447). Given the limited amount of work it required, it sounds quite exaggerated to mention it as a major limitation of the language.
- huijzer 6y ago
  These problems mostly seem to come from trying to use Julia in some kind of visual studio paradigm. The key is to leave the REPL running, I find. Then, with Revise.jl the speed is insane because nothing has to start up; only the changes you make will be recompiled during execution. Also take a look at Pluto.jl. Edit: By the Visual Studio paradigm, I mean a paradigm where all the stuff is made around visual interactions with the IDE. For example, showing autocomplete which works because the programmer is very constrained in OOP (foo.b completes to foo.bar) and statically typed languages with Java and C# as prime examples. I think these constraints are made around the idea that programmers are dumb and that it mostly is a distraction from how one actually wants to think about the problem at hand.
- clircle 6y ago
  Arrays start at 1!? What Joy! Maybe I can try this language!
- kortex 6y ago
  Edit: just learned about OffsetArrays.jl - ok now that's really cool. Keeping comment for context. > - Arrays start at one Oof. Big turnoff for me. For some reason, this particular context switch really grinds my gears. All languages I use (except bash, which grinds my gears) use Dijkstra-style array slices: python, go, c/++, js. I'm sure it makes it easier for Matlab converts. This, plus the mentioned weaknesses with libs like http, grpc et al, IMHO will relegate Julia to many years of just being wrapped by other languages. To avoid that, I think they should be thinking about continuing to woo more of the engineering crowd - which to their credit I think they've done pretty well so far. https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/E... https://github.com/JuliaArrays/OffsetArrays.jl https://github.com/JuliaArrays/OffsetArrays.jl
  - FranzFerdiNaN 6y ago
    They’re making the language mostly for statisticians, mathematicians and scientific programmers, where it makes sense to have arrays start at 1. Why should they cater to whiny engineers who can’t deal with anything they aren’t used to?
fizixer 6y ago
Wow, Julia is a "baby language" and there is not a single project where its dominating the usage landscape. What's next, the unreasonable effectiveness of an idea I just had last night? Come back in 10 years.
- patagurbon 6y ago
  SciML and JuMP are both best in class ecosystems, in some cases by enormous margins. The PPL landscape is also very significant.
langitbiru 6y ago
You know what? In the past, I was toying with the idea of building Julia IDE. In the end, I decided to build something else. So the pain point is still there. Looking at this thread, it could be a good idea to build Julia IDE. Then you can apply for funding in the next YC batch. 6 months should be enough to build an MVP. Bonus point if you could build Julia IDE with Rust. In Show HN, surely you would get a lot of karma points. But we need to be realistic. Rust does not have a solid GUI library. If I chose to build Julia IDE, I would use either C++ with Qt or C++ with wxWidgets.
- andi999 6y ago
  It doesnt sound good that you do not trust julia as a source language for the IDE, actually didnt Julia wanted to solve the two language problem?
  - langitbiru 6y ago
    I got the impression that Julia is catered to numerical computing domain (data science and alike), not general purpose programming language like Python. Does Julia have a good desktop programming library? I guess I am wrong. However when I looked at the website (https://julialang.org https://julialang.org), in the Ecosystem section, they listed these things: Visualization, General Purpose, Data Science, Machine Learning, Scientific Domains, Parallel Computing. I think they should market it in a different way. "General Purpose" should be expanded to "Web Programming", "Desktop Programming", etc.
zepearl 6y ago
Fyi, I liked this to get a quick and initial understanding of the language (I just read it): https://juliabyexample.helpmanual.io https://juliabyexample.helpmanual.io Question: I see that on Gentoo Linux all versions of the language-compiler (pkg "dev-lang/julia" versions 1.2, 1.3, 1.4, 1.5) are all marked as not yet being officially stable ( https://packages.gentoo.org/packages/dev-lang/julia https://packages.gentoo.org/packages/dev-lang/julia ) => any "real" reason for this? Are there any open/important bugs or is it just because of e.g. a low usage of the language itself on the Gentoo distro, or maybe because the specs of the language are still changing, etc...? I think that e.g. Rust more or less as old as Julia, but Rust on Gentoo is marked as being stable since a long time... .
- leephillips 6y ago
  That makes no sense. Everything since Julia v. 1.0 should be considered stable. There have been no real breaking changes since that release.
  - zepearl 6y ago
    Thx EDIT: Ok, maybe it's because of some dependencies. 3 are marked as not yet stable ("~" character). # emerge -pv dev-lang/julia These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] net-libs/mbedtls-2.24.0:0/5.13.1::gentoo USE="threads -doc -havege -libressl -programs -static-libs -test -zlib" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" 3,821 KiB [ebuild N ] media-libs/qhull-2015.2::gentoo USE="-doc -static-libs" 987 KiB [ebuild N ] sci-libs/amd-2.4.6::gentoo USE="fortran -doc" 336 KiB [ebuild N ] sci-libs/camd-2.4.6::gentoo USE="-doc" 310 KiB [ebuild N ] sci-libs/ccolamd-2.9.6::gentoo 299 KiB [ebuild N ~] sci-libs/openlibm-0.7.0:0/0.7.0.0::gentoo USE="-static-libs" 358 KiB [ebuild N ] sci-libs/lapack-3.8.0-r1::gentoo USE="-deprecated -doc -eselect-ldso -lapacke" 7,253 KiB [ebuild N ] media-libs/glfw-3.2.1::gentoo USE="-examples -wayland" 462 KiB [ebuild N ] sci-libs/metis-5.1.0-r4::gentoo USE="openmp -doc" 4,869 KiB [ebuild N ] virtual/lapack-3.8::gentoo USE="-eselect-ldso" 0 KiB [ebuild N ] virtual/blas-3.8::gentoo USE="-eselect-ldso" 0 KiB [ebuild N ~] dev-libs/openspecfun-0.5.1::gentoo USE="-static-libs" 119 KiB [ebuild N ] sci-mathematics/glpk-4.65:0/40::gentoo USE="-doc -examples -gmp -mysql -odbc" 4,070 KiB [ebuild N ~] sci-visualization/gr-0.50.0-r1::gentoo USE="X tiff truetype -cairo -ffmpeg -postscript" 8,411 KiB [ebuild N ] sci-libs/cholmod-3.0.13::gentoo USE="lapack matrixops modify partition (-cuda) -doc" 680 KiB [ebuild N ] sci-libs/arpack-3.1.5::gentoo USE="-doc -examples -mpi" 1,481 KiB [ebuild N ] sci-libs/spqr-2.0.9::gentoo USE="-doc -partition -tbb" 2,111 KiB [ebuild N ] sci-libs/umfpack-5.7.9::gentoo USE="cholmod -doc" 754 KiB [ebuild N ~] dev-lang/julia-1.4.0-r2::gentoo USE="-system-llvm" 44,278 KiB
orbifold 6y ago
Here is my main gripe with Julia: They blatantly copy Matlab. The same is also true to a lesser degree of Python's numpy and matplotlib, but Julia goes the whole nine yards, by basically replicating the syntax and copying most of the numerical APIs including the indexing convention. I understand why they do it and I get that Matlab is hugely popular, I just wish that there was more creative energy in the Open Source world and not just rehashing of 20-40 year old designs.
- ColanR 6y ago
  The thing is, there are parts of Matlab's syntax which are fantastic, and I would be disappointed if they didn't copy it.
- KenoFischer 6y ago
  This is not true. Indexing works very differently in julia than it does in Matlab. There is some overlap in function names, but less than you're implying, mostly in the names that everyone uses anyway (sin, cos, etc) and even where the names are the same, the API details often aren't. Ironically the number of overlapping method names has increased because Matlab has been copying method names from Julia in recent years. That said, Julia and Matlab are completely different languages, so I'd encourage you to look in one step further.
- andi999 6y ago
  If this is blatent, what is octave?
reggieband 6y ago
Julia is one of those programming languages where I find it interesting but I cannot imagine a practical use case for myself. I even took the time to `brew install` it and was thinking about it again recently when I saw it update. The same is true with numpy or R - my work just doesn't involve high performance numerical computing. And even if it did, it would always be secondary to some other purpose. Outside of a REPL, if I were to productionize some heavy computational work, even numerically based, I would probably still lean towards containerizing a C++ (or maybe Rust) binary. It's one of those "right tool for the job" type quandaries. For many popular languages (Javascript, Python, Go, C/C++, Rust, Java, OCaml) I have an intuition on when I would reach for them based on my experience. With Julia - I am not sure the shape or character of the problem where I would reach for it.
- jamil7 6y ago
  Sounds like you just may be the wrong target. Numpy, R and Julia are not designed for software engineering and are likely more approachable and useful to scientists than something like C++ or Rust.
  - deleted 6y ago
    [deleted]
ralphc 6y ago
How easy would it be to use Julia on a problem that would take advantage of the 16 hyperthreaded cores on my computer? Does it use threads, async/await mechanism, something else? Is it intuitive to use?
- leephillips 6y ago
  As easy as starting up the interpreter with -t auto and then using, e.g., pmap instead of map.
- ddragon 6y ago
  Julia's main parallel primitive (@spawnat) [1] is heavily inspired in Go, in which you just run anything in a managed lightweight thread, using a channel to pass data and sync with a fetch. That API is quite recent though (Julia 1.3), so there is still a lot of work going on both on the language side and on the library side to give higher level mechanisms, such as [2]. [1] https://julialang.org/blog/2019/07/multithreading/ https://julialang.org/blog/2019/07/multithreading/ [2] https://juliafolds.github.io/data-parallelism/tutorials/quick-introduction/ https://juliafolds.github.io/data-parallelism/tutorials/quic...
dalke 6y ago
At a conference presentation the other day a speaker made the claim that well-written Julia code was as fast as well-written C code while as expressive as Python, so it solved the two language problem. My Python package uses a C extension for performance. That C extension uses AVX2 intrinsics. How well does Julia support intrinsics? Even supporting something like __builtin_popcountll() would be nice, but http://rosettacode.org/wiki/Population_count#Julia http://rosettacode.org/wiki/Population_count#Julia suggests that it's not supported in Julia. I've no experience with Julia and my attempt at finding this out on my own failed - does anyone here know about Julia's support for intrinsics?
- bacr 6y ago
  My understanding is that the Julia community is quite interested in having SIMD via e.g. AVX “just work”. I recall reading this post on it a while back: https://juliacomputing.com/blog/2017/09/27/auto-vectorization-in-julia.html https://juliacomputing.com/blog/2017/09/27/auto-vectorizatio...
  - dalke 6y ago
    Sure, but you can only do that if either that's a way to express what you want directly or, in the example you gave, there's a common idiomatic style that the compiler can recognize and handle. What is the idiomatic way to write the popcount of the intersection of two 256-byte byte strings? My C code is: static int byte_intersect_256(const unsigned char *fp1, const unsigned char *fp2) { int num_words = 2048 / 64; int intersect_popcount = 0; /* Interpret as 64-bit integers and assume possible mis-alignment is okay. */ uint64_t *fp1_64 = (uint64_t *) fp1, *fp2_64 = (uint64_t *) fp2; for (int i=0; i<num_words; i++) { intersect_popcount += __builtin_popcountll(fp1_64[i] & fp2_64[i]); } return intersect_popcount; } I haven't figured out the Julia way to write it so it would use the POPCNT instruction (if available), the AVX2 popcount technique (if available), or the VPOPCNTDQ AVX-512 instruction (if available) - falling back, I suppose, to the SSSE3 and Lauradoux implementations - the last being the fastest generic C implementation I found. (See https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0398-8/tables/1 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... ).
IshKebab 6y ago
I wanted to like Julia, but loading the graphing library takes longer than starting the entirety of MATLAB. Not a good design.
naveen99 6y ago
Why should I use julia over c++11 ?
bshanks 6y ago
https://dl.acm.org/doi/pdf/10.1145/327649 https://dl.acm.org/doi/pdf/10.1145/327649 ("Julia: Dynamism and Performance Reconciled by Design") section 4.2.1 shows how multiple dispatch enables a library to compute the derivative of another program without modifying that other program to be aware of the derivative-computing library.
leephillips 6y ago
I’d be particularly interested in opinions about whether my popularized attempt to explain the expression problem is effective, or just confusing.
- chrispeel 6y ago
  I started your section about the fish and didn't finish it. I expect ArsTechnica articles to be technical; in this case I think actual code examples would be good. I think Stefan K's video referenced elsewhere in this thread used examples that are both accessible and educational.
- borishn 6y ago
  As a data point, I also found the cooking examples confusing. Probably because I am more familiar with programming languages than cooking :)
- choeger 6y ago
  It was rather confusing and I consider myself well-informed about the expression problem. But to give you some constructive feedback, your understanding, or your writing, of the expression problem lacks two fundamental concepts: First of all, in functional languages it is as simple as in object-oriented languages to add new data types. It hard to add a variant of an existing datatype. In object-oriented languages it is also trivial to add new functions, but it is hard to add a new function for all derivates of a class. So it would be more like having a table for perfectly cooking all variants of fish. And secondly, the solution offered by Julia is also incomplete. Both functional and object-oriented languages give you completeness guarantees. So the compiler will warn you when you miss a variant of fish (functional) for a method or a method for a particular fish (object-oriented) At least to my understanding the completeness of methods (that's the right term, no?) in Julia is unchecked. You can extend, but you can easily miss a case. The same (unchecked extensions) can be done in functional languages with open algebraic datatypes and in object-oriented languages with default methods that throw exceptions.
  - leephillips 6y ago
    Thanks for this detailed criticism, it’s quite useful. My understanding is that for functions with more than a few arguments, it is expected in Julia that only a small subset of the possibly thousands of combinations will be covered. Thousands, because with multiple dispatch you dispatch on the types of all the arguments. The binary operator “*” in Julia has 364 methods, and that just takes two arguments. So you define the ones that are useful.
tokai 6y ago
So how is julia nowadays? I stopped using and following it sometime ago as I found it as slow as python, or worse, for anything that wasn't numerical computing. Really wanted to love it.
- eigenspace 6y ago
  I think julia has evolved a ton over the past few years and a lot of work has gone into making things that aren't numerical computing feel like first class uses of the language. A very popular thing right now in the community is people making websites, dashboards, visualization tools, etc. with julia. There's a lot of people thinking about things like string handling due to NLP, bioinformatics and a few other technical fields that don't use the same standard datatypes you'd expect from an engineer or whatever. I do a lot of random hobby programming, often building random tools that aren't necessarily numerical programming related and I find it quite natural and easy to make these things highly performant (often with no runtime overhead).
- socialdemocrat 6y ago
  If you are not being specific it is impossible to know what you are talking about. It could be a library thing or it could be that your are really just describing the initial latency of the JIT compilation. If you look at the well known language shootout you will see Julia today beats almost everything even on multicore. The clear exception is anything related to garbage collection. But you can circumvent this with turning off the garbage collector temporarily.
- leephillips 6y ago
  First time I've heard anything like that. Most people report a massive speedup without doing anything special. Maybe try the current version?
  - cbkeller 6y ago
    I had an experience slightly like that at first where I was surprised to only be going 1-2x faster than my old MATLAB, but then I realized my code was full of trivially avoidable type instabilities and got another 100x speedup in Julia.
fouric 6y ago
Common Lisp has had multiple dispatch for decades. Yet another Lisp feature "taken" by another language - why not just start with a Lisp and improve it (e.g. Typed Racket, for performance) rather then repeatedly creating new languages and just adding a tiny piece of Lisp to them each time?
- leephillips 6y ago
  As I mention in the article. Note that Julia is in a sense lisp-based, and you can easily see the AST of your functions. You can say +(a, b, c) in Julia, for example.
  - pmoriarty 6y ago
    But unlike Lisp, the Julia language is not homoiconic, so its ASTs don't look like the parent language.
    - KenoFischer 6y ago
      Just swap out the parser ;) -https://github.com/swadey/LispSyntax.jl https://github.com/swadey/LispSyntax.jl
- tokai 6y ago
  >Yet another Lisp feature "taken" by another language Julia is a Lisp.
- KenoFischer 6y ago
  That is how Julia started. In the early days it was a scheme reader macro. You can still run `julia --lisp` to get a scheme prompt since the frontend is still written in it. Obviously lots of improvements have happened since then. Julia itself also feels a lot like a lisp, since Jeff is a huge fan, but of course people get hung up on the syntax.
  - pmoriarty 6y ago
    It's nice that you can write Julia in a more Lispy way, but if you did that you'd probably be one of the only Julia users did so, and your code would not be readily understandable or accpetable to the rest of Julia's users and if you wanted to integrate other Julia code in to your own you'd be stuck with having to use the more Python-like syntax that the vast majority of Julia code is written in. In short, the Julia ecosystem is not a Lisp ecosystem. If you wanted Lisp, you'd be far better off using a real Lisp to begin with, so you can unreservedly participate in an entire Lisp ecosystem, instead of using a language that hides its Lispyness behind a Python-like syntax.
DennisP 6y ago
I tried to get started with Julia a couple months ago, downloaded several libraries, and they wouldn't compile due to missing dependencies. Is there a problem with the package manager?
- oxinabox 6y ago
  That's weird using libraries almost always just works? Did you install them via the package manager? If you just git clones them then you will need to tell the package manager about them e.g. via `add`/`dev`ing the local path, + `Pkg.resolve`. I recommend asking for help on the Julia Discourse. http://Discourse.julialang.org/ http://Discourse.julialang.org/
- deleted 6y ago
  [deleted]
- leephillips 6y ago
  The package manager is one of my favorite things about Julia. No more dependency hell. Just type "add <package>" in the REPL.
- spacedome 6y ago
  I have only seen this happen with packages that depend on non-julia libraries, such as ARPACK, but the move to providing binaries with BinaryBuilder should fix this.
  - DennisP 6y ago
    That might well be the issue, these were game/gui libraries. What's the timeframe on BinaryBuilder?
    - 3JPLW 6y ago
      BinaryBuilder/BinaryProvider are 100% functional and have been deployed and working for a long time now. The timeframes for individual packages to move to using them instead of ad-hoc build scripts, however, varies. It's generally very quick to do this, but complicated chains of binary dependencies can be a pain.
pmoriarty 6y ago
I wish Julia was much more Lisp/Scheme-like in terms of syntax. If they'd just focused on making a more performant, typed-Scheme (or improving one of the existing ones), that would be much more interesting for me. I took a dip in to Julia recently, and found it to be a mishmash of Lisp, Haskell, Matlab and Fortran ideas, wrapped in Python-like syntax. I guess that can be appealing for people coming from those worlds, and for those for whom speed trumps everything else. For me the loss of a Lispy syntax and the price of working with a relatively immature language wasn't worth it for me. I'd just rather use something with reasonable performance that's more Lispy.
- deleted 6y ago
  [deleted]
scoot_718 6y ago
It's another R, something that seems like progress, but will end up stopping further progress.
- acomjean 6y ago
  I didn't like R at first. Some scientists I work with would say, "its amazing, it just works the way I think". One Biostatistics later class and I get it, Still don't love it but I get it. Plus the RSudio work environment and ggplot2 are simply amazing.
- Sukera 6y ago
  Mind elaborating on why you think so?
- jakobnissen 6y ago
  What would true progress look like to you? Specifically regarding Julia's main use case of interactive yet performant programming?
vsskanth 6y ago
I'm an engineer who writes the occasional matlab/python code to automate tasks, process data and fit models. Numerical libraries are extremely important to me. I recently started using Julia for simulating/fitting some differential equations and am thoroughly impressed with the speed, syntax and library documentation. Startup speed was initially painful but I am used to it now since I only pay it once. The IDE situation is not fully stable yet but Juno is very good. I wish they package a standalone Julia IDE (I am aware of the new VS Code plugin) based on either Atom or Code just to make it easier for those switching over from MATLAB and Python (Spyder) IDEs. Don't know how far they've progressed on static compilation yet, but if they get that even for a language subset it would truly be a game changer for general purpose programming.
- 3JPLW 6y ago
  > I wish they package a standalone Julia IDE That's precisely what Julia Computing's JuliaPro is. https://juliacomputing.com/products/juliapro https://juliacomputing.com/products/juliapro (Full disclosure, I am employed by Julia Computing)
  - vsskanth 6y ago
    Thank you. I thought this was some paid enterprise product. Didn't realize there was a free version. It looks like I have to sign-in to download but can't find a link anywhere to create an account, either on the homepage or anywhere else on juliacomputing.com.
    - pbowyer 6y ago
      Here's the register with email link: https://auth.juliahub.com/register.html https://auth.juliahub.com/register.html
      - vsskanth 6y ago
        thanks!
  - 6y ago
Tarrosion 6y ago
I quite liked this article, but both the article and the comments here mostly skip over my absolute favorite Julia feature: because Julia is fast end-to-end, you can code in the style that naturally matches your use case and mental models without sacrificing performance. Functional patterns and imperative patterns, vector-based or element-by-element, your types or built-in: they all work well and run quickly! I do most of my programming in Julia and Python. Python libraries like numpy and pandas are fast and efficient—if you're staying "within the lines" of how the library is designed to work. And most of the time this is okay! But not irregularly I want to do array operations on arrays of user-defined types, or I want to walk through a dataframe row by row without paying a huge performance penalty, etc. And all the sudden the well-tuned Python ecosystem feels very restrictive. In Julia, my workflow is roughly: 1) think about the problem. 2) Code an intuitive solution. 3) If necessary, tweak a little bit of code to improve performance by reducing allocations or type instability. That's a lot less mental work than my Python/Matlab/Java workflow of 1) think about the problem. 2) Think about how the solution can be expressed in the paradigm the language supports performance with. 3) Write a solution in this particular paradigm. 4) Tune for performance, which may be awkward if the initial solution was not intuitive.
- kccqzy 6y ago
  I believe the essence of what you are saying is that Julia is great for explorative programming, while Python et al are better suited for slightly more structured, more productionized programming. I personally use Mathematica for this purpose and find it indispensable. I wonder if the PL designers should focus more on this class of languages, instead of features for production languages (like fancy type systems).
  - dodobirdlord 6y ago
    Julia, by virtue of being compiled, is also probably better for production. But it’s much less well known and has less library support, which is a significant barrier to use.
    - mattkrause 6y ago

284 points by nikbackm 6y ago

Since everybody is so hyped about Julia, I took on learning it and using it in one of my pet projects. The use case seemed to be perfect for Julia: financial portfolio optimization. But honestly, it is a royal pain in the ass to use. Mostly because tooling is so crappy (no decent IDE). For example, I've made these notes on my journey with Julia: - Juno and VSCode julia is a pain - Debugging is pain - no watch expressions (but debug> works) - Works like with matlab (workspaces, designed for the single script use-cases) - Long registry lookup times (2-5mins) when starting a new session - No hints from IDE (Juno) on what methods are available for which types - Ctrl-click navigation does not work to lookup definitations in libraries - Debugger does not show the full list of elements - there are triple dots in the middle - Filters and maps are not lazy - Arrays start at one - Precompilation times can take 1-2min - Package management is done as a part of script execution - Delay (0.5-1s) when processing commands (compilation?) - Editors (Juno) do not check types at runtime when the information is readily available - Include() includes (is this PHP all over again?) a file into the script! - Reuses same REPL between different files - DataFrames.jl: Transform does nonsense (use map instead) - DataFrames.jl: No transpose Obviously, there is a bunch of good things like freakin insane speed and vectorized method calls are awesome, but I struggle seeing myself becoming more producting than Python + Pandas. It's really a shame that language is strongly and semi-statically typed yet no tooling is making a use of that.

jackcviers3 6y ago
To be fair, most of these are ide complaints, not language complaints. Much like all development work, language designers are now required to know much more than how to write an effective static compiler, an effective language syntax, and an effective standard library. A working IDE that integrates easily with all mainstream editors, extensions for all mainstream editors, cross-platform compilation for all major environments, a secure and effective dependency manager, a secure and effective build tool, and incremental compilation are all requirements for a language to get off the ground. Honestly, that seems like a big hurdle for one person or even a dedicated team to manage effectively without major funding outside of what is normally present in research grants. Hosting and curation alone for the dependency package system is a costly problem. Are we as developers painting ourselves into a corner of stagnating pl technology with our expectations that all these things exist in order for us to call a language "good?"
- tasubotadas 6y ago
  Yes. But only computer scientists are interested only in the language itself. I am an engineer so I have to deal with practical aspects of it.
  - jackcviers3 6y ago
    I'm an engineer. I primarily care about how good the language is at expressing and executing my intent. I've been around long enough to see several tooling fads come and go. I've also been around enough to know that the barrier for delivering quality software is becoming insurmountable, and the additional tooling and infrastructure isn't resulting in projects being more successful, nor being delivered more quickly, nor being less costly than when I started. Teams are enormous now. Software projects are enormous. Deployment requires deep knowledge of multiple hosting platforms and various cloud provider apis at the enterprise level. Available libraries, increasing reliance upon testing, type safety, and better development life cycle practices have had a far larger impact than the ide's I've seen come and go. In my experience, a particular tool has a popularity lifecycle of about ten years. Vscode and lsp will probably be the same. Give me types, good compiler error output, and access to a large universe of libraries with good documentation over relying upon ide api discovery any day of the week and twice on Sundays. The rest is an ever-changing tide of requirements fashion.
    - kuzuman 6y ago
      "I am an engineer" Did you pass the Fundamentals of Engineering exam, are you a member in order of your local association of engineers?, do you have an engineer seal? If you answer no to the questions above please don't call yourself engineer
  - KenoFischer 6y ago
    Give us some more time, there's just so many things to do and the reliance issues favor getting the fundamentals right first (much easier to switch the debugger UI than the names of functions that'll get put all over the place). We certainly understand the value of good tooling, so just stay tuned and maybe try it again every once in a while to see if it's good enough for you yet. Too much to do, too little time :)
bieganek 6y ago
> Long registry lookup times (2-5mins) when starting a new session I'm not sure what you are referring to here. Yes, sometimes after you've installed new packages, the Julia VS Code Language Server has to do some re-indexing which can take some time, but they've improved this and you can still program and use the integrated REPL while this is occurring. > Ctrl-click navigation does not work to lookup definitations in libraries In VS Code, F12 (Go to Definition) does work. You can also use `@edit foo(x)` in the REPL. > Filters and maps are not lazy Use Iterators.filter or use generator expressions. > Package management is done as a part of script execution I'm also not sure what this refers to, but the package manager in Julia is one of the best things about Julia. > Editors (Juno) do not check types at runtime when the information is readily available The VS Code Julia extension has a linter that is pretty helpful. It's not perfect, but they're actively working on developing it. > DataFrames.jl: Transform does nonsense (use map instead) The new select and transform functions in DataFrames.jl are actually quite powerful and useful. > DataFrames.jl: No transpose Transpose is not a generic concept for a table. Sure, it might make sense in specific cases, but in general it doesn't make sense to transpose a table.
- setr 6y ago
  >Sure, it might make sense in specific cases, but in general it doesn't make sense to transpose a table. From my intuition, I don't see anything stopping the function from existing -- it can be applied to any arbitrary table. You probably don't want to transpose your table, except when you want to, but that's true of any function -- I'm can't imagine any scenario where transpose(table)->table would as an algorithm fail (unless I suppose if julia tables include header rows, in which case there's probably no generally correct definition)
  - dragonwriter 6y ago
    > From my intuition, I don't see anything stopping the function from existing A datatable is (or is isomorphic to and can be analyzed as) a mapping from row numbers to tuples of a given shape[0]. The transpose of table will only be table (a mapping from rows numbers to tuples of a common shape) if the tuples of the starting table were homogenous (every field of the same type.) This works with, say, matrices where all the elements are numbers; but it fails in the general case. [0] Yes, I know, Julia defines them as columns of arrays, and columnar organization is ideal for all kinds of processing tasks. For me, thinking about and explaining the the problem with transpose works easier thinking about it in row-oriented form (which is logically equivalent). In column-oriented description, a datatable is an ordered set of columns, each of which is a homogenous array, but if you try to transpose it, each of the columns of the result would be a heterogenous array unless the columns were all of the same type to start with. So, again, it fails to be a table->table function except in the case where the starting table consists of columns of identical type.
ogogmad 6y ago
- Arrays start at one Why is this such a big deal? You just change the range of your for-loops from 0<=i<n to 1<=i<=n. Are you using cyclic buffers?
- adolgert 6y ago
  Dijkstra on the matter: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/E...
  - ogogmad 6y ago
    The points he makes are so extremely minor... You could just as easily say that natural language and mathematical convention are index-1.
    - e12e 6y ago
      I never understood the fascination with preferring "the element at zero offset from the start" to "the first element" - and I don't think djiskarta makes a compelling argument.
  - goto11 6y ago
    Dijkstra has a certain tone. He can make his opinions sound almost like a mathematical proof. But when you dig into it, he just says 0-based is "nicer".
- Athas 6y ago
  It matters when doing multidimensional index arithmetic. The formulae for ranking and unranking multi-dimensional to flat indexes are neater with 0-indexing than with 1-indexing.
  - ogogmad 6y ago
    Fair enough. But is this a common enough occurrence in the life of a programmer that it's worth being a religious extremist about?
JHonaker 6y ago
> Editors (Juno) do not check types at runtime when the information is readily available This complaint is really common with people that see types and expect them to do something like TypeScript. Julia is not a statically typed language. The types are not there for a type checker to check your code for correctness. Instead the types allow Julia to dispatch your method call to the correct implementation for the type of your variable at runtime. This is Julia's secret sauce, and it's the main reason you'll hear about Julia programmers declaring about how composable Julia packages are. I can "reach into your package", and define the behaviour of your functions on my own custom types and then whenever anyone tries to call the function/method with one of my types, it'll just work. I don't need to author a pull request to your package or futz around with anything that you've wrote. Novice Julia programmers often come in thinking that the type annotations are there for ensuring correctness, but that's not it at all. In actuality, you want to be as general with your types as you can, and by default most parameters will probably be untyped. If you need certain behaviour for the function, then you should probably annotate it, but it's definitely not required. I'm guilty of this misstep as well, one of the biggest things I struggled with when I was new was using ::Array everywhere, when what I wanted was ::AbstractArray. > Include() includes (is this PHP all over again?) a file into the script! Use Modules. Don't fault Julia for you being a beginner and not reading the necessary parts of the documentation.
- systemvoltage 6y ago
  > Instead the types allow Julia to dispatch your method call to the correct implementation for the type of your variable at runtime. This is Julia's secret sauce, and it's the main reason you'll hear about Julia programmers declaring about how composable Julia packages are. I can "reach into your package", and define the behaviour of your functions on my own custom types and then whenever anyone tries to call the function/method with one of my types, it'll just work. I don't need to author a pull request to your package or futz around with anything that you've wrote. That's not a bug, that's a feature. Libraries are abstractions. I get an API and it encapsulates the behavior of the library. That is a nice thing. Being able to modify runtime behavior of internal libraries can lead to insane jumblygoo of code spathetti that would bring the finest programmers to their knees. I don't want implicit behavior at runtime. I want explicit behavior in case of non-contractual externalities (wrong user input for e.g. at runtime). I want the program to fail so it can be patched.
nalimilan 6y ago
> - DataFrames.jl: Transform does nonsense (use map instead) What is this supposed to mean? > - DataFrames.jl: No transpose This is actually going to be fixed in the next few days (https://github.com/JuliaData/DataFrames.jl/pull/2447 https://github.com/JuliaData/DataFrames.jl/pull/2447). Given the limited amount of work it required, it sounds quite exaggerated to mention it as a major limitation of the language.
huijzer 6y ago
These problems mostly seem to come from trying to use Julia in some kind of visual studio paradigm. The key is to leave the REPL running, I find. Then, with Revise.jl the speed is insane because nothing has to start up; only the changes you make will be recompiled during execution. Also take a look at Pluto.jl. Edit: By the Visual Studio paradigm, I mean a paradigm where all the stuff is made around visual interactions with the IDE. For example, showing autocomplete which works because the programmer is very constrained in OOP (foo.b completes to foo.bar) and statically typed languages with Java and C# as prime examples. I think these constraints are made around the idea that programmers are dumb and that it mostly is a distraction from how one actually wants to think about the problem at hand.
clircle 6y ago
Arrays start at 1!? What Joy! Maybe I can try this language!
kortex 6y ago
Edit: just learned about OffsetArrays.jl - ok now that's really cool. Keeping comment for context. > - Arrays start at one Oof. Big turnoff for me. For some reason, this particular context switch really grinds my gears. All languages I use (except bash, which grinds my gears) use Dijkstra-style array slices: python, go, c/++, js. I'm sure it makes it easier for Matlab converts. This, plus the mentioned weaknesses with libs like http, grpc et al, IMHO will relegate Julia to many years of just being wrapped by other languages. To avoid that, I think they should be thinking about continuing to woo more of the engineering crowd - which to their credit I think they've done pretty well so far. https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/E... https://github.com/JuliaArrays/OffsetArrays.jl https://github.com/JuliaArrays/OffsetArrays.jl
- FranzFerdiNaN 6y ago
  They’re making the language mostly for statisticians, mathematicians and scientific programmers, where it makes sense to have arrays start at 1. Why should they cater to whiny engineers who can’t deal with anything they aren’t used to?

fizixer 6y ago

Wow, Julia is a "baby language" and there is not a single project where its dominating the usage landscape. What's next, the unreasonable effectiveness of an idea I just had last night? Come back in 10 years.

patagurbon 6y ago
SciML and JuMP are both best in class ecosystems, in some cases by enormous margins. The PPL landscape is also very significant.

langitbiru 6y ago

You know what? In the past, I was toying with the idea of building Julia IDE. In the end, I decided to build something else. So the pain point is still there. Looking at this thread, it could be a good idea to build Julia IDE. Then you can apply for funding in the next YC batch. 6 months should be enough to build an MVP. Bonus point if you could build Julia IDE with Rust. In Show HN, surely you would get a lot of karma points. But we need to be realistic. Rust does not have a solid GUI library. If I chose to build Julia IDE, I would use either C++ with Qt or C++ with wxWidgets.

andi999 6y ago
It doesnt sound good that you do not trust julia as a source language for the IDE, actually didnt Julia wanted to solve the two language problem?
- langitbiru 6y ago
  I got the impression that Julia is catered to numerical computing domain (data science and alike), not general purpose programming language like Python. Does Julia have a good desktop programming library? I guess I am wrong. However when I looked at the website (https://julialang.org https://julialang.org), in the Ecosystem section, they listed these things: Visualization, General Purpose, Data Science, Machine Learning, Scientific Domains, Parallel Computing. I think they should market it in a different way. "General Purpose" should be expanded to "Web Programming", "Desktop Programming", etc.

zepearl 6y ago

Fyi, I liked this to get a quick and initial understanding of the language (I just read it): https://juliabyexample.helpmanual.io https://juliabyexample.helpmanual.io Question: I see that on Gentoo Linux all versions of the language-compiler (pkg "dev-lang/julia" versions 1.2, 1.3, 1.4, 1.5) are all marked as not yet being officially stable ( https://packages.gentoo.org/packages/dev-lang/julia https://packages.gentoo.org/packages/dev-lang/julia ) => any "real" reason for this? Are there any open/important bugs or is it just because of e.g. a low usage of the language itself on the Gentoo distro, or maybe because the specs of the language are still changing, etc...? I think that e.g. Rust more or less as old as Julia, but Rust on Gentoo is marked as being stable since a long time... .

leephillips 6y ago
That makes no sense. Everything since Julia v. 1.0 should be considered stable. There have been no real breaking changes since that release.
- zepearl 6y ago
  Thx EDIT: Ok, maybe it's because of some dependencies. 3 are marked as not yet stable ("~" character). # emerge -pv dev-lang/julia These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] net-libs/mbedtls-2.24.0:0/5.13.1::gentoo USE="threads -doc -havege -libressl -programs -static-libs -test -zlib" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" 3,821 KiB [ebuild N ] media-libs/qhull-2015.2::gentoo USE="-doc -static-libs" 987 KiB [ebuild N ] sci-libs/amd-2.4.6::gentoo USE="fortran -doc" 336 KiB [ebuild N ] sci-libs/camd-2.4.6::gentoo USE="-doc" 310 KiB [ebuild N ] sci-libs/ccolamd-2.9.6::gentoo 299 KiB [ebuild N ~] sci-libs/openlibm-0.7.0:0/0.7.0.0::gentoo USE="-static-libs" 358 KiB [ebuild N ] sci-libs/lapack-3.8.0-r1::gentoo USE="-deprecated -doc -eselect-ldso -lapacke" 7,253 KiB [ebuild N ] media-libs/glfw-3.2.1::gentoo USE="-examples -wayland" 462 KiB [ebuild N ] sci-libs/metis-5.1.0-r4::gentoo USE="openmp -doc" 4,869 KiB [ebuild N ] virtual/lapack-3.8::gentoo USE="-eselect-ldso" 0 KiB [ebuild N ] virtual/blas-3.8::gentoo USE="-eselect-ldso" 0 KiB [ebuild N ~] dev-libs/openspecfun-0.5.1::gentoo USE="-static-libs" 119 KiB [ebuild N ] sci-mathematics/glpk-4.65:0/40::gentoo USE="-doc -examples -gmp -mysql -odbc" 4,070 KiB [ebuild N ~] sci-visualization/gr-0.50.0-r1::gentoo USE="X tiff truetype -cairo -ffmpeg -postscript" 8,411 KiB [ebuild N ] sci-libs/cholmod-3.0.13::gentoo USE="lapack matrixops modify partition (-cuda) -doc" 680 KiB [ebuild N ] sci-libs/arpack-3.1.5::gentoo USE="-doc -examples -mpi" 1,481 KiB [ebuild N ] sci-libs/spqr-2.0.9::gentoo USE="-doc -partition -tbb" 2,111 KiB [ebuild N ] sci-libs/umfpack-5.7.9::gentoo USE="cholmod -doc" 754 KiB [ebuild N ~] dev-lang/julia-1.4.0-r2::gentoo USE="-system-llvm" 44,278 KiB

orbifold 6y ago

Here is my main gripe with Julia: They blatantly copy Matlab. The same is also true to a lesser degree of Python's numpy and matplotlib, but Julia goes the whole nine yards, by basically replicating the syntax and copying most of the numerical APIs including the indexing convention. I understand why they do it and I get that Matlab is hugely popular, I just wish that there was more creative energy in the Open Source world and not just rehashing of 20-40 year old designs.

ColanR 6y ago
The thing is, there are parts of Matlab's syntax which are fantastic, and I would be disappointed if they didn't copy it.
KenoFischer 6y ago
This is not true. Indexing works very differently in julia than it does in Matlab. There is some overlap in function names, but less than you're implying, mostly in the names that everyone uses anyway (sin, cos, etc) and even where the names are the same, the API details often aren't. Ironically the number of overlapping method names has increased because Matlab has been copying method names from Julia in recent years. That said, Julia and Matlab are completely different languages, so I'd encourage you to look in one step further.
andi999 6y ago
If this is blatent, what is octave?

reggieband 6y ago

Julia is one of those programming languages where I find it interesting but I cannot imagine a practical use case for myself. I even took the time to `brew install` it and was thinking about it again recently when I saw it update. The same is true with numpy or R - my work just doesn't involve high performance numerical computing. And even if it did, it would always be secondary to some other purpose. Outside of a REPL, if I were to productionize some heavy computational work, even numerically based, I would probably still lean towards containerizing a C++ (or maybe Rust) binary. It's one of those "right tool for the job" type quandaries. For many popular languages (Javascript, Python, Go, C/C++, Rust, Java, OCaml) I have an intuition on when I would reach for them based on my experience. With Julia - I am not sure the shape or character of the problem where I would reach for it.

jamil7 6y ago
Sounds like you just may be the wrong target. Numpy, R and Julia are not designed for software engineering and are likely more approachable and useful to scientists than something like C++ or Rust.
- deleted 6y ago
  [deleted]

ralphc 6y ago

How easy would it be to use Julia on a problem that would take advantage of the 16 hyperthreaded cores on my computer? Does it use threads, async/await mechanism, something else? Is it intuitive to use?

leephillips 6y ago
As easy as starting up the interpreter with -t auto and then using, e.g., pmap instead of map.
ddragon 6y ago
Julia's main parallel primitive (@spawnat) [1] is heavily inspired in Go, in which you just run anything in a managed lightweight thread, using a channel to pass data and sync with a fetch. That API is quite recent though (Julia 1.3), so there is still a lot of work going on both on the language side and on the library side to give higher level mechanisms, such as [2]. [1] https://julialang.org/blog/2019/07/multithreading/ https://julialang.org/blog/2019/07/multithreading/ [2] https://juliafolds.github.io/data-parallelism/tutorials/quick-introduction/ https://juliafolds.github.io/data-parallelism/tutorials/quic...

dalke 6y ago

At a conference presentation the other day a speaker made the claim that well-written Julia code was as fast as well-written C code while as expressive as Python, so it solved the two language problem. My Python package uses a C extension for performance. That C extension uses AVX2 intrinsics. How well does Julia support intrinsics? Even supporting something like __builtin_popcountll() would be nice, but http://rosettacode.org/wiki/Population_count#Julia http://rosettacode.org/wiki/Population_count#Julia suggests that it's not supported in Julia. I've no experience with Julia and my attempt at finding this out on my own failed - does anyone here know about Julia's support for intrinsics?

bacr 6y ago
My understanding is that the Julia community is quite interested in having SIMD via e.g. AVX “just work”. I recall reading this post on it a while back: https://juliacomputing.com/blog/2017/09/27/auto-vectorization-in-julia.html https://juliacomputing.com/blog/2017/09/27/auto-vectorizatio...
- dalke 6y ago
  Sure, but you can only do that if either that's a way to express what you want directly or, in the example you gave, there's a common idiomatic style that the compiler can recognize and handle. What is the idiomatic way to write the popcount of the intersection of two 256-byte byte strings? My C code is: static int byte_intersect_256(const unsigned char *fp1, const unsigned char *fp2) { int num_words = 2048 / 64; int intersect_popcount = 0; /* Interpret as 64-bit integers and assume possible mis-alignment is okay. */ uint64_t *fp1_64 = (uint64_t *) fp1, *fp2_64 = (uint64_t *) fp2; for (int i=0; i<num_words; i++) { intersect_popcount += __builtin_popcountll(fp1_64[i] & fp2_64[i]); } return intersect_popcount; } I haven't figured out the Julia way to write it so it would use the POPCNT instruction (if available), the AVX2 popcount technique (if available), or the VPOPCNTDQ AVX-512 instruction (if available) - falling back, I suppose, to the SSSE3 and Lauradoux implementations - the last being the fastest generic C implementation I found. (See https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0398-8/tables/1 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... ).

IshKebab 6y ago

I wanted to like Julia, but loading the graphing library takes longer than starting the entirety of MATLAB. Not a good design.

naveen99 6y ago

Why should I use julia over c++11 ?

bshanks 6y ago

https://dl.acm.org/doi/pdf/10.1145/327649 https://dl.acm.org/doi/pdf/10.1145/327649 ("Julia: Dynamism and Performance Reconciled by Design") section 4.2.1 shows how multiple dispatch enables a library to compute the derivative of another program without modifying that other program to be aware of the derivative-computing library.

leephillips 6y ago

I’d be particularly interested in opinions about whether my popularized attempt to explain the expression problem is effective, or just confusing.

chrispeel 6y ago
I started your section about the fish and didn't finish it. I expect ArsTechnica articles to be technical; in this case I think actual code examples would be good. I think Stefan K's video referenced elsewhere in this thread used examples that are both accessible and educational.
borishn 6y ago
As a data point, I also found the cooking examples confusing. Probably because I am more familiar with programming languages than cooking :)
choeger 6y ago
It was rather confusing and I consider myself well-informed about the expression problem. But to give you some constructive feedback, your understanding, or your writing, of the expression problem lacks two fundamental concepts: First of all, in functional languages it is as simple as in object-oriented languages to add new data types. It hard to add a variant of an existing datatype. In object-oriented languages it is also trivial to add new functions, but it is hard to add a new function for all derivates of a class. So it would be more like having a table for perfectly cooking all variants of fish. And secondly, the solution offered by Julia is also incomplete. Both functional and object-oriented languages give you completeness guarantees. So the compiler will warn you when you miss a variant of fish (functional) for a method or a method for a particular fish (object-oriented) At least to my understanding the completeness of methods (that's the right term, no?) in Julia is unchecked. You can extend, but you can easily miss a case. The same (unchecked extensions) can be done in functional languages with open algebraic datatypes and in object-oriented languages with default methods that throw exceptions.
- leephillips 6y ago
  Thanks for this detailed criticism, it’s quite useful. My understanding is that for functions with more than a few arguments, it is expected in Julia that only a small subset of the possibly thousands of combinations will be covered. Thousands, because with multiple dispatch you dispatch on the types of all the arguments. The binary operator “*” in Julia has 364 methods, and that just takes two arguments. So you define the ones that are useful.

tokai 6y ago

So how is julia nowadays? I stopped using and following it sometime ago as I found it as slow as python, or worse, for anything that wasn't numerical computing. Really wanted to love it.

eigenspace 6y ago
I think julia has evolved a ton over the past few years and a lot of work has gone into making things that aren't numerical computing feel like first class uses of the language. A very popular thing right now in the community is people making websites, dashboards, visualization tools, etc. with julia. There's a lot of people thinking about things like string handling due to NLP, bioinformatics and a few other technical fields that don't use the same standard datatypes you'd expect from an engineer or whatever. I do a lot of random hobby programming, often building random tools that aren't necessarily numerical programming related and I find it quite natural and easy to make these things highly performant (often with no runtime overhead).
socialdemocrat 6y ago
If you are not being specific it is impossible to know what you are talking about. It could be a library thing or it could be that your are really just describing the initial latency of the JIT compilation. If you look at the well known language shootout you will see Julia today beats almost everything even on multicore. The clear exception is anything related to garbage collection. But you can circumvent this with turning off the garbage collector temporarily.
leephillips 6y ago
First time I've heard anything like that. Most people report a massive speedup without doing anything special. Maybe try the current version?
- cbkeller 6y ago
  I had an experience slightly like that at first where I was surprised to only be going 1-2x faster than my old MATLAB, but then I realized my code was full of trivially avoidable type instabilities and got another 100x speedup in Julia.

fouric 6y ago

Common Lisp has had multiple dispatch for decades. Yet another Lisp feature "taken" by another language - why not just start with a Lisp and improve it (e.g. Typed Racket, for performance) rather then repeatedly creating new languages and just adding a tiny piece of Lisp to them each time?

leephillips 6y ago
As I mention in the article. Note that Julia is in a sense lisp-based, and you can easily see the AST of your functions. You can say +(a, b, c) in Julia, for example.
- pmoriarty 6y ago
  But unlike Lisp, the Julia language is not homoiconic, so its ASTs don't look like the parent language.
  - KenoFischer 6y ago
    Just swap out the parser ;) -https://github.com/swadey/LispSyntax.jl https://github.com/swadey/LispSyntax.jl
tokai 6y ago
>Yet another Lisp feature "taken" by another language Julia is a Lisp.
KenoFischer 6y ago
That is how Julia started. In the early days it was a scheme reader macro. You can still run `julia --lisp` to get a scheme prompt since the frontend is still written in it. Obviously lots of improvements have happened since then. Julia itself also feels a lot like a lisp, since Jeff is a huge fan, but of course people get hung up on the syntax.
- pmoriarty 6y ago
  It's nice that you can write Julia in a more Lispy way, but if you did that you'd probably be one of the only Julia users did so, and your code would not be readily understandable or accpetable to the rest of Julia's users and if you wanted to integrate other Julia code in to your own you'd be stuck with having to use the more Python-like syntax that the vast majority of Julia code is written in. In short, the Julia ecosystem is not a Lisp ecosystem. If you wanted Lisp, you'd be far better off using a real Lisp to begin with, so you can unreservedly participate in an entire Lisp ecosystem, instead of using a language that hides its Lispyness behind a Python-like syntax.

DennisP 6y ago

I tried to get started with Julia a couple months ago, downloaded several libraries, and they wouldn't compile due to missing dependencies. Is there a problem with the package manager?

oxinabox 6y ago
That's weird using libraries almost always just works? Did you install them via the package manager? If you just git clones them then you will need to tell the package manager about them e.g. via `add`/`dev`ing the local path, + `Pkg.resolve`. I recommend asking for help on the Julia Discourse. http://Discourse.julialang.org/ http://Discourse.julialang.org/
deleted 6y ago
[deleted]
leephillips 6y ago
The package manager is one of my favorite things about Julia. No more dependency hell. Just type "add <package>" in the REPL.
spacedome 6y ago
I have only seen this happen with packages that depend on non-julia libraries, such as ARPACK, but the move to providing binaries with BinaryBuilder should fix this.
- DennisP 6y ago
  That might well be the issue, these were game/gui libraries. What's the timeframe on BinaryBuilder?
  - 3JPLW 6y ago
    BinaryBuilder/BinaryProvider are 100% functional and have been deployed and working for a long time now. The timeframes for individual packages to move to using them instead of ad-hoc build scripts, however, varies. It's generally very quick to do this, but complicated chains of binary dependencies can be a pain.

pmoriarty 6y ago

I wish Julia was much more Lisp/Scheme-like in terms of syntax. If they'd just focused on making a more performant, typed-Scheme (or improving one of the existing ones), that would be much more interesting for me. I took a dip in to Julia recently, and found it to be a mishmash of Lisp, Haskell, Matlab and Fortran ideas, wrapped in Python-like syntax. I guess that can be appealing for people coming from those worlds, and for those for whom speed trumps everything else. For me the loss of a Lispy syntax and the price of working with a relatively immature language wasn't worth it for me. I'd just rather use something with reasonable performance that's more Lispy.

deleted 6y ago
[deleted]

scoot_718 6y ago

It's another R, something that seems like progress, but will end up stopping further progress.

acomjean 6y ago
I didn't like R at first. Some scientists I work with would say, "its amazing, it just works the way I think". One Biostatistics later class and I get it, Still don't love it but I get it. Plus the RSudio work environment and ggplot2 are simply amazing.
Sukera 6y ago
Mind elaborating on why you think so?
jakobnissen 6y ago
What would true progress look like to you? Specifically regarding Julia's main use case of interactive yet performant programming?

vsskanth 6y ago

I'm an engineer who writes the occasional matlab/python code to automate tasks, process data and fit models. Numerical libraries are extremely important to me. I recently started using Julia for simulating/fitting some differential equations and am thoroughly impressed with the speed, syntax and library documentation. Startup speed was initially painful but I am used to it now since I only pay it once. The IDE situation is not fully stable yet but Juno is very good. I wish they package a standalone Julia IDE (I am aware of the new VS Code plugin) based on either Atom or Code just to make it easier for those switching over from MATLAB and Python (Spyder) IDEs. Don't know how far they've progressed on static compilation yet, but if they get that even for a language subset it would truly be a game changer for general purpose programming.

3JPLW 6y ago
> I wish they package a standalone Julia IDE That's precisely what Julia Computing's JuliaPro is. https://juliacomputing.com/products/juliapro https://juliacomputing.com/products/juliapro (Full disclosure, I am employed by Julia Computing)
- vsskanth 6y ago
  Thank you. I thought this was some paid enterprise product. Didn't realize there was a free version. It looks like I have to sign-in to download but can't find a link anywhere to create an account, either on the homepage or anywhere else on juliacomputing.com.
  - pbowyer 6y ago
    Here's the register with email link: https://auth.juliahub.com/register.html https://auth.juliahub.com/register.html
    - vsskanth 6y ago
      thanks!
- 6y ago

Tarrosion 6y ago

I quite liked this article, but both the article and the comments here mostly skip over my absolute favorite Julia feature: because Julia is fast end-to-end, you can code in the style that naturally matches your use case and mental models without sacrificing performance. Functional patterns and imperative patterns, vector-based or element-by-element, your types or built-in: they all work well and run quickly! I do most of my programming in Julia and Python. Python libraries like numpy and pandas are fast and efficient—if you're staying "within the lines" of how the library is designed to work. And most of the time this is okay! But not irregularly I want to do array operations on arrays of user-defined types, or I want to walk through a dataframe row by row without paying a huge performance penalty, etc. And all the sudden the well-tuned Python ecosystem feels very restrictive. In Julia, my workflow is roughly: 1) think about the problem. 2) Code an intuitive solution. 3) If necessary, tweak a little bit of code to improve performance by reducing allocations or type instability. That's a lot less mental work than my Python/Matlab/Java workflow of 1) think about the problem. 2) Think about how the solution can be expressed in the paradigm the language supports performance with. 3) Write a solution in this particular paradigm. 4) Tune for performance, which may be awkward if the initial solution was not intuitive.

kccqzy 6y ago
I believe the essence of what you are saying is that Julia is great for explorative programming, while Python et al are better suited for slightly more structured, more productionized programming. I personally use Mathematica for this purpose and find it indispensable. I wonder if the PL designers should focus more on this class of languages, instead of features for production languages (like fancy type systems).
- dodobirdlord 6y ago
  Julia, by virtue of being compiled, is also probably better for production. But it’s much less well known and has less library support, which is a significant barrier to use.
  - mattkrause 6y ago