12 ms·

It's great and all, but I can't justify switching languages for minor improvements over Python + numpy/scipy. I'd be abandoning: * My deep knowledge and expe

• deleted 9y ago
  [deleted]
• mkborregaard 9y ago
  Julia supports indexing with arbitrary bases, and it generally works ecosystem-wide.
  • Bromskloss 9y ago
    Oh, it does? I've taken to saying that "zero-based indexing – that's how Julia broke my heart", because at the time, I didn't get the impression that there was anything else in sight. How do I index based on zero?
    • wallnuss 9y ago
      If you have a particular algorithm that is better expressed using 0-based indexing use https://github.com/JuliaArrays/OffsetArrays.jl https://github.com/JuliaArrays/OffsetArrays.jl
      • Bromskloss 9y ago
        It's not just something particular I need it for; it's everything.
        • SolarNet 9y ago
          If you read the OP again carefully, you'll see that you can use offset arrays for every array you make and suffer no performance penalty, because the offset is compiled away. And the development overhead is a single library import call; I replace core language data-structures all the time in any language for features I want, this is no different.
          • omginternets 9y ago
            Yeeesh, that sounds like a maintenance nightmare...
            • ChrisRackauckas 9y ago
              Why? You don't have to write a package any differently to make it work with arbitrary indices.
              • omginternets 9y ago
                Because it's inconsistent.
          • ACow_Adonis 9y ago
            See my other comment in the thread for the thing I've always wondered: How can you compile away an arbitrary calculation that might need to be made at run time? (assuming we're not just arguing over arbitrary-array indexing to input predefined constants at the REPL/code level).
            • Bromskloss 9y ago
              Could it be that the offset calculation to find the start of the array is needed in any case?
• wallnuss 9y ago
  * My entire codebase Julia has great interoperability between Python (checkout PyCall.jl) and many other languages. You don't need to abandon your old codebase. With regards to your other points, a lot of that is subjective and hard to argue about, but for me Julia has drastically changed the way I approach writing scientific code and is not just an incremental improvement upon the status quo.
  • rattray 9y ago
    Interesting. How has it changed your approach?
• jernfrost 9y ago
  The improvements are not minor they are massive. Citing rust shows that the advantage of Julia has not been explained well enough. Julia allows you to write as performant code as Rust with a much smaller investment in learning. You cite your concern for spending time learning something new. That makes no sense considering the high learning curve and complexity of Rust compared to Julia. Julia is quite fast to learn and has far more overlap with Python than Rust to make the transition easy. You can call python code from Julia and ise many of the same tools like the notebook.
  • 88e282102ae2e5b 9y ago
    Is Julia really that much more performant than numpy?
    • wallnuss 9y ago
      In my opinion that is the wrong question to ask. The right question is: How fast will my code be. Numpy has been heavily optimised and is written in C and not Python. Take a look at the link below, comparing a simple sum in different languages (among them Python and Numpy). The power of Julia is that there is no privileged code. Your code will be as fast as the base library. http://nbviewer.jupyter.org/github/alanedelman/18.337_2017/blob/282c25c9229187b18a7c61e7167387c31f40f15e/lectures/Lecture01_0906%20AutoDiff%20%26%20JuliaIsFast/JuliaIsFast.ipynb http://nbviewer.jupyter.org/github/alanedelman/18.337_2017/b...
      • 81212w1 9y ago
        I find this benchmark a bit disorganized, but it seems that the "hand-written" Julia function is as slow as the "hand-written" C function. The C function is of course naive. I'm pretty sure that a hand-unrolled loop would be faster.
        • wallnuss 9y ago
          You are absolutely right, an optimised C Programn would be as fast or faster than the Numpy implementation.
• ViralBShah 9y ago
  In my opinion, this is not about abandoning, but supplementing. Python is good at many things, as is Rust, as is Julia. Stuff like the ODE library that Chris has, or JuMP (for mathematical programming) that Miles Lubin, Iain Dunning and Joey Huechette wrote, or a number of other packages are simply not available elsewhere. The Celeste project, for example, achieved 1.6 PetaFlop/sec of compute rate on half a million cores. Programming languages are abstractions. Some abstractions are better for certain kind of problems. Many of the amazing libraries that are available in Julia would simply not be easy to do otherwise. Think about it - if these were incremental advances and no serious community would ever form - then Chris would have never written his packages. Julia has seen over a million downloads. In my biased opinion, it is more than an incremental advance. I think the right approach is to understand where those advances are and apply it to the right kind of problems. There is a balance to be struck between the two extremes of nothing new will ever get adopted, and everything should adopt anything new and shiny that comes out.
  • throwaway7645 9y ago
    Python's PULP & Pyomo are pretty similar to JUMP.
    • StefanKarpinski 9y ago
      The JuMP paper [1] compares JuMP with all the major commercial and open source options in this area, including Pyomo. Pyomo is orders of magnitude slower than JuMP and doesn't scale as well so the comparison gets worse as problems get larger. (Pyomo had the worst performance of all the systems compared.) JuMP is the only open source option that has performance like commercial systems. Throw in JuMP's improved expressiveness and usability over Pyomo and JuMP's incredibly broad solver support [2] and there's really no contest. These days, your best option in Python is probably to call JuMP via pyjulia. [1] https://arxiv.org/pdf/1508.01982.pdf https://arxiv.org/pdf/1508.01982.pdf [2] http://jump.readthedocs.io/en/latest/installation.html#getting-solvers http://jump.readthedocs.io/en/latest/installation.html#getti...
      • throwaway7645 9y ago
• 3JPLW 9y ago
  Honest question: did you read the post at all? Even the quick summary addresses this. Chris' whole point is that the biggest benefits of switching to Julia will be felt by the folks that are developing the packages and libraries for others to use. He's advocating that the best way to get you to want to switch isn't incremental language-level features, but rather it's first-in-class domain-specific packages. This is something that will take time, but Julia's language-level features are uniquely positioned to enable the development of such packages. Chris is extremely productive (and definitely an outlier), but in less than two years he managed to coordinate and build a first-in-class ecosystem for differential equations. So he's not advocating for you to switch at all — he's advocating for folks to build the packages (like his) that will get you to want to switch.
  • rattray 9y ago
    Agreed. A relevant excerpt: > But for end users throwing together a 100 line script for a data analysis? I don't think that this crowd will actually see as much of a difference between other scripting languages if the packages in the other languages they are using are sufficiently performant (this isn't always true, but let's assume it is). To people who aren't "pros" in the language, it will probably look like it just has a different syntax. It will be a little faster than vectorized code in other languages if code is in type-stable functions, but most of the differences a user will notice will come from the mixture of features and performance of packages. Because of this, I am not sure if marketing the features of the language is actually the best way to approach the general audience. The general audience will be convinced Julia is worthwhile only by the package offering. (For context, this was after several hundred words of compelling, detailed examples of Julia's language-level advantages). EDIT: the author then goes on to claim > that "lack of packages" isn't really a problem with the ecosystem: you can find a great package that does what you're looking for. with several examples, pointing to poor discoverability, uniqueness, distribution and branding. He also recommends more practical, use-case-specific how-to guides.
    • sgt101 9y ago
      My experience is that Julia promotes lucid and error free code vs R (especially) and Python. For me this matters more and more due in part to the business impact of my 100 line scripts but more significantly due to the impact of other people's 100 line scripts.
• ssivark 9y ago
  Any time 1-based indexing is mentioned as a shortcoming of Julia reminds me of PG's 'Blub paradox' [1]. > As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub. Julia has offset array indexing (for essentially no additional cost) which is an amazingly useful feature! See eg. [2] Think of this feature as blurring the distinction between data (accessing an array) and computation (calling a function). The fact is that arrays as they are used (contiguous memory location collection) often carry more information than being just a dumb list of arbitrary data, and it's very convenient to expose that in their interface. [1]: http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html [2]: https://julialang.org/blog/2017/04/offset-arrays https://julialang.org/blog/2017/04/offset-arrays
  • 88e282102ae2e5b 9y ago
    I didn't mean to seem petty, it's just that going back and forth between zero-based and one-based languages just added extra mental overhead (admittedly this was with Fortran, so there may have been other issues).
    • chrispeel 9y ago
      I think it crucial to offer at this point the important Julia package TwoBasedIndexing.jl :-) https://github.com/simonster/TwoBasedIndexing.jl https://github.com/simonster/TwoBasedIndexing.jl
    • ssivark 9y ago
      The way I think of it is that different indexing schemes suit different problems. I want to think carefully about the problem domain and use the most convenient convention. For example, when my array stores a time series, I would like the index to correspond to timestamps (and still be performant, so long as my timestamps can be efficiently mapped to memory locations, which is true for affine transformations, for example). When another array stores the Fourier transform of that time series, I would like to access elements by the frequency they correspond to. That stops me from making annoying indexing errors (eg: off-by-1), because the data structure's interface maps nicely to the problem domain. I find that much easier than the cognitive cost that comes with trying to shoehorn a single convention on every situation. But it's difficult to appreciate that when thinking of language constructs divorced from specific problem domains, as one tends to do when typically studying data structures and/or algorithms.
• throwaway7645 9y ago
  Zero based indexing? I find it not too bad to switch between languages that use zero & one based indexing.
  • Bromskloss 9y ago
    It's not that one is incapable of doing it. It's that one way is wrong and the other is right!
    • goatlover 9y ago
      Right, so zero-based is wrong unless one is using a language close to assembly.
      • Bromskloss 9y ago
        Which one is "right" and which one is "wrong" (those are strong words I chose somewhat tongue-in-cheek) is a matter of opinion. I have my opinion, and that is that zero is the proper number to start enumerations on, not just in programming, but for everything.
        • goatlover 9y ago
          Sure, but why would zero be the proper number to start counting at outside of zero-based programming languages? Everyone starts counting objects at 1. Zero means an absence of objects to count or list. I count that there are 7 cars in the lot. The first one is 1. If there were 0 cars, I would not have any to enumerate over. If I make a list, I always number the first one as 1 (or A). If there was nothing on my list, I would have nothing (0) to number. Word processors start list numbering at 1. Printers number the first page as 1. Kids are taught to count starting with one, using one finger. I feel like this is only an argument because we have zero-based indexing in mainstream programming languages, although maybe there are some mathematicians who think otherwise?
          • Bromskloss 9y ago
• srean 9y ago
  The same argument applied to the numpy/scipy stack when it was new. R, SAS and a whole slew of other offerings were available at that time. This view of yours as applied to something new is as old as the woods, progress and shifts take place regardless of that. Small mercies actually but some are still writing and maintaining COBOL applications and earning extremely respectable salaries from it. All you are really saying is you will not be an early adopter. That's OK, there will be others. Your non-adoption is unlikely to be a shattering loss for you or for Julia. Regarding whether improvements are minor or significant that's a matter of opinion.
• BeetleB 9y ago
  All of the above were arguments for not switching away from MATLAB 10 years ago, when I switched to Python + NumPy.
• dnautics 9y ago
  At work, we had some code that was written in Python. We needed it to exercise a drive (and had to be I/o bound for the metrics to make sense). I couldn't believe it when after several weeks of coworkers using it I noticed the CPU was pegged at 100%... I rewrote in Julia (which was much more terse and debuggable) and the CPU load dropped to 30%.
• flavio81 9y ago
  You will want to switch as soon as you have to confront the Global Interpreter Lock (GIL)
  • 88e282102ae2e5b 9y ago
    I've confronted it plenty of times, and for my use cases it's not really an issue. Many numpy functions release the GIL, so by keeping my performance-critical stuff in numpy it's just not an issue for me. I sometimes split tasks up with multiprocessing, which is a GIL workaround to be sure, but the memory footprint of extra interpreters is negligible compared to the data I work with.
• strictfp 9y ago
  I also found this effect super annoying when I switched to using a car - having to sell my carriage, get rid of my stable, sell or rent all my horses. All my connections in remote stables were suddenly useless. And I had to sell of all that hay! And oil, I had to get lots of oil! And it's nowhere to be found! Outrageous, I say.