6 ms·

Something about Julia from me. I was big Julia fan using it for last 1.5 year and in my company we are using Julia for data preprocessing and parsing hundred of

Something about Julia from me. I was big Julia fan using it for last 1.5 year and in my company we are using Julia for data preprocessing and parsing hundred of GB's of data. (We also use Python for ML and start to moving julia code to Nim for data prepro) If u consider Julia as ur next language dont use it to any medium to big project outside pure scientific purposes it just isnt ready yet. We have about 3.5k loc in Julia at the moment for our preprocessing service and as much problems as we have with GC in Julia i never saw in any other language: random crashesh, memory leaks, big slowdowns with long running processes and there is a lot of topic on julia forum about that still not solved. Julia is beautiful and easy to use but it is nowhere near stable and/or fast as any other "new" languages (RUST, Nim, Golang, even Swift). And if u consider using Julia for Deep Learning then best performance and stability in this order are: Pytorch, Tensorflow/MXNet/(JAX), Nothing, Flux. Its a long way to go for Flux before it'll be even close in usability considering other flag frameworks.

oxinabox 7y ago
> If u consider Julia as ur next language dont use it to any medium to big project outside pure scientific purposes it just isnt ready yet My employer, Invenia, uses Julia for what I would say is at least a medium sized project. ~200k LOC, ~30 people concurrently working on that code base, literally the core product/system that makes us money. Julia's not perfect, but I would not blanket discount it out of hand for medium projects. Like all technologies the tradeoffs need to be investigated with reference to the task (and team) at hand.
tomerbd 7y ago
I have to admit that over the past 20 years, java is like the most stable robust, huge project, multiple teams, IDE friendly, type safe, infinite tooling and libraries language of choice!
- komuher 7y ago
  Java is good for big projects but not for Machine Learning purposes (Also im C# fanboy so still will prefer C# over Java :D)
- m4r35n357 7y ago
  Not too surprising. Java was _designed_ by ex Unix engineers at a time that the word "engineer" meant something. Now we have the cloud/web stack and _Javascript_. The bar was low enough for Julia to sound good at the start, but it gives the impression of being "designed" "on-the-fly".
  - pjmlp 7y ago
    Luckily in some parts of the world engineer still means something by law, even in informatics, and one is not allowed to use it just because they went through a six weeks programming bootcamp.
  - the_monster 7y ago
    Engineering implies scientific rigour and high intellect. So, no wonder software developers, usually a derided and mistreated collective, just love being called engineer...
    - auxym 7y ago
      Engineering (as in PEng in the US, but my jurisdiction just calls it Engineer) also implies liability, and a risk of losing your license if you fuck up.
- dunefox 7y ago
  Type safe? NPE en masse. IDE friendly? IDE support is only good because you absolutely need an IDE for this language. It's an unwieldy, verbose mess of a language.
  - apta 7y ago
    You need an IDE for any non-trivial project in any language.
    - iamed2 7y ago
      This just isn't true. There's a lot of tooling nowadays that provides IDE-like features in text editors, or via command-line tuning. And different people work differently. Where some people use IDEs, others never leave the terminal and just use e.g. vim/emacs, a build system, and a debugger.
    - hactually 7y ago
      agreed - it's almost as bad as Java/Scala for needing a heavy IDE to make it work
_Wintermute 7y ago
I really like the idea of Julia, and my biggest hope is that it can still replace the use of R in statistics and scientific computing, but I have to agree it is a bit unwieldy in larger projects or as a general programming language and that nim is really promising in this regard.
ced 7y ago
We're using it in a soft-realtime setting to monitor industrial chlorine production, and for us it has been a very pleasant experience overall. Yes, we've had some issues, but similar to other ecosystem IMO, and our support contract with Julia computing helped us in the one case we really couldn't solve ourselves. Julia works really well for power users. There are no huge libraries full of C code like pandas or scipy. Instead there are dozens of small, well-tested packages that fill the same role, all hosted on github. That makes fixing issues so much easier. Granted: outside of numerical/technical computing, the libraries can be lacking (eg. web development) compared to other languages. We're doing it anyway, but it's a more difficult decision.
- logicallee 7y ago
  >and our support contract with Julia computing helped us in the one case we really couldn't solve ourselves. Could you elaborate? What was the problem and what was the solution?
  - ced 7y ago
    See this issue: https://github.com/JuliaLang/julia/issues/30653#issuecomment-475836003 https://github.com/JuliaLang/julia/issues/30653#issuecomment... Memory leak. As explained, it's not really Julia's fault. This I also found amazing: https://github.com/JuliaLang/julia/issues/28726 https://github.com/JuliaLang/julia/issues/28726 . It took 4 hours to go from bug report about bad compiler code generation to a patch.
    - nh2 7y ago
      It seems that Julia is simply discovering the effects of memory fragmentation and the corresponding glibc malloc tunables. I have commented with some details there: https://github.com/JuliaLang/julia/issues/30653#issuecomment-603592336 https://github.com/JuliaLang/julia/issues/30653#issuecomment...
- komuher 7y ago
  How much LOC did u guys have at the moment, and how much data are u monitoring? For smaller problems Julia was working perfect for us.
logicchains 7y ago
> near stable and/or fast as any other "new" languages (RUST, Nim, Golang, even Swift) This is not my experience, at least for numeric code. It generates faster code than Golang, because it uses an LLVM backend (and actually supports macros and parametric polymorphism, so doesn't need to do the work at runtime), faster numeric code than Rust via @inbounds and @simd annotations (way more work to disable bounds checks in Rust), and faster than Swift because it doesn't have pervasive reference counting that can sneak in and destroy performance.
- komuher 7y ago
  For numeric code sure but its because Julia are using BLAS (or any other instruction for CPU/GPU u give to LLVM). Julia (no BLAS) -> In matmul its on pair with Golang and Swift and a bit slower then RUST and Nim. If u need BLAS then just use lib with that :)
  - ViralBShah 7y ago
    Providing a well functioning BLAS experience in a language is a non-trivial task. Saying "just use lib" seems to severely understate this. How many go or swift or Nim issues discuss matmul vs the Julia repo?
    - komuher 7y ago
      Nim have Arraymancer https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer sure it isn't easy but still most of top dogs have some libs supporting BLAS/CUDA etc.
  - the_monster 7y ago
    Please double check your grammar before posting
    - komuher 7y ago
      Sorry im not native :( I tried my best
xiphias2 7y ago
I'm also fighting the GC all the time, although I love Julia. I wish I could tell the compiler to compile a function only if it can do it without using memory allocations. Often I would rather see a compilation error than the GC making performance of a function unpredictable.
- KenoFischer 7y ago
  Yep, you're not alone in wanting tighter control over memory allocation and GC. It's something that we're planning to work on this year.
  - xiphias2 7y ago
    It's really cool. Of course borrow checker would be helpful, but I guess that's too hard to implement, that's why something like this would be a good balance.
  - snicker7 7y ago
    BTW, does the core Julia team maintain an active list of this and other priorities?
    - iamed2 7y ago
      I have not seen one, but I would really like to! It's one thing I envy about some other open-source language projects. I also think Julia would benefit a lot from a list of "we want these but haven't had time for them". People like contributing things they know are desired, and with Julia it's hard to know whether people want or will accept something until you make the PR.
cmrdporcupine 7y ago
Sounds like a great opportunity to contribute fixes and improvements.
deleted 7y ago
[deleted]
ViralBShah 7y ago
Sorry to hear that. I do hope you filed issues or chimed in on existing issues. On deep learning, Flux has a different flexible approach. This is great for certain people. For people who want a library of existing models, Julia can already call TensorFlow and we didn't see the need to reinvent Tensorflow.

oxinabox 7y ago

> If u consider Julia as ur next language dont use it to any medium to big project outside pure scientific purposes it just isnt ready yet My employer, Invenia, uses Julia for what I would say is at least a medium sized project. ~200k LOC, ~30 people concurrently working on that code base, literally the core product/system that makes us money. Julia's not perfect, but I would not blanket discount it out of hand for medium projects. Like all technologies the tradeoffs need to be investigated with reference to the task (and team) at hand.

tomerbd 7y ago

I have to admit that over the past 20 years, java is like the most stable robust, huge project, multiple teams, IDE friendly, type safe, infinite tooling and libraries language of choice!

komuher 7y ago
Java is good for big projects but not for Machine Learning purposes (Also im C# fanboy so still will prefer C# over Java :D)
m4r35n357 7y ago
Not too surprising. Java was _designed_ by ex Unix engineers at a time that the word "engineer" meant something. Now we have the cloud/web stack and _Javascript_. The bar was low enough for Julia to sound good at the start, but it gives the impression of being "designed" "on-the-fly".
- pjmlp 7y ago
  Luckily in some parts of the world engineer still means something by law, even in informatics, and one is not allowed to use it just because they went through a six weeks programming bootcamp.
- the_monster 7y ago
  Engineering implies scientific rigour and high intellect. So, no wonder software developers, usually a derided and mistreated collective, just love being called engineer...
  - auxym 7y ago
    Engineering (as in PEng in the US, but my jurisdiction just calls it Engineer) also implies liability, and a risk of losing your license if you fuck up.
dunefox 7y ago
Type safe? NPE en masse. IDE friendly? IDE support is only good because you absolutely need an IDE for this language. It's an unwieldy, verbose mess of a language.
- apta 7y ago
  You need an IDE for any non-trivial project in any language.
  - iamed2 7y ago
    This just isn't true. There's a lot of tooling nowadays that provides IDE-like features in text editors, or via command-line tuning. And different people work differently. Where some people use IDEs, others never leave the terminal and just use e.g. vim/emacs, a build system, and a debugger.
  - hactually 7y ago
    agreed - it's almost as bad as Java/Scala for needing a heavy IDE to make it work

_Wintermute 7y ago

I really like the idea of Julia, and my biggest hope is that it can still replace the use of R in statistics and scientific computing, but I have to agree it is a bit unwieldy in larger projects or as a general programming language and that nim is really promising in this regard.

ced 7y ago

We're using it in a soft-realtime setting to monitor industrial chlorine production, and for us it has been a very pleasant experience overall. Yes, we've had some issues, but similar to other ecosystem IMO, and our support contract with Julia computing helped us in the one case we really couldn't solve ourselves. Julia works really well for power users. There are no huge libraries full of C code like pandas or scipy. Instead there are dozens of small, well-tested packages that fill the same role, all hosted on github. That makes fixing issues so much easier. Granted: outside of numerical/technical computing, the libraries can be lacking (eg. web development) compared to other languages. We're doing it anyway, but it's a more difficult decision.

logicallee 7y ago
>and our support contract with Julia computing helped us in the one case we really couldn't solve ourselves. Could you elaborate? What was the problem and what was the solution?
- ced 7y ago
  See this issue: https://github.com/JuliaLang/julia/issues/30653#issuecomment-475836003 https://github.com/JuliaLang/julia/issues/30653#issuecomment... Memory leak. As explained, it's not really Julia's fault. This I also found amazing: https://github.com/JuliaLang/julia/issues/28726 https://github.com/JuliaLang/julia/issues/28726 . It took 4 hours to go from bug report about bad compiler code generation to a patch.
  - nh2 7y ago
    It seems that Julia is simply discovering the effects of memory fragmentation and the corresponding glibc malloc tunables. I have commented with some details there: https://github.com/JuliaLang/julia/issues/30653#issuecomment-603592336 https://github.com/JuliaLang/julia/issues/30653#issuecomment...
komuher 7y ago
How much LOC did u guys have at the moment, and how much data are u monitoring? For smaller problems Julia was working perfect for us.

logicchains 7y ago

> near stable and/or fast as any other "new" languages (RUST, Nim, Golang, even Swift) This is not my experience, at least for numeric code. It generates faster code than Golang, because it uses an LLVM backend (and actually supports macros and parametric polymorphism, so doesn't need to do the work at runtime), faster numeric code than Rust via @inbounds and @simd annotations (way more work to disable bounds checks in Rust), and faster than Swift because it doesn't have pervasive reference counting that can sneak in and destroy performance.

komuher 7y ago
For numeric code sure but its because Julia are using BLAS (or any other instruction for CPU/GPU u give to LLVM). Julia (no BLAS) -> In matmul its on pair with Golang and Swift and a bit slower then RUST and Nim. If u need BLAS then just use lib with that :)
- ViralBShah 7y ago
  Providing a well functioning BLAS experience in a language is a non-trivial task. Saying "just use lib" seems to severely understate this. How many go or swift or Nim issues discuss matmul vs the Julia repo?
  - komuher 7y ago
    Nim have Arraymancer https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer sure it isn't easy but still most of top dogs have some libs supporting BLAS/CUDA etc.
- the_monster 7y ago
  Please double check your grammar before posting
  - komuher 7y ago
    Sorry im not native :( I tried my best

xiphias2 7y ago

I'm also fighting the GC all the time, although I love Julia. I wish I could tell the compiler to compile a function only if it can do it without using memory allocations. Often I would rather see a compilation error than the GC making performance of a function unpredictable.

KenoFischer 7y ago
Yep, you're not alone in wanting tighter control over memory allocation and GC. It's something that we're planning to work on this year.
- xiphias2 7y ago
  It's really cool. Of course borrow checker would be helpful, but I guess that's too hard to implement, that's why something like this would be a good balance.
- snicker7 7y ago
  BTW, does the core Julia team maintain an active list of this and other priorities?
  - iamed2 7y ago
    I have not seen one, but I would really like to! It's one thing I envy about some other open-source language projects. I also think Julia would benefit a lot from a list of "we want these but haven't had time for them". People like contributing things they know are desired, and with Julia it's hard to know whether people want or will accept something until you make the PR.

cmrdporcupine 7y ago

Sounds like a great opportunity to contribute fixes and improvements.

deleted 7y ago

[deleted]

ViralBShah 7y ago

Sorry to hear that. I do hope you filed issues or chimed in on existing issues. On deep learning, Flux has a different flexible approach. This is great for certain people. For people who want a library of existing models, Julia can already call TensorFlow and we didn't see the need to reinvent Tensorflow.