4 ms·

I keep my eye on Julia, but still did not have a chance to dig into it. I would be interested to learn more about its place and capabilities: - I want to build

I keep my eye on Julia, but still did not have a chance to dig into it. I would be interested to learn more about its place and capabilities: - I want to build an new (yet another) distributed data processing framework, say, Hadoop or Spark Next, generic or for a particular industry, and I do not want to use Java or C++. Can Julia be a feasible choice for a distributed computation like Spark, for distributed file system like HDFS, for resource allocator like Yarn? Can it be a good choice for a database engine, or its place is in layers above the engine - say a layer to support aggregation and computations? Or it's not a right choice comparing with Rust, C++, or Java for core systems, and it's better to stick with it just for computations on the top of those core systems?

The_rationalist 7y ago
What about kotlin? You would benefit from existing java/scala libraries.
- playing_colours 7y ago
  My question was more about figuring out if Julia can be a good candidate, not like writing off Java or C++. Something when you build things for fun, for personal project, for prototype, or research project. If I am a Chief Engineer / CTO of a company, and need to choose a language for Yet Another data platform, I would definitely look first at Java / Kotlin / Scala for implementation - they are a proven pragmatic choice for such tasks. If there are memory and garbage collection concerns of the former, I would look at C++ or Rust (still risky for staffing). What I like in Julia is its expressiveness, and that it was built for writing maths and computations. So it's a great candidate for top level libraries above core components - it may be a reasonable choice for a business now. But Julia promises good performance as well. Therefore, if one does not have very strict requirements to avoid GC it may look like an interesting idea to stick with the same language for the core components. Here was my question above :)
xiaodai 7y ago
Yes to distributed data processing. JuliaDB.jl has promise.
socialdemocrat 7y ago
I don't know the domain you describe, but you have e.g. JuliaDB which has quite good performance from what I read. https://juliadb.org https://juliadb.org JuliaDB is about storing and retrieving Julia data end to end, although it works with e.g. CSV files etc as far as I know. The benefit of using Julia over something like Java or C++ is that user defined function can easily be added and they are JIT compiled for maximum performance. The main consideration with respect to using Julia is about whether your domain has problems with JIT compilation or not. Anything that is started and shut down frequently and only runs for a short time will not work well with a JIT based system like Julia. Also places where you need fine control over latency such as computer games or real time systems may not be suitable for Julia. Then I suspect Rust or Swift would be better. Other than that Julia is good for almost anything. I has great support for parallelism, concurrency and crunch numbers really fast. While C++ and Rust may beat Julia in performance I think you should be able to outperform Java, because Julia is designed much more with performance in mind than Java. E.g. you have better control over memory layout, cache misses etc in Julia than in Java.
- pjmlp 7y ago
  It was really a bad decision to design Java without value types, given the already existing plethora of GC enabled systems languages since the mid-70's. However that will be eventually fixed, and then we won't be able to bash Java any longer for lack of value types. In what GC languages are concerned, it would be more interesting to compare Julia's generated code against .NET Core languages, D (specially ldc), Nim.
  - BubRoss 7y ago
    Java has been around for multiple decades but the types are just around the corner? That reminds me of static compilation, which was also coming soon for 20 years.
    - pjmlp 7y ago
      AOT compilation has been an option to Java shops willing to buy third party commercial JVMs since around 2000. You can play around with value types experimental releases already. https://jdk.java.net/valhalla/ https://jdk.java.net/valhalla/ Adopting value types, while keeping 20+ year old jars working without any kind of changes is an engineering feat. I bet that Java will get value types before Go gets its generics, if ever.
oxinabox 7y ago
There is definately space for compeditors to JuliaDB. I think its shown the area can work. But some competition to drive innovation is a win for everyone

The_rationalist 7y ago

What about kotlin? You would benefit from existing java/scala libraries.

playing_colours 7y ago
My question was more about figuring out if Julia can be a good candidate, not like writing off Java or C++. Something when you build things for fun, for personal project, for prototype, or research project. If I am a Chief Engineer / CTO of a company, and need to choose a language for Yet Another data platform, I would definitely look first at Java / Kotlin / Scala for implementation - they are a proven pragmatic choice for such tasks. If there are memory and garbage collection concerns of the former, I would look at C++ or Rust (still risky for staffing). What I like in Julia is its expressiveness, and that it was built for writing maths and computations. So it's a great candidate for top level libraries above core components - it may be a reasonable choice for a business now. But Julia promises good performance as well. Therefore, if one does not have very strict requirements to avoid GC it may look like an interesting idea to stick with the same language for the core components. Here was my question above :)

xiaodai 7y ago

Yes to distributed data processing. JuliaDB.jl has promise.

socialdemocrat 7y ago

I don't know the domain you describe, but you have e.g. JuliaDB which has quite good performance from what I read. https://juliadb.org https://juliadb.org JuliaDB is about storing and retrieving Julia data end to end, although it works with e.g. CSV files etc as far as I know. The benefit of using Julia over something like Java or C++ is that user defined function can easily be added and they are JIT compiled for maximum performance. The main consideration with respect to using Julia is about whether your domain has problems with JIT compilation or not. Anything that is started and shut down frequently and only runs for a short time will not work well with a JIT based system like Julia. Also places where you need fine control over latency such as computer games or real time systems may not be suitable for Julia. Then I suspect Rust or Swift would be better. Other than that Julia is good for almost anything. I has great support for parallelism, concurrency and crunch numbers really fast. While C++ and Rust may beat Julia in performance I think you should be able to outperform Java, because Julia is designed much more with performance in mind than Java. E.g. you have better control over memory layout, cache misses etc in Julia than in Java.

pjmlp 7y ago
It was really a bad decision to design Java without value types, given the already existing plethora of GC enabled systems languages since the mid-70's. However that will be eventually fixed, and then we won't be able to bash Java any longer for lack of value types. In what GC languages are concerned, it would be more interesting to compare Julia's generated code against .NET Core languages, D (specially ldc), Nim.
- BubRoss 7y ago
  Java has been around for multiple decades but the types are just around the corner? That reminds me of static compilation, which was also coming soon for 20 years.
  - pjmlp 7y ago
    AOT compilation has been an option to Java shops willing to buy third party commercial JVMs since around 2000. You can play around with value types experimental releases already. https://jdk.java.net/valhalla/ https://jdk.java.net/valhalla/ Adopting value types, while keeping 20+ year old jars working without any kind of changes is an engineering feat. I bet that Java will get value types before Go gets its generics, if ever.

oxinabox 7y ago

There is definately space for compeditors to JuliaDB. I think its shown the area can work. But some competition to drive innovation is a win for everyone