6 ms·

Open Big Data Computing with Julia

70 points by astrieanna 13y ago

pjmlp 13y ago
Quite nice to see Julia taking off.
glifchits 13y ago
As a beginner in scientific/data computing, is it worth building a foundation in Julia or do with some other tool instead? I'm comfortable in Python but I don't know if its best to teach myself how to use NumPy and friends when I have the opportunity to immerse myself in this brand new shiny technology.
- sanskritabelt 13y ago
  If you're a beginner, stick with python, or matlab.
- acidflask 13y ago
  It depends on what kind of user you are. If you are comfortable with the risks and rewards of being an early adopter using bleeding-edge tools, then by all means try Julia. If you need the stability and functionality of a mature platform, then you may not want to learn Julia as your first tool.
- burntsushi 13y ago
  It depends. There is a certain allure to working with a shiny new technology. It's sort of like an adventure, because not many have paved a strong path forward yet. But not everyone likes an adventure. Sometimes there will be huge roadblocks that don't have obvious solutions; especially for the beginner. On the other hand, the scientific computing crowd for Python/NumPy is well established and there will be plenty of people who have probably come across any problem that you will see. There is already an established ecosystem of tools that you'll be able to draw from, and it's likely that many of them have been well tested and are reliable. The decision is really up to you. Do you have the time and desire to go on adventure? If so, maybe Julia would be a nice pick. Are you short on time or weary of adventure? Then maybe Python/NumPy would be a better pick.
- trthatcher 13y ago
  I don't think it will be a problem to do both. It doesn't take long to learn Julia well enough to write a Julia package that others will use. For me, that was about 3 weeks @ 10-15 hours a week, and I'm no programmer. It's a small language and it's quite readable. I'd reccomend going through the Julia manual; it doesn't take long and only requires some diligence. The problem is the lack of packages/libraries and lots of documentation is still missing. That, and the packages that do exist with multiple contributors are still in disagreement about standards and consistency.
RivieraKid 13y ago
I hope that the web libs will improve, I'm trying to create a very simple web app, but the libs are in alpha stage.
- adambard 13y ago
  I'm not sure Julia is the most appropriate for this application. It's nominally general-purpose, sure, and you can run a web server from it, but you can do that with any language with a socket library and strings. Would you make a website with MATLAB?
  - pjmlp 13y ago
    > Would you make a website with MATLAB? The world is a nail. :)
    - adambard 13y ago
      In retrospect I know several people who probably would serve their website in MATLAB.
  - RivieraKid 13y ago
    I think that if you ignore libs, Julia is a better general-purpose language than other dynamic languages like Python or Ruby. Which is definitely not true for MATLAB :) This is an ultra simple web app, that does some calculations on the backend that would be slow in Python (or I would have to preprocess the data).
  - zhemao 13y ago
    Julia uses libuv, the same library used by Node.JS, for its networking functions. Therefore, given better library support, Julia could be quite suitable for writing web applications. If you wanted to provide a web frontend to some numerical computation (as I think the Sudoku solver mentioned in the article does), such functionality would be quite useful.
- astrieanna 13y ago
  Hi, I'm one of the maintainers of Julia's webstack packages. If you're having problems, please make an issue(s) on github. We would definitely appreciate feedback. I'll try to make sure to respond.
EvanMiller 13y ago
I like Julia because in many ways, it's a better C than C: * You can inspect a function's generated LLVM or (x86, ARM) assembly code from the REPL. (code_llvm or code_native) * You can interface with C libraries without writing any C. That let me wrap a 5 kLoC C library with 100 lines of Julia: https://github.com/WizardMac/DataRead.jl https://github.com/WizardMac/DataRead.jl * You can use the dynamic features of the language to write something quickly, then add type annotations to make it fast later * Certain tuples are compiled to SIMD vectors. In contrast, the only way to access SIMD features in C is to pray that you have one of those "sufficiently smart compilers". * Like C, there's no OO junk and the associated handwringing about where methods belong. There are structures and there are functions, that's it. But then multiple dispatch in Julia gives you the main benefits of OO without all the ontological crap that comes with it. For me, Julia feels like it's simultaneously higher-level and lower-level than C. The deep LLVM integration is fantastic because I can get an idea for how functions will be compiled without having to learn the behemoth that is the modern x86 ISA. (LLVM IR is relatively simple, and its SSA format makes code relatively easy to follow.) Anyway, I only started with Julia recently, but I'm a fan. I should also mention that the members of the developer community are very, very smart. (Most are associated with MIT.) BTW I am starting a Julia meetup in Chicago for folks in the Midwest who want to learn more: http://www.meetup.com/JuliaChicago/ http://www.meetup.com/JuliaChicago/
- pjmlp 13y ago
  > There are structures and there are functions, that's it. But then multiple dispatch in Julia gives you the main benefits of OO without all the ontological crap that comes with it. Multi-dispatch is a form of OO.
  - StefanKarpinski 13y ago
    I would argue that it's the other way around: single-dispatch o.o. is a special case of multiple dispatch.
    - pjmlp 13y ago
      Agree. The point being that OO is not plain Java or C++, there are many concepts around what OO is all about.
mrcactu5 13y ago
I have installed iJulia -- which involved compiling Julia from source. What little bit I've tried with Julia so far is impressive. What is the story anyway? Did a bunch of MIT people decide they needed something faster than R & numpy ? Julia inventor Alan Edelman is an fellow alumnus of Hampshire College Summer Studies in Mathematics (HCSSiM). Scores points in my book. https://en.wikipedia.org/wiki/Alan_Edelman https://en.wikipedia.org/wiki/Alan_Edelman He wrote a terrific paper back in '95 http://arxiv.org/abs/math/9501224 http://arxiv.org/abs/math/9501224 "How many zeros of a random polynomial are real?"
- StefanKarpinski 13y ago
  Alan is a rare and brilliant fellow. Just to clarify, he has been Julia's patron from the very early days, but the actual design and development of the language was done by Jeff Bezanson, Viral B. Shah, and myself. These days it's a collaborative community effort.
army 13y ago
Very cool. I think the point about transcribing equations directly from papers glosses over some of the hard issues with building numeric libraries around floating point precision and numeric stability. It's not that uncommon to see code that would have worked perfectly if double precision floating point numbers were actually real numbers, but which blows up badly once rounding errors happen. Unfortunately there's not really any way to abstract those issues away. In general, I'd hope that any the authors of widely used numeric libraries have carefully thought through these issues.

70 points by astrieanna 13y ago

pjmlp 13y ago

Quite nice to see Julia taking off.

As a beginner in scientific/data computing, is it worth building a foundation in Julia or do with some other tool instead? I'm comfortable in Python but I don't know if its best to teach myself how to use NumPy and friends when I have the opportunity to immerse myself in this brand new shiny technology.

sanskritabelt 13y ago
If you're a beginner, stick with python, or matlab.
acidflask 13y ago
It depends on what kind of user you are. If you are comfortable with the risks and rewards of being an early adopter using bleeding-edge tools, then by all means try Julia. If you need the stability and functionality of a mature platform, then you may not want to learn Julia as your first tool.
burntsushi 13y ago
It depends. There is a certain allure to working with a shiny new technology. It's sort of like an adventure, because not many have paved a strong path forward yet. But not everyone likes an adventure. Sometimes there will be huge roadblocks that don't have obvious solutions; especially for the beginner. On the other hand, the scientific computing crowd for Python/NumPy is well established and there will be plenty of people who have probably come across any problem that you will see. There is already an established ecosystem of tools that you'll be able to draw from, and it's likely that many of them have been well tested and are reliable. The decision is really up to you. Do you have the time and desire to go on adventure? If so, maybe Julia would be a nice pick. Are you short on time or weary of adventure? Then maybe Python/NumPy would be a better pick.
trthatcher 13y ago
I don't think it will be a problem to do both. It doesn't take long to learn Julia well enough to write a Julia package that others will use. For me, that was about 3 weeks @ 10-15 hours a week, and I'm no programmer. It's a small language and it's quite readable. I'd reccomend going through the Julia manual; it doesn't take long and only requires some diligence. The problem is the lack of packages/libraries and lots of documentation is still missing. That, and the packages that do exist with multiple contributors are still in disagreement about standards and consistency.

RivieraKid 13y ago

I hope that the web libs will improve, I'm trying to create a very simple web app, but the libs are in alpha stage.

adambard 13y ago
I'm not sure Julia is the most appropriate for this application. It's nominally general-purpose, sure, and you can run a web server from it, but you can do that with any language with a socket library and strings. Would you make a website with MATLAB?
- pjmlp 13y ago
  > Would you make a website with MATLAB? The world is a nail. :)
  - adambard 13y ago
    In retrospect I know several people who probably would serve their website in MATLAB.
- RivieraKid 13y ago
  I think that if you ignore libs, Julia is a better general-purpose language than other dynamic languages like Python or Ruby. Which is definitely not true for MATLAB :) This is an ultra simple web app, that does some calculations on the backend that would be slow in Python (or I would have to preprocess the data).
- zhemao 13y ago
  Julia uses libuv, the same library used by Node.JS, for its networking functions. Therefore, given better library support, Julia could be quite suitable for writing web applications. If you wanted to provide a web frontend to some numerical computation (as I think the Sudoku solver mentioned in the article does), such functionality would be quite useful.
astrieanna 13y ago
Hi, I'm one of the maintainers of Julia's webstack packages. If you're having problems, please make an issue(s) on github. We would definitely appreciate feedback. I'll try to make sure to respond.

EvanMiller 13y ago

I like Julia because in many ways, it's a better C than C: * You can inspect a function's generated LLVM or (x86, ARM) assembly code from the REPL. (code_llvm or code_native) * You can interface with C libraries without writing any C. That let me wrap a 5 kLoC C library with 100 lines of Julia: https://github.com/WizardMac/DataRead.jl https://github.com/WizardMac/DataRead.jl * You can use the dynamic features of the language to write something quickly, then add type annotations to make it fast later * Certain tuples are compiled to SIMD vectors. In contrast, the only way to access SIMD features in C is to pray that you have one of those "sufficiently smart compilers". * Like C, there's no OO junk and the associated handwringing about where methods belong. There are structures and there are functions, that's it. But then multiple dispatch in Julia gives you the main benefits of OO without all the ontological crap that comes with it. For me, Julia feels like it's simultaneously higher-level and lower-level than C. The deep LLVM integration is fantastic because I can get an idea for how functions will be compiled without having to learn the behemoth that is the modern x86 ISA. (LLVM IR is relatively simple, and its SSA format makes code relatively easy to follow.) Anyway, I only started with Julia recently, but I'm a fan. I should also mention that the members of the developer community are very, very smart. (Most are associated with MIT.) BTW I am starting a Julia meetup in Chicago for folks in the Midwest who want to learn more: http://www.meetup.com/JuliaChicago/ http://www.meetup.com/JuliaChicago/

pjmlp 13y ago
> There are structures and there are functions, that's it. But then multiple dispatch in Julia gives you the main benefits of OO without all the ontological crap that comes with it. Multi-dispatch is a form of OO.
- StefanKarpinski 13y ago
  I would argue that it's the other way around: single-dispatch o.o. is a special case of multiple dispatch.
  - pjmlp 13y ago
    Agree. The point being that OO is not plain Java or C++, there are many concepts around what OO is all about.

mrcactu5 13y ago

I have installed iJulia -- which involved compiling Julia from source. What little bit I've tried with Julia so far is impressive. What is the story anyway? Did a bunch of MIT people decide they needed something faster than R & numpy ? Julia inventor Alan Edelman is an fellow alumnus of Hampshire College Summer Studies in Mathematics (HCSSiM). Scores points in my book. https://en.wikipedia.org/wiki/Alan_Edelman https://en.wikipedia.org/wiki/Alan_Edelman He wrote a terrific paper back in '95 http://arxiv.org/abs/math/9501224 http://arxiv.org/abs/math/9501224 "How many zeros of a random polynomial are real?"

StefanKarpinski 13y ago
Alan is a rare and brilliant fellow. Just to clarify, he has been Julia's patron from the very early days, but the actual design and development of the language was done by Jeff Bezanson, Viral B. Shah, and myself. These days it's a collaborative community effort.

army 13y ago

Very cool. I think the point about transcribing equations directly from papers glosses over some of the hard issues with building numeric libraries around floating point precision and numeric stability. It's not that uncommon to see code that would have worked perfectly if double precision floating point numbers were actually real numbers, but which blows up badly once rounding errors happen. Unfortunately there's not really any way to abstract those issues away. In general, I'd hope that any the authors of widely used numeric libraries have carefully thought through these issues.