4 ms·

5 years ago I thought for sure I would be using Julia today. (I mostly use R) I have found that news tools have come into play that I haven't really felt the ne

by baldfat 10y ago

5 years ago I thought for sure I would be using Julia today. (I mostly use R) I have found that news tools have come into play that I haven't really felt the need for a faster language then R for my work. I'm interested in people's everyday use of Julia and how it has impacted your workflow. I don't work with "Big Data" most of my data sets are bellow 100k in size. Anyone using Julia for medium and small data sets?

deleted 10y ago
[deleted]
dreww2 10y ago
I thought the same thing as well. Unfortunately, there is still a bit of work to be done on the Julia side. The tidyverse ecosystem in R has relentlessly focused on ease of use and package compatibility, such that everything "fits" together in a logical way. I can't emphasize enough how much of a difference this makes in being productive with a language. Many of the things I do on a regular basis in R are actually rather mundane, and are possible to do in Julia. But it's just much more of a pleasure to do them in R. Fortunately the DataFrames ecosystem and related packages (like Query.jl and StructuredQueries.jl) seem to be headed in the right direction.
- daemonomania 10y ago
  As the maintainer of StructuredQueries.jl, I appreciate the recognition :) The JuliaStats community is indeed aiming for a more integrated ecosystem. It will take time, but I agree with you that our general direction is heartening.
  - dreww2 10y ago
    Of all the Julia packages out there I am most excited about StructuredQueries :) Please keep up the great work, things look great so far.
n00b101 10y ago
I have found that news tools have come into play that I haven't really felt the need for a faster language then R for my work Could you elaborate? Which new tools?
- baldfat 10y ago
  Hadley verse in general. dplyr/tidyr and piping %>% has made things so seemless that I would hate to lose them for the ability to run faster. I swear I have forgotten 75% of what I learned in base R because these libraries (Outside of data.table) just work so well with one another. RStudio has also been a God send for me. RMarkdown and now R Notebooks just amaze me. Then I work in a very MS Office environment that ReportRs (http://davidgohel.github.io/ReporteRs/ http://davidgohel.github.io/ReporteRs/) is the most under valued library right now. I didn't have these things 5 years ago. (RStudio was initially released Feb 2011 but didn't start with it till 2013.
StefanKarpinski 10y ago
Not to be snarky, but we announced Julia in Feb 2012, which is less than 5 years ago. Unless you happened to be in the MIT class that used Julia in the Fall of 2011, you're probably overestimating how long you've been following the project. If you're comfortable using R and the tooling and performance it offers, you should probably keep using it. Julia isn't just about big data, but it does tend to appeal to people who are struggling with problems that are sufficiently hard that their old tools left them in a world of pain. That pain may comes from size, complexity, CPU-intensiveness, or need for more language expressiveness (a hard thing to define). In particular, Julia offers a unique combination of productivity and speed for numerical work that can't be found anywhere else.
- baldfat 10y ago
  Okay 4.5 years :) I really was wowed by the promise of Julia and the road map you presented for it, great job by the way. I just am surprised I haven't picked it up yet personally. I have a feeling that my R ecosystem wouldn't be as strong as it is right now if it wasn't for the friendly Julia competition you have provided. I guess I am seeking the one "pet project" that would get me to jump in and give Julia a test drive. Thanks for all your work even though I don't directly benefit from your work.
kkylin 10y ago
I've used Julia for prototyping Monte Carlo and data assimilation algorithms, and also solving simple (1d) nonlinear PDEs (Fourier-spectral code with exponential RK timestepping -- not very complicated as these things go). It's also more or less completely replaced Matlab for me in terms of plotting and visualization (most of what I need is very simple). So far it has been very pleasant, so long as one remembers that breaking changes can and do occur between versions. I generally find Julia to be a more expressive and fun language to program in than R or Python. It may be my background -- I've done a fair bit of work in Scheme, and in many ways Julia has lots of what I liked about Lisp in it. I don't need all the numerics and stats libraries that R or Python have; what little I need is easy to cobble together very quickly in Julia or has already been implemented. And the FFI is easy to use; calling C is pretty straightforward.
- baldfat 10y ago
  I taught myself Racket and that was the single biggest changer in terms of my programming in R and enjoyment of the language. After digging around I learned that scheme was a major influence in R's creation. Your comment now makes me excited to give Julia a spin.
  - kkylin 10y ago
    I should add that one of the things I really like about Julia is having closures and higher-order functions, and also lambda. Unlike other Schemers switching to Julia, I don't miss not having tail recursion (if I was writing the kind of code that's more naturally recursive, I likely would have used Scheme / Lisp to start with). About the only thing I have a hard time wrapping my head around is writing macros in Julia -- CL-style macros are somehow a lot easier to think about. But this is likely to change with more experience. Also, I haven't needed Julia macros very oten.
papaf 10y ago
Anyone using Julia for medium and small data sets? I have used Julia with a 50GB dataset for feature extraction while I was waiting for R to process the same dataset (10 minutes to an hour depending on the function). I actually learned some Julia while waiting for R. For what I was doing, Julia felt roughly 100 times faster (most delays under a minute). However, I did not know about dplyr at the time and I will certainly try it in the next project. Although, I am skeptical that using fast libraries is a silver bullet since, at some point, the data will pass through some slow custom R code that I wrote myself. I prefer to use Julia - the API feels cleaner and more manageable, I don't suffer from subtle type errors, and there is also the possibility of running things in parallel from the same repl. I also had no memory leaks -- I suffered a lot from the bloat caused by my use of R closures. That said, the power and output quality of ggplot means that I will probably stay with R for plotting. There will also be times when CRAN has the only library that does something we need.
- baldfat 10y ago
  dplyr is faster, BUT not that much faster. dplyr advantage is the way it works for programming. Really a lot easier and more consistent way to think about working with your data. data.table is fast for your project I would imagine.

by baldfat 10y ago

deleted 10y ago

[deleted]

dreww2 10y ago

I thought the same thing as well. Unfortunately, there is still a bit of work to be done on the Julia side. The tidyverse ecosystem in R has relentlessly focused on ease of use and package compatibility, such that everything "fits" together in a logical way. I can't emphasize enough how much of a difference this makes in being productive with a language. Many of the things I do on a regular basis in R are actually rather mundane, and are possible to do in Julia. But it's just much more of a pleasure to do them in R. Fortunately the DataFrames ecosystem and related packages (like Query.jl and StructuredQueries.jl) seem to be headed in the right direction.

daemonomania 10y ago
As the maintainer of StructuredQueries.jl, I appreciate the recognition :) The JuliaStats community is indeed aiming for a more integrated ecosystem. It will take time, but I agree with you that our general direction is heartening.
- dreww2 10y ago
  Of all the Julia packages out there I am most excited about StructuredQueries :) Please keep up the great work, things look great so far.

n00b101 10y ago

I have found that news tools have come into play that I haven't really felt the need for a faster language then R for my work Could you elaborate? Which new tools?

baldfat 10y ago
Hadley verse in general. dplyr/tidyr and piping %>% has made things so seemless that I would hate to lose them for the ability to run faster. I swear I have forgotten 75% of what I learned in base R because these libraries (Outside of data.table) just work so well with one another. RStudio has also been a God send for me. RMarkdown and now R Notebooks just amaze me. Then I work in a very MS Office environment that ReportRs (http://davidgohel.github.io/ReporteRs/ http://davidgohel.github.io/ReporteRs/) is the most under valued library right now. I didn't have these things 5 years ago. (RStudio was initially released Feb 2011 but didn't start with it till 2013.

StefanKarpinski 10y ago

Not to be snarky, but we announced Julia in Feb 2012, which is less than 5 years ago. Unless you happened to be in the MIT class that used Julia in the Fall of 2011, you're probably overestimating how long you've been following the project. If you're comfortable using R and the tooling and performance it offers, you should probably keep using it. Julia isn't just about big data, but it does tend to appeal to people who are struggling with problems that are sufficiently hard that their old tools left them in a world of pain. That pain may comes from size, complexity, CPU-intensiveness, or need for more language expressiveness (a hard thing to define). In particular, Julia offers a unique combination of productivity and speed for numerical work that can't be found anywhere else.

baldfat 10y ago
Okay 4.5 years :) I really was wowed by the promise of Julia and the road map you presented for it, great job by the way. I just am surprised I haven't picked it up yet personally. I have a feeling that my R ecosystem wouldn't be as strong as it is right now if it wasn't for the friendly Julia competition you have provided. I guess I am seeking the one "pet project" that would get me to jump in and give Julia a test drive. Thanks for all your work even though I don't directly benefit from your work.

kkylin 10y ago

I've used Julia for prototyping Monte Carlo and data assimilation algorithms, and also solving simple (1d) nonlinear PDEs (Fourier-spectral code with exponential RK timestepping -- not very complicated as these things go). It's also more or less completely replaced Matlab for me in terms of plotting and visualization (most of what I need is very simple). So far it has been very pleasant, so long as one remembers that breaking changes can and do occur between versions. I generally find Julia to be a more expressive and fun language to program in than R or Python. It may be my background -- I've done a fair bit of work in Scheme, and in many ways Julia has lots of what I liked about Lisp in it. I don't need all the numerics and stats libraries that R or Python have; what little I need is easy to cobble together very quickly in Julia or has already been implemented. And the FFI is easy to use; calling C is pretty straightforward.

baldfat 10y ago
I taught myself Racket and that was the single biggest changer in terms of my programming in R and enjoyment of the language. After digging around I learned that scheme was a major influence in R's creation. Your comment now makes me excited to give Julia a spin.
- kkylin 10y ago
  I should add that one of the things I really like about Julia is having closures and higher-order functions, and also lambda. Unlike other Schemers switching to Julia, I don't miss not having tail recursion (if I was writing the kind of code that's more naturally recursive, I likely would have used Scheme / Lisp to start with). About the only thing I have a hard time wrapping my head around is writing macros in Julia -- CL-style macros are somehow a lot easier to think about. But this is likely to change with more experience. Also, I haven't needed Julia macros very oten.

papaf 10y ago

Anyone using Julia for medium and small data sets? I have used Julia with a 50GB dataset for feature extraction while I was waiting for R to process the same dataset (10 minutes to an hour depending on the function). I actually learned some Julia while waiting for R. For what I was doing, Julia felt roughly 100 times faster (most delays under a minute). However, I did not know about dplyr at the time and I will certainly try it in the next project. Although, I am skeptical that using fast libraries is a silver bullet since, at some point, the data will pass through some slow custom R code that I wrote myself. I prefer to use Julia - the API feels cleaner and more manageable, I don't suffer from subtle type errors, and there is also the possibility of running things in parallel from the same repl. I also had no memory leaks -- I suffered a lot from the bloat caused by my use of R closures. That said, the power and output quality of ggplot means that I will probably stay with R for plotting. There will also be times when CRAN has the only library that does something we need.

baldfat 10y ago
dplyr is faster, BUT not that much faster. dplyr advantage is the way it works for programming. Really a lot easier and more consistent way to think about working with your data. data.table is fast for your project I would imagine.