6 ms·

A parallel recommendation engine in Julia

139 points by one-more-minute 10y ago

StreamBright 10y ago
This was written 22 Apr 2016, still pretty good read.
- coldtea 10y ago
  Still? Because being merely 6 months is supposed to date a programming article?
  - optimuspaul 10y ago
    It was ancient history in July. It's a fast paced word my friend.
  - StreamBright 10y ago
    You are on HackerNews, the site name might suggest that you are getting news. news n(y)o͞oz/Submit noun newly received or noteworthy information, especially about recent or important events.
    - adrianN 10y ago
      From the guidelines: What to Submit On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity. Off-Topic: Most stories about politics, or crime, or sports, unless they're evidence of some interesting new phenomenon. Videos of pratfalls or disasters, or cute animal pictures. If they'd cover it on TV news, it's probably off-topic.
  - geodel 10y ago
    It depends. Specially if it has anything to do with Javascript related framework. In that world I think best practices and technologies change weekly.
  - the_duke 10y ago
    It's called sarcasm, people.
  - ellisv 10y ago
    I think the point was to note that this isn't a new article. Also Julia has had several bug fixes and a minor version bump (with some major changes) since the article was written.
  - ChrisRackauckas 10y ago
    6 months is a long time ago in Julia land. Since then, the standard Julia install is a minor version higher which makes -O3 optimization standard, has . syntax for automatic fused broadcasts (simple loop fusing for MATLAB-style vectorization), anonymous functions are orders of magnitude faster, Base has been slimmed and many new organizations like JuliaMath and JuliaDiffEq have been developed with new packages to enhance performance for these kinds of things. In Julia you can program really fast so the language (written in Julia) and the package ecosystem evolves fast as well. Code from 6 months ago is recognizably different (at least right now).
    - coldtea 10y ago
      >months is a long time ago in Julia land. Regarding minutiae such as these maybe. But I've been following Julia on and off for close to 5 years, and it's mostly crickets even among "major" versions (e.g 0.2 to 0.3. to 0.4 etc). That said are little better/faster/picking up this year.
mikestaszel 10y ago
Does anyone have a link to the code that was used for the comparison?
- yarapavan 10y ago
  https://github.com/abhijithch/RecSys.jl https://github.com/abhijithch/RecSys.jl should have it
kynights 10y ago
Lets focus on relevancy instead.
yarapavan 10y ago
Impressive results! Congrats Julia team.
shipman05 10y ago
Why test distributed computing technologies on a single core? Am I missing something?
- optimali 10y ago
  The experiments were conducted on a 30 core Intel Xeon machine with 132 GB memory and 2 hyperthreads per core
  - qznc 10y ago
    Probably meant "on a single machine".
  - shipman05 10y ago
    But the "--master local[1]" setting they're using for Spark will run it on a single thread. And, in the article they state "The algorithm took around 500 seconds to train on the NETFLIX dataset on a SINGLE processor, which is good for data as large as 1 billion ratings." -emphasis mine
    - optimali 10y ago
      edit: local[1] has been updated to local[N], thank you for the update! Ok thanks, I didn't know that's what "local[1]" did, so the more relevant comparison would be with --master local[30]? The algorithm took around 500 seconds to train on the NETFLIX dataset on a SINGLE processor, which is good for data as large as 1 billion ratings. - this is from the sequential portion of the test, the parallel portion is the next section.
      - ViralBShah 10y ago
        That was a typo in the blog post. If you look at the graph, with more cores, spark gets faster as does Julia. The typo is now fixed.
        shipman05 10y ago
minimaxir 10y ago
The Spark comparison, given the April 2016 posting of the article, was likely done with Spark 1.6. Spark 2.0, released in July, added significant performance improvements (https://docs.cloud.databricks.com/docs/latest/sample_applications/04%20Apache%20Spark%202.0%20Examples/03%20Performance%20Apache%20(Spark%202.0%20vs%201.6).html https://docs.cloud.databricks.com/docs/latest/sample_applica...), so it is possible the performance difference may be different nowadays.
- ViralBShah 10y ago
  Quite possible, and would be interesting to see how this stacks up today. I was just glad to see that Julia's parallel computing could out of the box give results comparable to Spark, with the ALS algorithm completely written in Julia without crazy optimized code.
pathsjs 10y ago
The experiments were conducted by invoking spark with flags --master local[1] This means running on a single core. Not really a fair comparison with the Julia multithreaded or multiprocess version
- ViralBShah 10y ago
  I am pretty sure that is a typo in the blog post, since the performance of Spark improves as more cores are used, as does Julia's and they both show similar scaling characteristics. The performance plot would probably be more readable if the y axis was on log scale. The typo is also now fixed.
jlrubin 10y ago
Since Viral seems to be responding here... What's going on with Multithread support? I was trying to do a project a while back to make a pure julia mapreduce like engine with a distributed file system, but it was hard to get off the ground due to poor multithreading support. For the uninitiated, Julia has two types of concurrency built in. Tasks, which are co-routines on the same thread and Clusters, which are "separate machines".
- ViralBShah 10y ago
  The multi-threading in Julia is really new and limited. The plan is first to get the whole codebase to be thread-safe and provide some simple parallelism models and then figure out what a good composable multi-threading model could be. For now, since the GC effectively runs only in one thread, you get good speedup with multi-threading if you avoid allocation and thus GC in the parallel code sections. In some cases this is possible, but in many cases it is unnatural. Of course, all this is under heavy development. To build a julia mapreduce engine on a distributed filesystem, Julia's multi-processing should be pretty good though. For simple problems we attempted with packages like Elly.jl, that is what our experience has been.
  - StefanKarpinski 10y ago
    In particular, note that if you were doing this experiment more than a month ago, there was no threading support (except on master); there has been support for distributed computing from the first release. In the new 0.5 release there is support for multithreading, but it's still (as Viral said), experimental.

StreamBright 10y ago

This was written 22 Apr 2016, still pretty good read.

coldtea 10y ago
Still? Because being merely 6 months is supposed to date a programming article?
- optimuspaul 10y ago
  It was ancient history in July. It's a fast paced word my friend.
- StreamBright 10y ago
  You are on HackerNews, the site name might suggest that you are getting news. news n(y)o͞oz/Submit noun newly received or noteworthy information, especially about recent or important events.
  - adrianN 10y ago
    From the guidelines: What to Submit On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity. Off-Topic: Most stories about politics, or crime, or sports, unless they're evidence of some interesting new phenomenon. Videos of pratfalls or disasters, or cute animal pictures. If they'd cover it on TV news, it's probably off-topic.
- geodel 10y ago
  It depends. Specially if it has anything to do with Javascript related framework. In that world I think best practices and technologies change weekly.
- the_duke 10y ago
  It's called sarcasm, people.
- ellisv 10y ago
  I think the point was to note that this isn't a new article. Also Julia has had several bug fixes and a minor version bump (with some major changes) since the article was written.
- ChrisRackauckas 10y ago
  6 months is a long time ago in Julia land. Since then, the standard Julia install is a minor version higher which makes -O3 optimization standard, has . syntax for automatic fused broadcasts (simple loop fusing for MATLAB-style vectorization), anonymous functions are orders of magnitude faster, Base has been slimmed and many new organizations like JuliaMath and JuliaDiffEq have been developed with new packages to enhance performance for these kinds of things. In Julia you can program really fast so the language (written in Julia) and the package ecosystem evolves fast as well. Code from 6 months ago is recognizably different (at least right now).
  - coldtea 10y ago
    >months is a long time ago in Julia land. Regarding minutiae such as these maybe. But I've been following Julia on and off for close to 5 years, and it's mostly crickets even among "major" versions (e.g 0.2 to 0.3. to 0.4 etc). That said are little better/faster/picking up this year.

mikestaszel 10y ago

Does anyone have a link to the code that was used for the comparison?

yarapavan 10y ago
https://github.com/abhijithch/RecSys.jl https://github.com/abhijithch/RecSys.jl should have it

kynights 10y ago

Lets focus on relevancy instead.

yarapavan 10y ago

Impressive results! Congrats Julia team.

shipman05 10y ago

Why test distributed computing technologies on a single core? Am I missing something?

optimali 10y ago
The experiments were conducted on a 30 core Intel Xeon machine with 132 GB memory and 2 hyperthreads per core
- qznc 10y ago
  Probably meant "on a single machine".
- shipman05 10y ago
  But the "--master local[1]" setting they're using for Spark will run it on a single thread. And, in the article they state "The algorithm took around 500 seconds to train on the NETFLIX dataset on a SINGLE processor, which is good for data as large as 1 billion ratings." -emphasis mine
  - optimali 10y ago
    edit: local[1] has been updated to local[N], thank you for the update! Ok thanks, I didn't know that's what "local[1]" did, so the more relevant comparison would be with --master local[30]? The algorithm took around 500 seconds to train on the NETFLIX dataset on a SINGLE processor, which is good for data as large as 1 billion ratings. - this is from the sequential portion of the test, the parallel portion is the next section.
    - ViralBShah 10y ago
      That was a typo in the blog post. If you look at the graph, with more cores, spark gets faster as does Julia. The typo is now fixed.
      - shipman05 10y ago

minimaxir 10y ago

The Spark comparison, given the April 2016 posting of the article, was likely done with Spark 1.6. Spark 2.0, released in July, added significant performance improvements (https://docs.cloud.databricks.com/docs/latest/sample_applications/04%20Apache%20Spark%202.0%20Examples/03%20Performance%20Apache%20(Spark%202.0%20vs%201.6).html https://docs.cloud.databricks.com/docs/latest/sample_applica...), so it is possible the performance difference may be different nowadays.

ViralBShah 10y ago
Quite possible, and would be interesting to see how this stacks up today. I was just glad to see that Julia's parallel computing could out of the box give results comparable to Spark, with the ALS algorithm completely written in Julia without crazy optimized code.

pathsjs 10y ago

The experiments were conducted by invoking spark with flags --master local[1] This means running on a single core. Not really a fair comparison with the Julia multithreaded or multiprocess version

ViralBShah 10y ago
I am pretty sure that is a typo in the blog post, since the performance of Spark improves as more cores are used, as does Julia's and they both show similar scaling characteristics. The performance plot would probably be more readable if the y axis was on log scale. The typo is also now fixed.

jlrubin 10y ago

Since Viral seems to be responding here... What's going on with Multithread support? I was trying to do a project a while back to make a pure julia mapreduce like engine with a distributed file system, but it was hard to get off the ground due to poor multithreading support. For the uninitiated, Julia has two types of concurrency built in. Tasks, which are co-routines on the same thread and Clusters, which are "separate machines".

ViralBShah 10y ago
The multi-threading in Julia is really new and limited. The plan is first to get the whole codebase to be thread-safe and provide some simple parallelism models and then figure out what a good composable multi-threading model could be. For now, since the GC effectively runs only in one thread, you get good speedup with multi-threading if you avoid allocation and thus GC in the parallel code sections. In some cases this is possible, but in many cases it is unnatural. Of course, all this is under heavy development. To build a julia mapreduce engine on a distributed filesystem, Julia's multi-processing should be pretty good though. For simple problems we attempted with packages like Elly.jl, that is what our experience has been.
- StefanKarpinski 10y ago
  In particular, note that if you were doing this experiment more than a month ago, there was no threading support (except on master); there has been support for distributed computing from the first release. In the new 0.5 release there is support for multithreading, but it's still (as Viral said), experimental.