7 ms·

Parallelism in Crystal

139 points by sdogruyol 7y ago

faitswulff 7y ago
Aside, but is there a good class or resource for learning about parallelism in computing? I'm reading the blog post and vaguely recalling differences between threading and forking. I regularly encounter the same concepts over and over but they haven't been sticking. A hands-on project would be great.
Doctor_Fegg 7y ago
Delighted with this. I've been working on and off on a shortest-path routing engine in Crystal. Parallelism means I can get performance on a par with C++ even on large graphs, but with the simplicity and readability of Ruby. Full marks to the Crystal devs for tackling the no 1 feature request.
DSingularity 7y ago
Why Crystal over C++?
- penagwin 7y ago
  For very similar reasons that they choose ruby over C++ ?
- ravenstine 7y ago
  Why C++ over assembly?
  - DSingularity 7y ago
    I’m asking because they presented Crystal as Ruby with OOP and high performance. I’m asking because I’m curious, is it the ruby syntax that much better?
    - todd3834 7y ago
      You’ll get different answers depending on who you ask but I think most will agree it is much more accessible to new programmers.
    - pmontra 7y ago
      Coming from C (no ++) and Java the first thing I noticed after a month with Ruby (in 2005?) was that I didn't write any for loop. I mean, those useless for (i = 0; i < length (array); i++). They become array.each. Then I loved the blocks and later on the postfix if and unless. It's not perfect, it's got its quirks and metaprogramming sometimes is easier in Python, sometimes not, but overall it's the easiest language I used so far. I never tried Crystal because of the compilation step. I really don't need the extra speed for my sw and no customer asked me Crystal so far. I guess it's going to be a similarly pleasant experience.
      - e12e 7y ago
        > metaprogramming sometimes is easier in Python, sometimes not Now, I've been exposed to more ruby, and more metaprogramming in ruby, than in python - but this stil surprises/intrigues me. What kind of metaprogramming do you find easier in python?
        pmontra 7y ago
        If I'm not wrong I can add a methods like this def m2(): print("nuovo") object.new_method = m2 instead of redefining the class. It's similar to JavaScript. However I don't have a computer at hand now. I can't check the details and what could be done on Ruby.
        d_k_f 7y ago
        True, unfortunately. You can't pass functions around as first class objects, you'll have to use lambdas or procs for that (although even then composition doesn't work like that in Ruby). (No judgement from me on whether that's better/worse/about equal, just confirming.)
  - imtringued 7y ago
    Because you are tying yourself to a specific ISA. This is bad even within the same chip family. If you want to migrate your x86 32 bit code to 64 bit then you have to rewrite everything.
- coldtea 7y ago
  Because it's a much saner, coherent, and easier to use language with less footguns... Now, why not Crystal over C++? Because it's way less mature, with way less documentation, way less resources, way worse tooling, way less libraries, way more implementation bugs, and, if we're realistic, it might never go anywhere...
  - dragonwriter 7y ago
    > Because it's a much saner, coherent, and easier to use language with less footguns... Which for parallelism probably switches the question to “why Crystal over Pony”, since Crystal seems to preserve the key C++ footguns specifically linked to parallelism.
  - Jenz 7y ago
    > and, if we're realistic, it might never go anywhere... I don’t see why this should be the case, unless we as a community decides it will not go anywhere.
    - coldtea 7y ago
      Most languages never get anywhere. Languages of the quality of Smalltalk, and Ada, and Dylan, and Self, and CL have vanquished in the margins, and e.g. even something as celebrated as Haskell does hardly better in the industry (some financial companies, this or that project, and so on), so the chances of a new language like Crystal are slim to begin with...
      - Jenz 7y ago
        Oh yes, been there, done that, created proglangs, and they certainly did not get anywhere. But I feel Crystal has already gained such momentum that it indeed will get somewhere, it’s beyond that phase "to begin with." After all, if any language can get anywhere, why not Crystal?
        coldtea 7y ago
        >But I feel Crystal has already gained such momentum that it indeed will get somewhere, it’s beyond that phase "to begin with." Let's hope so, but I fear that's wishful thinking. What proof do we have of that? It's not used anywhere important ("killer app"), barely makes the rest of the TIOBE index, along with staples like NATURAL, Alice, BBC Basic, Euphoria, Factor, Forth, and Icon, and has no major force like Google or Mozilla backing it. It's only us here in HN who keep hearing about it. Julia has had tons more publicity (even from some major outlets), and it's still going nowhere fast.
- deleted 7y ago
  [deleted]
- odiroot 7y ago
  To me is just Python-like (or actually Ruby-like but close enough) but with much better performance. I still haven't learnt to use it well though. It's a joy to write due to its syntax.
aidenn0 7y ago
I get why N=2 is slower than single-threaded, but why is N=4 more than twice as fast as N=2?
- ricardobeat 7y ago
  If you're looking at the first graph, the explanation is right above - the overhead of synchronization.
  - aidenn0 7y ago
    That's nonsensical. There is surely at least as much synchronization overhead with 4 threads as with 2.
    - jonesetc 7y ago
      If the overhead is X, and the amount of work that can be done ignoring it is Y, you have Y - X. If you double the amount of work that can be down, but overhead stays the same you have 2Y - X, which is more than 2(Y - X) = 2Y - 2X. You're over performing doubling by not having to double the synchronization cost.
    - paulddraper 7y ago
      Not necessarily. There can be overhead introduced between the "1 thread" and ">1 thread" cases.
    - ricardobeat 7y ago
      From 1 -> 2 threads the overhead is introduced, if the cost is somewhat fixed, the gains from 2 -> 4 can be proportionally larger.
jackcodes 7y ago
I’ve been using Crystal on-and-off for a couple of years and have started to use it as my primary language for most non-enterprise work. The current web-app I’m building is primarily written in Crystal and it has been an absolute joy to use, and even if I don’t strictly need parallelism in this project I’m glad they’re pushing though with the big language changes. It’s been so successful at everything I’ve needed it for, I wouldn’t consider going back to either Ruby or similar interpreted languages, regardless whether it’s a little CLI helper or a larger monolith server app. I’ve found my ability to allow the compiler to catch bugs incredibly reliable without feeling bogged down with types. I hadn’t worked much with typed languages previously and as a Ruby developer it allowed me to ‘learn’ types without even feeling like I was doing any learning. It’s still somewhat of an obscure language so I thought it would be helpful to describe my experiences with it.
- h1d 7y ago
  Do you use much external libraries because ecosystem I'd pretty much what makes a language useful after the language being solid by itself. Also, I assume Googleability is pretty much close to zero, you'd have to go to the primary sites (their forum?) to ask questions when stuck?
  - IfOnlyYouKnew 7y ago
    It's obviously far smaller, but I've had luck so far with everything I need being available. What I've seen, mostly Postgres, Redis, and XML/HTML parsing, have also worked flawlessly so far. Having performance improvements of 100x in real-world scenarios is just astonishing. Ruby is still my go-to, especially because debugging and console support are so much better. But everything that gives me time to think gets reimplemented. ["crystal lang" <question>] usually works for google.
  - jackcodes 7y ago
    There's actually a surprising amount available in the stdlib for your typical web project. By way of example, I'd always hated how in Ruby each API wrapper gem would make their own decision on which HTTP client they used, so you could end up pulling in 3 or 4 separate libraries just to make HTTP calls to a 3rd party API. People may disagree but I think this is (was?) largely due to Ruby's Net:HTTP feeling janky to use, at least compared to Faraday and the like. Here, the HTTP modules within the standard library feel right, and I've not needed to reach out for all that much from the ecosystem. But as a simple example I accept that API calls to third party services is low hanging fruit so in terms of finding drivers for lower level protocols I've not had an issue that I'd class as a blocker. There are times where I've had to extend libraries or write a parser for this-and-that (e.g. PostGIS and Paseto, both of which were unused this time round) but I find I need to do this less and less as time goes on. You do need to reach into the forums or gitter on rare occasions but I can count on one hand the number of times I've needed to, and it's typically because I was trying to do something unusual and quite low-level on the HTTP stack - as in something I've never had to do in a non-Crystal project, such as sending a specific packet. I've always found help there too and people are happy to engage.
rishav_sharan 7y ago
I have been using crystal+ vanilla js for my pet project and the sheer joy i get from using this language is something else. 2019 has been a good year for crystal. Basic multithreading is in. Windows support is making good headway. I am optimistic that a working windows compiler would be out by end of year. The last major dragon for crystal to slay would be the compilation time. It still feels unreasonably high. Hopefully that would be a major focus of 2020
- freshbagels 7y ago
  I'm curious as to what your pet project is/if it's available to online to play with.
  - rishav_sharan 7y ago
    The Crystal underbelly is here https://github.com/rishavs/noir https://github.com/rishavs/noir the api is consumed by a vanilla js spa. I talk more about my VanillaJS spa's architecture here https://dev.to/rishavs/making-a-single-page-app-in-ye-good-olde-js-es6-3eng https://dev.to/rishavs/making-a-single-page-app-in-ye-good-o...
rarecoil 7y ago
I just learned about Crystal a couple of weeks ago and it's completely subsumed every new personal project I have into its domain. I have long missed Ruby's syntax and human-friendly feel, but began to prefer the speed I could find in other languages like Go.
squarefoot 7y ago
Waiting badly for an installable ARM port; small boards would benefit a lot from it. Is this planned in the future?
maitredusoi 7y ago
Until crystal get a REPL I would stick to ruby / rails for POC and MVP, then scaling problem "could" arose, so translate it to amber or lucky (Web Framework in crystal) could solve the issue. About Type and Casting Nil, gem like sorbet should be a good replacement in ruby

139 points by sdogruyol 7y ago

Aside, but is there a good class or resource for learning about parallelism in computing? I'm reading the blog post and vaguely recalling differences between threading and forking. I regularly encounter the same concepts over and over but they haven't been sticking. A hands-on project would be great.

Doctor_Fegg 7y ago

Delighted with this. I've been working on and off on a shortest-path routing engine in Crystal. Parallelism means I can get performance on a par with C++ even on large graphs, but with the simplicity and readability of Ruby. Full marks to the Crystal devs for tackling the no 1 feature request.

DSingularity 7y ago

Why Crystal over C++?

penagwin 7y ago
For very similar reasons that they choose ruby over C++ ?
ravenstine 7y ago
Why C++ over assembly?
- DSingularity 7y ago
  I’m asking because they presented Crystal as Ruby with OOP and high performance. I’m asking because I’m curious, is it the ruby syntax that much better?
  - todd3834 7y ago
    You’ll get different answers depending on who you ask but I think most will agree it is much more accessible to new programmers.
  - pmontra 7y ago
    Coming from C (no ++) and Java the first thing I noticed after a month with Ruby (in 2005?) was that I didn't write any for loop. I mean, those useless for (i = 0; i < length (array); i++). They become array.each. Then I loved the blocks and later on the postfix if and unless. It's not perfect, it's got its quirks and metaprogramming sometimes is easier in Python, sometimes not, but overall it's the easiest language I used so far. I never tried Crystal because of the compilation step. I really don't need the extra speed for my sw and no customer asked me Crystal so far. I guess it's going to be a similarly pleasant experience.
    - e12e 7y ago
      > metaprogramming sometimes is easier in Python, sometimes not Now, I've been exposed to more ruby, and more metaprogramming in ruby, than in python - but this stil surprises/intrigues me. What kind of metaprogramming do you find easier in python?
      - pmontra 7y ago
        If I'm not wrong I can add a methods like this def m2(): print("nuovo") object.new_method = m2 instead of redefining the class. It's similar to JavaScript. However I don't have a computer at hand now. I can't check the details and what could be done on Ruby.
        d_k_f 7y ago
        True, unfortunately. You can't pass functions around as first class objects, you'll have to use lambdas or procs for that (although even then composition doesn't work like that in Ruby). (No judgement from me on whether that's better/worse/about equal, just confirming.)
- imtringued 7y ago
  Because you are tying yourself to a specific ISA. This is bad even within the same chip family. If you want to migrate your x86 32 bit code to 64 bit then you have to rewrite everything.
coldtea 7y ago
Because it's a much saner, coherent, and easier to use language with less footguns... Now, why not Crystal over C++? Because it's way less mature, with way less documentation, way less resources, way worse tooling, way less libraries, way more implementation bugs, and, if we're realistic, it might never go anywhere...
- dragonwriter 7y ago
  > Because it's a much saner, coherent, and easier to use language with less footguns... Which for parallelism probably switches the question to “why Crystal over Pony”, since Crystal seems to preserve the key C++ footguns specifically linked to parallelism.
- Jenz 7y ago
  > and, if we're realistic, it might never go anywhere... I don’t see why this should be the case, unless we as a community decides it will not go anywhere.
  - coldtea 7y ago
    Most languages never get anywhere. Languages of the quality of Smalltalk, and Ada, and Dylan, and Self, and CL have vanquished in the margins, and e.g. even something as celebrated as Haskell does hardly better in the industry (some financial companies, this or that project, and so on), so the chances of a new language like Crystal are slim to begin with...
    - Jenz 7y ago
      Oh yes, been there, done that, created proglangs, and they certainly did not get anywhere. But I feel Crystal has already gained such momentum that it indeed will get somewhere, it’s beyond that phase "to begin with." After all, if any language can get anywhere, why not Crystal?
      - coldtea 7y ago
        >But I feel Crystal has already gained such momentum that it indeed will get somewhere, it’s beyond that phase "to begin with." Let's hope so, but I fear that's wishful thinking. What proof do we have of that? It's not used anywhere important ("killer app"), barely makes the rest of the TIOBE index, along with staples like NATURAL, Alice, BBC Basic, Euphoria, Factor, Forth, and Icon, and has no major force like Google or Mozilla backing it. It's only us here in HN who keep hearing about it. Julia has had tons more publicity (even from some major outlets), and it's still going nowhere fast.
deleted 7y ago
[deleted]
odiroot 7y ago
To me is just Python-like (or actually Ruby-like but close enough) but with much better performance. I still haven't learnt to use it well though. It's a joy to write due to its syntax.

aidenn0 7y ago

I get why N=2 is slower than single-threaded, but why is N=4 more than twice as fast as N=2?

ricardobeat 7y ago
If you're looking at the first graph, the explanation is right above - the overhead of synchronization.
- aidenn0 7y ago
  That's nonsensical. There is surely at least as much synchronization overhead with 4 threads as with 2.
  - jonesetc 7y ago
    If the overhead is X, and the amount of work that can be done ignoring it is Y, you have Y - X. If you double the amount of work that can be down, but overhead stays the same you have 2Y - X, which is more than 2(Y - X) = 2Y - 2X. You're over performing doubling by not having to double the synchronization cost.
  - paulddraper 7y ago
    Not necessarily. There can be overhead introduced between the "1 thread" and ">1 thread" cases.
  - ricardobeat 7y ago
    From 1 -> 2 threads the overhead is introduced, if the cost is somewhat fixed, the gains from 2 -> 4 can be proportionally larger.

jackcodes 7y ago

I’ve been using Crystal on-and-off for a couple of years and have started to use it as my primary language for most non-enterprise work. The current web-app I’m building is primarily written in Crystal and it has been an absolute joy to use, and even if I don’t strictly need parallelism in this project I’m glad they’re pushing though with the big language changes. It’s been so successful at everything I’ve needed it for, I wouldn’t consider going back to either Ruby or similar interpreted languages, regardless whether it’s a little CLI helper or a larger monolith server app. I’ve found my ability to allow the compiler to catch bugs incredibly reliable without feeling bogged down with types. I hadn’t worked much with typed languages previously and as a Ruby developer it allowed me to ‘learn’ types without even feeling like I was doing any learning. It’s still somewhat of an obscure language so I thought it would be helpful to describe my experiences with it.

h1d 7y ago
Do you use much external libraries because ecosystem I'd pretty much what makes a language useful after the language being solid by itself. Also, I assume Googleability is pretty much close to zero, you'd have to go to the primary sites (their forum?) to ask questions when stuck?
- IfOnlyYouKnew 7y ago
  It's obviously far smaller, but I've had luck so far with everything I need being available. What I've seen, mostly Postgres, Redis, and XML/HTML parsing, have also worked flawlessly so far. Having performance improvements of 100x in real-world scenarios is just astonishing. Ruby is still my go-to, especially because debugging and console support are so much better. But everything that gives me time to think gets reimplemented. ["crystal lang" <question>] usually works for google.
- jackcodes 7y ago
  There's actually a surprising amount available in the stdlib for your typical web project. By way of example, I'd always hated how in Ruby each API wrapper gem would make their own decision on which HTTP client they used, so you could end up pulling in 3 or 4 separate libraries just to make HTTP calls to a 3rd party API. People may disagree but I think this is (was?) largely due to Ruby's Net:HTTP feeling janky to use, at least compared to Faraday and the like. Here, the HTTP modules within the standard library feel right, and I've not needed to reach out for all that much from the ecosystem. But as a simple example I accept that API calls to third party services is low hanging fruit so in terms of finding drivers for lower level protocols I've not had an issue that I'd class as a blocker. There are times where I've had to extend libraries or write a parser for this-and-that (e.g. PostGIS and Paseto, both of which were unused this time round) but I find I need to do this less and less as time goes on. You do need to reach into the forums or gitter on rare occasions but I can count on one hand the number of times I've needed to, and it's typically because I was trying to do something unusual and quite low-level on the HTTP stack - as in something I've never had to do in a non-Crystal project, such as sending a specific packet. I've always found help there too and people are happy to engage.

rishav_sharan 7y ago

I have been using crystal+ vanilla js for my pet project and the sheer joy i get from using this language is something else. 2019 has been a good year for crystal. Basic multithreading is in. Windows support is making good headway. I am optimistic that a working windows compiler would be out by end of year. The last major dragon for crystal to slay would be the compilation time. It still feels unreasonably high. Hopefully that would be a major focus of 2020

freshbagels 7y ago
I'm curious as to what your pet project is/if it's available to online to play with.
- rishav_sharan 7y ago
  The Crystal underbelly is here https://github.com/rishavs/noir https://github.com/rishavs/noir the api is consumed by a vanilla js spa. I talk more about my VanillaJS spa's architecture here https://dev.to/rishavs/making-a-single-page-app-in-ye-good-olde-js-es6-3eng https://dev.to/rishavs/making-a-single-page-app-in-ye-good-o...

rarecoil 7y ago

I just learned about Crystal a couple of weeks ago and it's completely subsumed every new personal project I have into its domain. I have long missed Ruby's syntax and human-friendly feel, but began to prefer the speed I could find in other languages like Go.

squarefoot 7y ago

Waiting badly for an installable ARM port; small boards would benefit a lot from it. Is this planned in the future?

maitredusoi 7y ago

Until crystal get a REPL I would stick to ruby / rails for POC and MVP, then scaling problem "could" arose, so translate it to amber or lucky (Web Framework in crystal) could solve the issue. About Type and Casting Nil, gem like sorbet should be a good replacement in ruby