6 ms·

There's only so many GC engineers in the world, and the majority work in academia, industrial research labs and places like the Oracle VM team. So some web-app

There's only so many GC engineers in the world, and the majority work in academia, industrial research labs and places like the Oracle VM team. So some web-app company isn't likely to hire one. Also they aren't interested in making some toy GC for a language they don't use.

ioquatix 13y ago
Really? Seriously?
- FooBarWidget 13y ago
  I think the guy has a good point. GC is a relatively obscure area, with few people having expertise in it. This, plus the fact that the number of Rubyists who also know C (and know C well enough to work on the GC) is very small, doesn't help the current situation. The Ruby GC is also extremely tied to the VM implementation, making it generally very hard to hack on. I made the Ruby garbage collector copy-on-write friendly several years ago.
justincormack 13y ago
And this is why Silicon Valley has a hiring problem. No, you hire someone who can code C and read the GC literature. The problem space is well understood. You are not breaking new ground, just applying known algorithms.
- dasil003 13y ago
  This is an absolutely ridiculous comment. Writing a GC for a dynamic language like Ruby is not just a question of knowing C and reading some literature. It's a massive problem to solve. This is not something that you double your burn rate in the hopes that 2-3 years down the line you can double performance of your application cluster. It's simply not worth it. Startups have immediate problems to solve or they die. And BTW, the reason we have a hiring problem is not because we are too picky about specific experience, it's because when it comes to software engineering some people have it and some people don't, and the ones who don't are the ones whose resumes are consistently out there filling up inboxes, and the ones who do are rarely looking for a job, and when they are the Facebooks, Twitters and Googles of the world are throwing around $200k + benefits like pocket change.
  - justincormack 13y ago
    The OP was suggesting that only someone who had worked at Oracle could solve the problem. At least you are saying that someone who "has it" might have a chance. The issue of whether startups would fund it is a different question. Quite clearly the performance of Ruby is not in fact a huge problem for most people, and we haven't had a huge organisation that could fund stuff in the way, say, Facebook has with PHP. I haven't looked at Ruby's GC, but I am guessing it was written by one person originally, and that could be done again. The poster at the top said it was simple. Its a big problem but as I said, its not a perfect solution needed just a better one.
  - coldtea 13y ago
    >This is an absolutely ridiculous comment. Writing a GC for a dynamic language like Ruby is not just a question of knowing C and reading some literature. It's a massive problem to solve. No it's not. There are tons of good GC's around. You can even improve Ruby's GC with 1990 technology. (E.g did the LuaJit guy solved a "massive problem"? Mostly by himself?)
    - jamesaguilar 13y ago
      Some people routinely underestimate what a team of two or three smart guys and gals can do if they get in a room together and really work on a problem.
    - Tuna-Fish 13y ago
      GC in Lua is a fundamentally much easier problem because it doesn't do threads. Also, "Mike Pall can do it so everyone can" is not a very good argument -- Mike Pall is fucking brilliant, and quite a lot of his code is stuff that your typical great C coder can't ever do.
- AaronFriel 13y ago
  Implementing GC rarely seems as simple as "reading the literature" and implementing the algorithms they tell you. One problem is that you're working with an existing code base, and you can't afford to make mistakes in your GC. Sure, there are plenty of (good) GC algorithms out there that are public knowledge, but implementing it alone may be difficult, implementing it efficiently is another order of magnitude more-so.
- sanderjd 13y ago
  There was a line from a talk about Rust[0] that really stuck with me, commenting on the design choice to only garbage-collect task-local heaps using reference counting with cycle detection: "we wanted to avoid spending the next 20 years writing another world-class garbage collector." It is not just a single person project to write a really good GC. 0: I think it was this one - http://www.infoq.com/presentations/Rust http://www.infoq.com/presentations/Rust Edit: wording and precision
  - steveklabnik 13y ago
    Note that in Rust it's not as big a deal as in Ruby because of two factors: GC usage is not idiomatic, and since it's task-local it won't ever stop the world.
    - sanderjd 13y ago
      Absolutely. My point being they seem to have purposefully worked the attributes you mention into the design of the language largely because of how hard it is to write a really good GC. The Rust team has lots of people who know C and are very capable of reading literature, and still didn't really want to tackle it, because it's hard.
astrodust 13y ago
So this leads to the question: Why there isn't an open-source garbage collector that can be used across different projects? Surely Python, V8, and Ruby have enough in common, allocation-wise, that the code can be shared? They're all C in the back-end. It's just pointers and more pointers at the lowest level. It's not about making a toy GC, it's about making a top-rate GC that works and packaging it up as a tool.
- Derander 13y ago
  To a certain extent that's what the JVM is -- you just have to target its bytecode :-).
  - astrodust 13y ago
    That's not exactly a portable back-end toolkit you can just link in.
    - Tuna-Fish 13y ago
      It's not really possible to do that with GC. It requires analyzing and dealing with basically every assumption about object lifetimes that is done within the system, and that typically requires you to read and understand every single line of code. I have worked on GC projects, and imho bolting on a good GC system on an existing language runtime is considerably harder than reimplementing the entire runtime on top of a GC platform.
- __alexs 13y ago
  The Boehm GC [1] has been integrated with various other languages in the past. I'm aware of various attempts to get both Python and Ruby code running with it. 1. http://www.hpl.hp.com/personal/Hans_Boehm/gc/ http://www.hpl.hp.com/personal/Hans_Boehm/gc/
- silentbicycle 13y ago
  Garbage collection is a particularly cross-cutting implementation issue, and tends to interfere with all kinds of abstraction. You're essentially making a system that knows which specific ways abstraction can be broken safely, so that you can defer resource management to it in the future. Retrofitting a GC is even harder. You can bolt on a conservative collector (like boehm-gc), sure, but that can lead to many unexpected performance issues (see e.g. http://timetobleed.com/6-line-eventmachine-bugfix-2x-faster-gc-1300-requestssec/ http://timetobleed.com/6-line-eventmachine-bugfix-2x-faster-...). Writing a virtual machine for a GC'd language and then trying to give it a portable, reasonably convenient C API convinced me that it's a significantly harder issue than most people realize. (Particularly after scrapping my third prototype!) The C API also impacts many things, and tends to bring up nasty edge cases in the GC. Lua seems to have settled on a design that works unusually well, and is worth studying. Still, Lua can safely cut corners that Python and Ruby cannot - they have threads, which also break a lot of assumptions.

ioquatix 13y ago

Really? Seriously?

FooBarWidget 13y ago
I think the guy has a good point. GC is a relatively obscure area, with few people having expertise in it. This, plus the fact that the number of Rubyists who also know C (and know C well enough to work on the GC) is very small, doesn't help the current situation. The Ruby GC is also extremely tied to the VM implementation, making it generally very hard to hack on. I made the Ruby garbage collector copy-on-write friendly several years ago.

justincormack 13y ago

And this is why Silicon Valley has a hiring problem. No, you hire someone who can code C and read the GC literature. The problem space is well understood. You are not breaking new ground, just applying known algorithms.

dasil003 13y ago
This is an absolutely ridiculous comment. Writing a GC for a dynamic language like Ruby is not just a question of knowing C and reading some literature. It's a massive problem to solve. This is not something that you double your burn rate in the hopes that 2-3 years down the line you can double performance of your application cluster. It's simply not worth it. Startups have immediate problems to solve or they die. And BTW, the reason we have a hiring problem is not because we are too picky about specific experience, it's because when it comes to software engineering some people have it and some people don't, and the ones who don't are the ones whose resumes are consistently out there filling up inboxes, and the ones who do are rarely looking for a job, and when they are the Facebooks, Twitters and Googles of the world are throwing around $200k + benefits like pocket change.
- justincormack 13y ago
  The OP was suggesting that only someone who had worked at Oracle could solve the problem. At least you are saying that someone who "has it" might have a chance. The issue of whether startups would fund it is a different question. Quite clearly the performance of Ruby is not in fact a huge problem for most people, and we haven't had a huge organisation that could fund stuff in the way, say, Facebook has with PHP. I haven't looked at Ruby's GC, but I am guessing it was written by one person originally, and that could be done again. The poster at the top said it was simple. Its a big problem but as I said, its not a perfect solution needed just a better one.
- coldtea 13y ago
  >This is an absolutely ridiculous comment. Writing a GC for a dynamic language like Ruby is not just a question of knowing C and reading some literature. It's a massive problem to solve. No it's not. There are tons of good GC's around. You can even improve Ruby's GC with 1990 technology. (E.g did the LuaJit guy solved a "massive problem"? Mostly by himself?)
  - jamesaguilar 13y ago
    Some people routinely underestimate what a team of two or three smart guys and gals can do if they get in a room together and really work on a problem.
  - Tuna-Fish 13y ago
    GC in Lua is a fundamentally much easier problem because it doesn't do threads. Also, "Mike Pall can do it so everyone can" is not a very good argument -- Mike Pall is fucking brilliant, and quite a lot of his code is stuff that your typical great C coder can't ever do.
AaronFriel 13y ago
Implementing GC rarely seems as simple as "reading the literature" and implementing the algorithms they tell you. One problem is that you're working with an existing code base, and you can't afford to make mistakes in your GC. Sure, there are plenty of (good) GC algorithms out there that are public knowledge, but implementing it alone may be difficult, implementing it efficiently is another order of magnitude more-so.
sanderjd 13y ago
There was a line from a talk about Rust[0] that really stuck with me, commenting on the design choice to only garbage-collect task-local heaps using reference counting with cycle detection: "we wanted to avoid spending the next 20 years writing another world-class garbage collector." It is not just a single person project to write a really good GC. 0: I think it was this one - http://www.infoq.com/presentations/Rust http://www.infoq.com/presentations/Rust Edit: wording and precision
- steveklabnik 13y ago
  Note that in Rust it's not as big a deal as in Ruby because of two factors: GC usage is not idiomatic, and since it's task-local it won't ever stop the world.
  - sanderjd 13y ago
    Absolutely. My point being they seem to have purposefully worked the attributes you mention into the design of the language largely because of how hard it is to write a really good GC. The Rust team has lots of people who know C and are very capable of reading literature, and still didn't really want to tackle it, because it's hard.

astrodust 13y ago

So this leads to the question: Why there isn't an open-source garbage collector that can be used across different projects? Surely Python, V8, and Ruby have enough in common, allocation-wise, that the code can be shared? They're all C in the back-end. It's just pointers and more pointers at the lowest level. It's not about making a toy GC, it's about making a top-rate GC that works and packaging it up as a tool.

Derander 13y ago
To a certain extent that's what the JVM is -- you just have to target its bytecode :-).
- astrodust 13y ago
  That's not exactly a portable back-end toolkit you can just link in.
  - Tuna-Fish 13y ago
    It's not really possible to do that with GC. It requires analyzing and dealing with basically every assumption about object lifetimes that is done within the system, and that typically requires you to read and understand every single line of code. I have worked on GC projects, and imho bolting on a good GC system on an existing language runtime is considerably harder than reimplementing the entire runtime on top of a GC platform.
__alexs 13y ago
The Boehm GC [1] has been integrated with various other languages in the past. I'm aware of various attempts to get both Python and Ruby code running with it. 1. http://www.hpl.hp.com/personal/Hans_Boehm/gc/ http://www.hpl.hp.com/personal/Hans_Boehm/gc/
silentbicycle 13y ago
Garbage collection is a particularly cross-cutting implementation issue, and tends to interfere with all kinds of abstraction. You're essentially making a system that knows which specific ways abstraction can be broken safely, so that you can defer resource management to it in the future. Retrofitting a GC is even harder. You can bolt on a conservative collector (like boehm-gc), sure, but that can lead to many unexpected performance issues (see e.g. http://timetobleed.com/6-line-eventmachine-bugfix-2x-faster-gc-1300-requestssec/ http://timetobleed.com/6-line-eventmachine-bugfix-2x-faster-...). Writing a virtual machine for a GC'd language and then trying to give it a portable, reasonably convenient C API convinced me that it's a significantly harder issue than most people realize. (Particularly after scrapping my third prototype!) The C API also impacts many things, and tends to bring up nasty edge cases in the GC. Lua seems to have settled on a design that works unusually well, and is worth studying. Still, Lua can safely cut corners that Python and Ruby cannot - they have threads, which also break a lot of assumptions.