5 ms·

Mongrel2 Almost Valgrind Pure, Bstring Happy

69 points by FraaJad 16y ago

modeless 16y ago
"Despite all the evidence that C is quite alright for many applications, and that no, the world doesn't end because of its use" The world only doesn't end because people are so used to the terrible security record of C apps that they don't even blink when new remote code execution vulnerabilities are found daily. Perhaps someday we'll realize it doesn't have to be this way. Also, of course the Boehm GC isn't very good. It's ancient and in no way representative of the latest technology in garbage collection. GC can only really be good when it's integrated with the language. A precise generational collector would perform far better but wouldn't be so easy to drop into a C codebase.
- jrockway 16y ago
  Indeed. What Zed demonstrates by writing stuff in C is that he is good at cutting off his nose to spite his face. The reason mongrel2 works so well is because it has made good design decisions, not because it's written in C. I've played with similarly-designed servers written in Perl (Coro/EV), and they are so fast that the disk is the limiting factor for file serving (and use so few resources on top of your web app that you don't even know you have a server). Good design, not language choice, is what makes for good software. (In other words, the conclusion "C is fast and good for apps" is not what you should reach from watching the mongrel2 development. "You can use any language if you have good design and a clue," is.) I would have picked Haskell. Then I could drop to C when needed, but probably not ever need to. But then what would there have been to blog about? :)
  - zedshaw 16y ago
    You're right, it doesn't matter too much what language you do a piece of software in, it's more the design. BUT! When the language tries to make sockets way more complicated than needed, screws with memory management to the point of uselessness, or cram its little version of "stateless" false dogma down your throat, it sure doesn't make it easy. Alright then, if you'd do it in Haskell, why haven't you? I spent about 2-3 weeks so far on Mongrel2 and it already does quite a lot. I figure, hey, Haskell's supposedly so much more bad ass, let's see you get it done. In fact, I'll be you don't even use any of these Haskell web servers as your main web server: http://www.haskell.org/haskellwiki/Haskell_Web_Server http://www.haskell.org/haskellwiki/Haskell_Web_Server Thus my point. Language fetishists become distracted by the hype and dogma of their favorite language tribe and then become blind to the evidence. Evidence so far says C makes good web servers. Web servers in other languages tend to need to use C. Lots of great software powers the world and is written in C. Using C does not cause the death of the universe. It isn't hard, or scary, it's actually pretty simple. After a while, you have to start wondering, is it true that the language is so dangerous, or is it just more marketing?
    - jrockway 16y ago
      Most people like to reuse code, rather than rewrite, especially if they don't have any ideas for improvement. Hence the longevity of, say Apache. Since C has been around 1972, it follows that a lot of software is going to be written in C.
      - ramchip 16y ago
        Since C has been around 1972, it follows that a lot of software is going to be written in C. I agree with your conclusion but not your reasoning. Try substituting "Lisp" and "1958".
        jrockway 16y ago
        Not really a fair comparison. From 1972 until 2000, C was the most performance/expression balance that industry could bear. Lisp sacrificed raw computational performance for increased readability and writability, but that's a tradeoff that was unpopular in industry. Hence, it was never a viable competitor. (I like it, but I am not industry.) A fair comparison is C to C++ or Java to C#. C has been around longer than C++, and there are more C apps than C++ apps on UNIX. (Windows is another story; MS pushed C++ pretty hard back in the day. Same for Obj-C on Mac.) Java has been around longer than C#, and there is more stuff written in it, even though it is technically inferior. Time always beats language features. If you want an accurate measurement, start a totally new project today that depends on nothing. Clone yourself. Then start writing it in C, and in $some_other_language. Then see which one meets your expectations soonest. It will probably be $some_other_language. But add in some dependencies on historical code, and C becomes competitive again. Standing on the shoulders of a giant is faster, in the short term, than becoming a taller giant.
    - bguthrie 16y ago
      It's not more marketing, Zed, it's years and years and years of evidence that poorly-managed memory is a major source of error in design, and folks in the application development and security fields trying to pass along a message to avoid it if possible. State is also a major source of error in design, which is why stateless dogmatists keep hammering on it. You can get good design without them, and some people do. But everyone makes mistakes sometimes. Picking technologies that help you avoid making them isn't marketing, it's just good sense. People have a bad habit of calling any received wisdom they disagree with "marketing". There are people out there far smarter than I am, and possibly smarter than you are (though I'm not in a position to judge), who disagree strongly with what you say, and they're not marketeers. No need to go down that route in this debate.
- zedshaw 16y ago
  Boehm's a very good garbage collector, given this: > A precise generational collector would perform far better but wouldn't be so easy to drop into a C codebase. It's that last part that's key. Without language support at even the basic level, you won't get much better. As for this "precise generational collector", I doubt it. Like everything, saying you need a GC all the time to be safe is just more boogey man fear mongering. You don't need a GC. They're nice yeah. But if you know the application very well then memory management isn't too big a deal.
  - aufreak3 16y ago
    > But if you know the application very well then memory management isn't too big a deal. Agreed. A related case is with Objective-C's memory management scheme which isn't a full GC, but lets you scope your memory allocations. It gets you a level of control that is somewhere between a GC and the kind that boost::shared_ptr does ... neither having the "wait for the trashman" problem that GC brings, nor overdoing the reference count touching like boost can get you into.
    - masklinn 16y ago
      > A related case is with Objective-C's memory management scheme which isn't a full GC, but lets you scope your memory allocations. Obj-C 1 right? I believe Obj-C 2 (on OSX, not iOS) has an optional GC, at least for the objective part. Or would I be wrong?
      - mishmash 16y ago
        Yeah, you're right it does. And many of the C-based APIs also have functions to put memory "into" the collector.
  - thaumaturgy 16y ago
    Nice to see someone preaching that old time religion.
  - modeless 16y ago
    GC isn't required to be safe, and it doesn't guarantee safety, but it does makes safety a whole lot more likely. Boehm is the best garbage collector given its constraints, but that doesn't make it a good garbage collector overall, or even one worth using. I agree that good GC requires language support; what I disagree with is the attitude that people should use C anyway and forget GC because Boehm isn't good.
ihodes 16y ago
Before we go too deep into Zed-hating, I’ve got to say I am psyched for Mongrel2, and I don't even have a particular use in mind for it. Aside from the fact that it'll undoubtedly be a great piece of software, it's informative and interesting to follow along with his progress on his blog and on his fossil repo. And if what he promises in terms if features and performance comes to fruition, it'll be a fscking awesome server/gift to hack on and hack with. Cheers, Zed. Keep it up!
- zedshaw 16y ago
  Thanks man. I'm pretty stoked about it too. It's shaping up nicely and is tons of fun, not to mention I really think it's got potential. I just hope it honestly puts an end to this insistence on "one language to rule them all".
  - demallien 16y ago
    Zed, do you think you could have written Mongrel2 in C, whilst doing a good job, without having already had the experience with Mongrel v1? What I'm trying to get at is our we just seeing another example of scripting languages being good for prototyping, but getting better raw performance(memory footprint,/speed/disk usage) from a lower level language? Or do you think you could have ended up with a decent architecture on a first go?
    - jonhohle 16y ago
      I'm not Zed, but I have taken this approach several times (Primarily Ruby->Objective-C, but also Ruby->C). I often find it helpful to validate an idea in code quickly, and rewrite for speed/scalabilty/platform integration/etc once I'm happy with a design.
    - silentbicycle 16y ago
      I'm not Zed either, but it's worth noting that you can usually do this without having to throw away the higher-level language entirely - most scripting languages have C APIs so that you can profile and move the expensive parts to C. Sometimes it's necessary to change the architecture of the entire project, but more localized changes are often good enough. Lua and Tcl were designed with this approach in mind (they expect to be embedded), but you can do it with Python, Ruby, Lisp, Erlang, etc. too.
pfrtgi 16y ago
A library for handling strings ? LMAO.
johkra 16y ago
I had to smile, when I read the head line and later the article. During my first year of university, I wrote a verb conjugation engine and a very simple webserver for the UI (I wanted cross-plattform compatibility and it was easier than learning GTK+). I, too, choose bstring for strings and used valgrind a lot. It's nice to see I'm not the only one writing C code this way. (Although I don't write much C anymore.)
danieldk 16y ago
"Memory is predictable, and with Valgrind not a danger." I find that statement a tad too optimistic. If you use Valgrind continually (and people really should), you do catch memory misuse in normal situations. It does not say much about how well programs function with 'abnormal' inputs (accidents, exploits). Given that bugs do happen, even to the best of programmers, would you rather have a buffer overflow or an exception? It depends on the application and context, but it is harder to answer (and get it right) than the author pretends.
- zedshaw 16y ago
  Well, your logic applies to every language, no matter what, always. The only difference is the types of errors it can't handle and what tools you need to avoid them. I think of valgrind as a GC compiler.
  - danieldk 16y ago
    Of course, except that many runtimes/libraries handle tripping over array boundaries gracefully, rather than overwriting memory. And some languages avoid null/wild pointers completely. Your programs will still be suspectible to buffer overflows, invalid dereferencing, etc. While, say, Haskell programmers do not have the same problems. Valgrind is no substitute to runtime checks and a proper type system.
    - zedshaw 16y ago
      Your runtime checks and proper type system and at dynamic linking. That means you don't have them, and they aren't a complete total truth of any kind at all. They're just a highly probable enhancement.
bnoordhuis 16y ago
> stupid J2ME clients that send chunked-encoding when they aren't supposed to (it's for servers morons). Sorry, Mr. Shaw, but you are wrong (but I still like you and I'm looking forward to Mongrel2). RFC 2616 is careful to say 'server' when it means servers and 'client' when it means clients. Section 3.6.1 specifically states that HTTP/1.1 applications (so both clients and servers!) MUST support chunked encoding. The developers of that Java HTTP client (Jakarta?) seem to have read the spec more thoroughly than you did. :-)
- xal 16y ago
  Furthermore, Chunked to the client is having a renaissance at the moment thanks to the BigPipe ideas coming out of Facebook. Speaking of which, Mongrel2 will be a tremendous server for apps that need this optimization.
  - zedshaw 16y ago
    No, this is chunked from the client, not to the client. To the client is quite alright and works great. From the client is absolutely retarded and total abuse of the protocol. Which brings up another point: HTTP is like the old testament. Wanna kill your brother? Probably find something to support it. Wanna sacrifice a goat? Yep, you could do that. Don't want people to sacrifice goats? Sure, rock on. Want adultery? You could probably find that too. It's got everything depending on what you read, where you read, and what context you take it in. The HTTP RFC is like that. Want to send chunked encoding mixed inside a keep-alive with request pipelining when you don't really need to? Rock on. Want google web accelerator to hit links without you asking it to (in clear violation of every other part of the browser standard)? Go for it. Don't want people to do that? Cool there's another part. After a while, you have to go with what's most probable, and just be explicit in your own design about the rest.
jkmcf 16y ago
The moral of these threads: don't argue with Zed. He'll out-point you! Don't hate the coder, love the code.
JabavuAdams 16y ago
Switching stacks on the GC was like setting its house on fire and then shooting it as it ran out the front door in its lingerie. That's bad, right? I'm still giggling.

69 points by FraaJad 16y ago

"Despite all the evidence that C is quite alright for many applications, and that no, the world doesn't end because of its use" The world only doesn't end because people are so used to the terrible security record of C apps that they don't even blink when new remote code execution vulnerabilities are found daily. Perhaps someday we'll realize it doesn't have to be this way. Also, of course the Boehm GC isn't very good. It's ancient and in no way representative of the latest technology in garbage collection. GC can only really be good when it's integrated with the language. A precise generational collector would perform far better but wouldn't be so easy to drop into a C codebase.

jrockway 16y ago
Indeed. What Zed demonstrates by writing stuff in C is that he is good at cutting off his nose to spite his face. The reason mongrel2 works so well is because it has made good design decisions, not because it's written in C. I've played with similarly-designed servers written in Perl (Coro/EV), and they are so fast that the disk is the limiting factor for file serving (and use so few resources on top of your web app that you don't even know you have a server). Good design, not language choice, is what makes for good software. (In other words, the conclusion "C is fast and good for apps" is not what you should reach from watching the mongrel2 development. "You can use any language if you have good design and a clue," is.) I would have picked Haskell. Then I could drop to C when needed, but probably not ever need to. But then what would there have been to blog about? :)
- zedshaw 16y ago
  You're right, it doesn't matter too much what language you do a piece of software in, it's more the design. BUT! When the language tries to make sockets way more complicated than needed, screws with memory management to the point of uselessness, or cram its little version of "stateless" false dogma down your throat, it sure doesn't make it easy. Alright then, if you'd do it in Haskell, why haven't you? I spent about 2-3 weeks so far on Mongrel2 and it already does quite a lot. I figure, hey, Haskell's supposedly so much more bad ass, let's see you get it done. In fact, I'll be you don't even use any of these Haskell web servers as your main web server: http://www.haskell.org/haskellwiki/Haskell_Web_Server http://www.haskell.org/haskellwiki/Haskell_Web_Server Thus my point. Language fetishists become distracted by the hype and dogma of their favorite language tribe and then become blind to the evidence. Evidence so far says C makes good web servers. Web servers in other languages tend to need to use C. Lots of great software powers the world and is written in C. Using C does not cause the death of the universe. It isn't hard, or scary, it's actually pretty simple. After a while, you have to start wondering, is it true that the language is so dangerous, or is it just more marketing?
  - jrockway 16y ago
    Most people like to reuse code, rather than rewrite, especially if they don't have any ideas for improvement. Hence the longevity of, say Apache. Since C has been around 1972, it follows that a lot of software is going to be written in C.
    - ramchip 16y ago
      Since C has been around 1972, it follows that a lot of software is going to be written in C. I agree with your conclusion but not your reasoning. Try substituting "Lisp" and "1958".
      - jrockway 16y ago
        Not really a fair comparison. From 1972 until 2000, C was the most performance/expression balance that industry could bear. Lisp sacrificed raw computational performance for increased readability and writability, but that's a tradeoff that was unpopular in industry. Hence, it was never a viable competitor. (I like it, but I am not industry.) A fair comparison is C to C++ or Java to C#. C has been around longer than C++, and there are more C apps than C++ apps on UNIX. (Windows is another story; MS pushed C++ pretty hard back in the day. Same for Obj-C on Mac.) Java has been around longer than C#, and there is more stuff written in it, even though it is technically inferior. Time always beats language features. If you want an accurate measurement, start a totally new project today that depends on nothing. Clone yourself. Then start writing it in C, and in $some_other_language. Then see which one meets your expectations soonest. It will probably be $some_other_language. But add in some dependencies on historical code, and C becomes competitive again. Standing on the shoulders of a giant is faster, in the short term, than becoming a taller giant.
  - bguthrie 16y ago
    It's not more marketing, Zed, it's years and years and years of evidence that poorly-managed memory is a major source of error in design, and folks in the application development and security fields trying to pass along a message to avoid it if possible. State is also a major source of error in design, which is why stateless dogmatists keep hammering on it. You can get good design without them, and some people do. But everyone makes mistakes sometimes. Picking technologies that help you avoid making them isn't marketing, it's just good sense. People have a bad habit of calling any received wisdom they disagree with "marketing". There are people out there far smarter than I am, and possibly smarter than you are (though I'm not in a position to judge), who disagree strongly with what you say, and they're not marketeers. No need to go down that route in this debate.
zedshaw 16y ago
Boehm's a very good garbage collector, given this: > A precise generational collector would perform far better but wouldn't be so easy to drop into a C codebase. It's that last part that's key. Without language support at even the basic level, you won't get much better. As for this "precise generational collector", I doubt it. Like everything, saying you need a GC all the time to be safe is just more boogey man fear mongering. You don't need a GC. They're nice yeah. But if you know the application very well then memory management isn't too big a deal.
- aufreak3 16y ago
  > But if you know the application very well then memory management isn't too big a deal. Agreed. A related case is with Objective-C's memory management scheme which isn't a full GC, but lets you scope your memory allocations. It gets you a level of control that is somewhere between a GC and the kind that boost::shared_ptr does ... neither having the "wait for the trashman" problem that GC brings, nor overdoing the reference count touching like boost can get you into.
  - masklinn 16y ago
    > A related case is with Objective-C's memory management scheme which isn't a full GC, but lets you scope your memory allocations. Obj-C 1 right? I believe Obj-C 2 (on OSX, not iOS) has an optional GC, at least for the objective part. Or would I be wrong?
    - mishmash 16y ago
      Yeah, you're right it does. And many of the C-based APIs also have functions to put memory "into" the collector.
- thaumaturgy 16y ago
  Nice to see someone preaching that old time religion.
- modeless 16y ago
  GC isn't required to be safe, and it doesn't guarantee safety, but it does makes safety a whole lot more likely. Boehm is the best garbage collector given its constraints, but that doesn't make it a good garbage collector overall, or even one worth using. I agree that good GC requires language support; what I disagree with is the attitude that people should use C anyway and forget GC because Boehm isn't good.

ihodes 16y ago

Before we go too deep into Zed-hating, I’ve got to say I am psyched for Mongrel2, and I don't even have a particular use in mind for it. Aside from the fact that it'll undoubtedly be a great piece of software, it's informative and interesting to follow along with his progress on his blog and on his fossil repo. And if what he promises in terms if features and performance comes to fruition, it'll be a fscking awesome server/gift to hack on and hack with. Cheers, Zed. Keep it up!

zedshaw 16y ago
Thanks man. I'm pretty stoked about it too. It's shaping up nicely and is tons of fun, not to mention I really think it's got potential. I just hope it honestly puts an end to this insistence on "one language to rule them all".
- demallien 16y ago
  Zed, do you think you could have written Mongrel2 in C, whilst doing a good job, without having already had the experience with Mongrel v1? What I'm trying to get at is our we just seeing another example of scripting languages being good for prototyping, but getting better raw performance(memory footprint,/speed/disk usage) from a lower level language? Or do you think you could have ended up with a decent architecture on a first go?
  - jonhohle 16y ago
    I'm not Zed, but I have taken this approach several times (Primarily Ruby->Objective-C, but also Ruby->C). I often find it helpful to validate an idea in code quickly, and rewrite for speed/scalabilty/platform integration/etc once I'm happy with a design.
  - silentbicycle 16y ago
    I'm not Zed either, but it's worth noting that you can usually do this without having to throw away the higher-level language entirely - most scripting languages have C APIs so that you can profile and move the expensive parts to C. Sometimes it's necessary to change the architecture of the entire project, but more localized changes are often good enough. Lua and Tcl were designed with this approach in mind (they expect to be embedded), but you can do it with Python, Ruby, Lisp, Erlang, etc. too.

pfrtgi 16y ago

A library for handling strings ? LMAO.

johkra 16y ago

I had to smile, when I read the head line and later the article. During my first year of university, I wrote a verb conjugation engine and a very simple webserver for the UI (I wanted cross-plattform compatibility and it was easier than learning GTK+). I, too, choose bstring for strings and used valgrind a lot. It's nice to see I'm not the only one writing C code this way. (Although I don't write much C anymore.)

danieldk 16y ago

"Memory is predictable, and with Valgrind not a danger." I find that statement a tad too optimistic. If you use Valgrind continually (and people really should), you do catch memory misuse in normal situations. It does not say much about how well programs function with 'abnormal' inputs (accidents, exploits). Given that bugs do happen, even to the best of programmers, would you rather have a buffer overflow or an exception? It depends on the application and context, but it is harder to answer (and get it right) than the author pretends.

zedshaw 16y ago
Well, your logic applies to every language, no matter what, always. The only difference is the types of errors it can't handle and what tools you need to avoid them. I think of valgrind as a GC compiler.
- danieldk 16y ago
  Of course, except that many runtimes/libraries handle tripping over array boundaries gracefully, rather than overwriting memory. And some languages avoid null/wild pointers completely. Your programs will still be suspectible to buffer overflows, invalid dereferencing, etc. While, say, Haskell programmers do not have the same problems. Valgrind is no substitute to runtime checks and a proper type system.
  - zedshaw 16y ago
    Your runtime checks and proper type system and at dynamic linking. That means you don't have them, and they aren't a complete total truth of any kind at all. They're just a highly probable enhancement.

bnoordhuis 16y ago

> stupid J2ME clients that send chunked-encoding when they aren't supposed to (it's for servers morons). Sorry, Mr. Shaw, but you are wrong (but I still like you and I'm looking forward to Mongrel2). RFC 2616 is careful to say 'server' when it means servers and 'client' when it means clients. Section 3.6.1 specifically states that HTTP/1.1 applications (so both clients and servers!) MUST support chunked encoding. The developers of that Java HTTP client (Jakarta?) seem to have read the spec more thoroughly than you did. :-)

xal 16y ago
Furthermore, Chunked to the client is having a renaissance at the moment thanks to the BigPipe ideas coming out of Facebook. Speaking of which, Mongrel2 will be a tremendous server for apps that need this optimization.
- zedshaw 16y ago
  No, this is chunked from the client, not to the client. To the client is quite alright and works great. From the client is absolutely retarded and total abuse of the protocol. Which brings up another point: HTTP is like the old testament. Wanna kill your brother? Probably find something to support it. Wanna sacrifice a goat? Yep, you could do that. Don't want people to sacrifice goats? Sure, rock on. Want adultery? You could probably find that too. It's got everything depending on what you read, where you read, and what context you take it in. The HTTP RFC is like that. Want to send chunked encoding mixed inside a keep-alive with request pipelining when you don't really need to? Rock on. Want google web accelerator to hit links without you asking it to (in clear violation of every other part of the browser standard)? Go for it. Don't want people to do that? Cool there's another part. After a while, you have to go with what's most probable, and just be explicit in your own design about the rest.

jkmcf 16y ago

The moral of these threads: don't argue with Zed. He'll out-point you! Don't hate the coder, love the code.

JabavuAdams 16y ago

Switching stacks on the GC was like setting its house on fire and then shooting it as it ran out the front door in its lingerie. That's bad, right? I'm still giggling.