4 ms·

>You really need to go at least this far. Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen t

>You really need to go at least this far. Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen the benchmarks in question? Why not just go do it, if it's so essential and straightforward?

kbenson 14y ago
> Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen the benchmarks in question? Okay, let me clarify. When I say "you really need to go this far" I mean that stopping at any point before that (but after the simple metric of how many requests a second it can serve which they already do) makes no sense, IMHO. If you are going to compare frameworks and you want to go beyond that initial performance metric, you might as well aim high enough to be useful. I agree you never see anything approaching that in other reviews/benchmarks. Is that a good reason to not try it here? > Why not just go do it, if it's so essential and straightforward? I'm envisioning this as a community process, not a "Go off and write this in 20 frameworks or you're useless" sort or ultimatum. As such, just speccing out a possibly route is helping. Also, I plan to help with the existing benchmarks. After the second round, I pointed them out to the author of my favorite framework in the hopes he would have time to put together something for the benchmark, otherwise I was going to in the next week or two when I had time. I still plan to. Oh, and that framework author's answer? That these benchmarks are laughable because all they measure is performance, and there's a clear performance to convenience trade-off shown in the results, and that of course there's a performance hit when the framework handles most the work for you. I have to say I agree. Sure, there's possibly some that are clear winners giving good performance with lots of conveniences for common operations, but is there any way to tell as much from the data presented so far?
- Terretta 14y ago
  > I mean that stopping at any point before that (but after the simple metric of how many requests a second it can serve which they already do) makes no sense, IMHO. If you are going to compare frameworks and you want to go beyond that initial performance metric, you might as well aim high enough to be useful. The query 20 random rows, build an object, and return that, is way beyond requests per second. Enough that the list is dramatically re-ordered. > That these benchmarks are laughable because all they measure is performance, and there's a clear performance to convenience trade-off shown in the results, and that of course there's a performance hit when the framework handles most the work for you. I have to say I agree. On the contrary, performance versus magic is absolutely not a "given" here. Yes, some bare languages are near the top, but there are also heavy frameworks (e.g. Spring) performing well, and lean frameworks performing poorly. As for being "laughable", that makes me suspicious of the framework author's understanding of where optimization needs to happen. Presumably, pages will be run more often than they are authored. Presumably there's a recurring bill for the server farm. Optimizing for performance helps end users stay happy, and helps the company stay in business able to continue employing developers. > possibly some that are clear winners giving good performance with lots of conveniences for common operations, but is there any way to tell as much from the data presented so far? I agree with that point. I would like to see 3 additional columns added to the results: LOC, number of src files/templates across number of directories, and number of libs. This helps suss out your point: how much does a coder have to type in this framework, and how much incidental complexity (files, libs) do they have to wrap their heads around? Consider: [300 loc; 15 files 2 dirs; 8 libs] [650 loc; 1 file 1 dir; 0 libs] [150 loc; 15 files 15 dirs; 23 libs] Multiply by 10 to imagine real world code, and these would each feel very different to an author, and to a new hire hired to maintain a project already in production.
  - bhauer 14y ago
    To your last point, Terretta, my colleagues and I have had lengthy (but unfortunately inconclusive) discussions about how to represent the efficiency dimension that we expect many readers would like to visualize. The easier dimension--performance--became our first goal and the source code in Github is expected to very loosely fill the role of answering questions of efficiency. But we know that's a barely serviceable solution to the challenge, and that is especially true as pull requests increase the number of frameworks. The challenge of representing efficiency succinctly remains. We have considered lines of code and I think that for all its weaknesses, that is the best proxy for efficiency that I am aware of. Nevertheless, I'd like to contend with some of the following specific issues: 1. Many frameworks create boilerplate when you create a new application. Do we count those LOC? 2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right. :) 3. Do build scripts count as LOC? 4. Do configuration files count as LOC? Ultimately, I retain (irrational?) fear of contributing to enshrining LOC as a metric because unlike performance--where higher is always definitively better--lower LOC is not always definitively better. I was asked separately if we had any data about how long it took us to build tests in the various frameworks. We didn't take detailed logs, but we do have rough numbers. Nevertheless, I've been reluctant to share how long it took us to implement the test code because it's a biased sample. Our previous experiences make us well disposed to some platforms and languages while we make silly time-consuming mistakes on others.
    - kbenson 14y ago
      This is how I would answer these questions: > 1. Many frameworks create boilerplate when you create a new application. Do we count those LOC? No, as long as the lines did need to be changed. Any line you must alter (not just add code before/after) should be considered a line of code you had to write to get to a functioning implementation. You had to understand the line before altering, so could have written it entirely yourself. As such, diffing against a reference boilerplate file is a good indication of lines required. > 2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right. :) Take whatever you are given as a boilerplate starting point for a new project, and diff the final implementation tree against that. Special care may need to be given to implementation that default to installing third party libs into the local framework lib path (if any exist), so those are not counted. > 3. Do build scripts count as LOC? No, I would think not. > 4. Do configuration files count as LOC? This is a bit more complicated, but for simplicity's sake it may be easiest to treat it just like any other reference file as in #1 and #2. > Ultimately, I retain (irrational?) fear of contributing to enshrining LOC as a metric because unlike performance--where higher is always definitively better--lower LOC is not always definitively better. I agree, but without something better, it's what we have. I do think there's ways we could get to a more useful metric using relative lines of code required to implement something between languages, and then compare relative LOC to host language, but that's way outside the scope of this benchmark, and requires information we don't have (to my satisfaction) yet.

kbenson 14y ago

> Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen the benchmarks in question? Okay, let me clarify. When I say "you really need to go this far" I mean that stopping at any point before that (but after the simple metric of how many requests a second it can serve which they already do) makes no sense, IMHO. If you are going to compare frameworks and you want to go beyond that initial performance metric, you might as well aim high enough to be useful. I agree you never see anything approaching that in other reviews/benchmarks. Is that a good reason to not try it here? > Why not just go do it, if it's so essential and straightforward? I'm envisioning this as a community process, not a "Go off and write this in 20 frameworks or you're useless" sort or ultimatum. As such, just speccing out a possibly route is helping. Also, I plan to help with the existing benchmarks. After the second round, I pointed them out to the author of my favorite framework in the hopes he would have time to put together something for the benchmark, otherwise I was going to in the next week or two when I had time. I still plan to. Oh, and that framework author's answer? That these benchmarks are laughable because all they measure is performance, and there's a clear performance to convenience trade-off shown in the results, and that of course there's a performance hit when the framework handles most the work for you. I have to say I agree. Sure, there's possibly some that are clear winners giving good performance with lots of conveniences for common operations, but is there any way to tell as much from the data presented so far?

Terretta 14y ago
> I mean that stopping at any point before that (but after the simple metric of how many requests a second it can serve which they already do) makes no sense, IMHO. If you are going to compare frameworks and you want to go beyond that initial performance metric, you might as well aim high enough to be useful. The query 20 random rows, build an object, and return that, is way beyond requests per second. Enough that the list is dramatically re-ordered. > That these benchmarks are laughable because all they measure is performance, and there's a clear performance to convenience trade-off shown in the results, and that of course there's a performance hit when the framework handles most the work for you. I have to say I agree. On the contrary, performance versus magic is absolutely not a "given" here. Yes, some bare languages are near the top, but there are also heavy frameworks (e.g. Spring) performing well, and lean frameworks performing poorly. As for being "laughable", that makes me suspicious of the framework author's understanding of where optimization needs to happen. Presumably, pages will be run more often than they are authored. Presumably there's a recurring bill for the server farm. Optimizing for performance helps end users stay happy, and helps the company stay in business able to continue employing developers. > possibly some that are clear winners giving good performance with lots of conveniences for common operations, but is there any way to tell as much from the data presented so far? I agree with that point. I would like to see 3 additional columns added to the results: LOC, number of src files/templates across number of directories, and number of libs. This helps suss out your point: how much does a coder have to type in this framework, and how much incidental complexity (files, libs) do they have to wrap their heads around? Consider: [300 loc; 15 files 2 dirs; 8 libs] [650 loc; 1 file 1 dir; 0 libs] [150 loc; 15 files 15 dirs; 23 libs] Multiply by 10 to imagine real world code, and these would each feel very different to an author, and to a new hire hired to maintain a project already in production.
- bhauer 14y ago
  To your last point, Terretta, my colleagues and I have had lengthy (but unfortunately inconclusive) discussions about how to represent the efficiency dimension that we expect many readers would like to visualize. The easier dimension--performance--became our first goal and the source code in Github is expected to very loosely fill the role of answering questions of efficiency. But we know that's a barely serviceable solution to the challenge, and that is especially true as pull requests increase the number of frameworks. The challenge of representing efficiency succinctly remains. We have considered lines of code and I think that for all its weaknesses, that is the best proxy for efficiency that I am aware of. Nevertheless, I'd like to contend with some of the following specific issues: 1. Many frameworks create boilerplate when you create a new application. Do we count those LOC? 2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right. :) 3. Do build scripts count as LOC? 4. Do configuration files count as LOC? Ultimately, I retain (irrational?) fear of contributing to enshrining LOC as a metric because unlike performance--where higher is always definitively better--lower LOC is not always definitively better. I was asked separately if we had any data about how long it took us to build tests in the various frameworks. We didn't take detailed logs, but we do have rough numbers. Nevertheless, I've been reluctant to share how long it took us to implement the test code because it's a biased sample. Our previous experiences make us well disposed to some platforms and languages while we make silly time-consuming mistakes on others.
  - kbenson 14y ago
    This is how I would answer these questions: > 1. Many frameworks create boilerplate when you create a new application. Do we count those LOC? No, as long as the lines did need to be changed. Any line you must alter (not just add code before/after) should be considered a line of code you had to write to get to a functioning implementation. You had to understand the line before altering, so could have written it entirely yourself. As such, diffing against a reference boilerplate file is a good indication of lines required. > 2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right. :) Take whatever you are given as a boilerplate starting point for a new project, and diff the final implementation tree against that. Special care may need to be given to implementation that default to installing third party libs into the local framework lib path (if any exist), so those are not counted. > 3. Do build scripts count as LOC? No, I would think not. > 4. Do configuration files count as LOC? This is a bit more complicated, but for simplicity's sake it may be easiest to treat it just like any other reference file as in #1 and #2. > Ultimately, I retain (irrational?) fear of contributing to enshrining LOC as a metric because unlike performance--where higher is always definitively better--lower LOC is not always definitively better. I agree, but without something better, it's what we have. I do think there's ways we could get to a more useful metric using relative lines of code required to implement something between languages, and then compare relative LOC to host language, but that's way outside the scope of this benchmark, and requires information we don't have (to my satisfaction) yet.