4 ms·

From the "JSON serialization" requirements: > For each request, an object mapping the key message to Hello, World! must be instantiated. Why mandate how the b

From the "JSON serialization" requirements: > For each request, an object mapping the key message to Hello, World! must be instantiated. Why mandate how the benchmark is implemented? What if some frameworks could be faster by not instantiating an object? For maximum speed, you'd want to stream-process the input into output while materializing as few intermediate objects as possible. Benchmarks should be defined in terms of their observable inputs/outputs. Otherwise it's like defining a car's 0-60 measurement with a rule that says "the car must then take gas from its tank and inject it into the engine." And then a Tesla could only compete by adding a gas engine that it doesn't need or want.

meepmorp 13y ago
For JSON serialization, I'd assume it's because they're testing object serialization.
- haberman 13y ago
  Clients of a system don't know or care if the server actually constructed an object and serialized it. They care that they get the reply they expect, and do not care how it was constructed.
  - meepmorp 13y ago
    Right, but then you're just testing how fast a framework can vomit out a small string and some headers. I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
    - haberman 13y ago
      See: https://news.ycombinator.com/item?id=6650499 https://news.ycombinator.com/item?id=6650499
      - meepmorp 13y ago
        I wholeheartedly agree. But like said: > I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
- mhixson 13y ago
  Right. Related to this, I feel like a couple of frameworks are cheating on the plaintext test because they aren't actually testing string... serialization. They encode the "Hello, World!" string into bytes once, retain those bytes in memory, and send those bytes on each request just to avoid encoding the string again. It's such a small, silly optimization. It doesn't bother me because it's not going to affect the overall results much, but I do feel it's against the spirit of the test. That kind of thing would be prevented if we made the test echo a request parameter, as was suggested elsewhere in the comments.
bhauer 13y ago
I want the test to require the work of creating an object or allocating the memory for the response and then use of a JSON serializer. These tests are intended to be stand-ins for realistic application behaviors. They are not realistic applications, but they should make use of the same functionality that a real application would. I want the tests to establish realistic high-water marks for the types of operations they exercise. Running a single query per request? The high-water mark you might expect is what you see as our single-query test. It would be very difficult to make a single-query operation in a real application more trivial than what we have. If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back. Bottom line: nothing beats benchmarking your own application. But the tests in this project are intended to give a first-pass filter of sorts, assuming performance is on your list of requirements.
- haberman 13y ago
  > If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back. A better way to avoid this would be to make the output depend on the input. Require that the reply include some JSON that contains portions of the request. Likewise for databases, require that the reply contain contents of a previous request. If you want to, make the request contain an ID and make the reply return the data keyed under that ID. If you want to require that the data is persisted durably, require that the benchmark continue to function even if all processes are killed and restarted in the middle of a run. What really matters about a system are its inputs and outputs. If an innovative system comes along that works differently inside, it shouldn't be disqualified from a benchmark just for being innovative.
  - emn13 13y ago
    That wouldn't work sufficiently well; unless the benchmark is quite complex it would almost always be better to implement an in-memory custom datastore rather than use a traditional backend. Even if you require persistance, since the intentionally small dataset means you can get away quite well with raw memory dumps or even an mmap variant. You wouldn't use a full-fledged JSON serializer but a limited string concatenation. In any case, the way I read it, the requirement doesn't mean you have to make anything particularly heavy, just that you need to use a representation that's general enough to be usable in other scenarios without rewriting the entire app. Of course, a little bit of extra encouragement in the form of a benchmark that's slightly harder to game wouldn't be bad.
thezilch 13y ago
You're going to stream a Hello, World value? Why use JSON, at all, when we can just print the resulting JSON text! Benchmarks should be defined, that's all.
- haberman 13y ago
  > Why use JSON, at all, when we can just print the resulting JSON text! Indeed. But if you want to actually force the server to encode some JSON, then make the reply include portions of the request. Then you don't have to have artificial rules about how it is implemented to get interesting/meaningful results: https://news.ycombinator.com/item?id=6650499 https://news.ycombinator.com/item?id=6650499
  - emn13 13y ago
    No, you still need to, because you can otherwise do a trivial string concat in most cases, even though in reality that would be most unwise for security, robustness, and maintainability reasons.
    - haberman 13y ago
      Since you don't know the inputs ahead of time, the benchmark would be incorrect if it could not handle the corner cases like special characters that need to be escaped in JSON strings. If you wanted you could also make the request specify the key at which a value must be written. Be creative. :) But don't think that apps always have to be written the way they are now to be "legitimate."
ahoge 13y ago
> Why mandate how the benchmark is implemented? Otherwise you won't be able to compare the results. For example, the Benchmarks Game [1] specifies how the algorithms in those benchmarks should work. For some of the benchmarks, there are faster algorithms. However, it's not about comparing algorithms, it's about comparing language implementations. [1] http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/

meepmorp 13y ago

For JSON serialization, I'd assume it's because they're testing object serialization.

haberman 13y ago
Clients of a system don't know or care if the server actually constructed an object and serialized it. They care that they get the reply they expect, and do not care how it was constructed.
- meepmorp 13y ago
  Right, but then you're just testing how fast a framework can vomit out a small string and some headers. I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
  - haberman 13y ago
    See: https://news.ycombinator.com/item?id=6650499 https://news.ycombinator.com/item?id=6650499
    - meepmorp 13y ago
      I wholeheartedly agree. But like said: > I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
mhixson 13y ago
Right. Related to this, I feel like a couple of frameworks are cheating on the plaintext test because they aren't actually testing string... serialization. They encode the "Hello, World!" string into bytes once, retain those bytes in memory, and send those bytes on each request just to avoid encoding the string again. It's such a small, silly optimization. It doesn't bother me because it's not going to affect the overall results much, but I do feel it's against the spirit of the test. That kind of thing would be prevented if we made the test echo a request parameter, as was suggested elsewhere in the comments.

bhauer 13y ago

I want the test to require the work of creating an object or allocating the memory for the response and then use of a JSON serializer. These tests are intended to be stand-ins for realistic application behaviors. They are not realistic applications, but they should make use of the same functionality that a real application would. I want the tests to establish realistic high-water marks for the types of operations they exercise. Running a single query per request? The high-water mark you might expect is what you see as our single-query test. It would be very difficult to make a single-query operation in a real application more trivial than what we have. If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back. Bottom line: nothing beats benchmarking your own application. But the tests in this project are intended to give a first-pass filter of sorts, assuming performance is on your list of requirements.

haberman 13y ago
> If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back. A better way to avoid this would be to make the output depend on the input. Require that the reply include some JSON that contains portions of the request. Likewise for databases, require that the reply contain contents of a previous request. If you want to, make the request contain an ID and make the reply return the data keyed under that ID. If you want to require that the data is persisted durably, require that the benchmark continue to function even if all processes are killed and restarted in the middle of a run. What really matters about a system are its inputs and outputs. If an innovative system comes along that works differently inside, it shouldn't be disqualified from a benchmark just for being innovative.
- emn13 13y ago
  That wouldn't work sufficiently well; unless the benchmark is quite complex it would almost always be better to implement an in-memory custom datastore rather than use a traditional backend. Even if you require persistance, since the intentionally small dataset means you can get away quite well with raw memory dumps or even an mmap variant. You wouldn't use a full-fledged JSON serializer but a limited string concatenation. In any case, the way I read it, the requirement doesn't mean you have to make anything particularly heavy, just that you need to use a representation that's general enough to be usable in other scenarios without rewriting the entire app. Of course, a little bit of extra encouragement in the form of a benchmark that's slightly harder to game wouldn't be bad.

thezilch 13y ago

You're going to stream a Hello, World value? Why use JSON, at all, when we can just print the resulting JSON text! Benchmarks should be defined, that's all.

haberman 13y ago
> Why use JSON, at all, when we can just print the resulting JSON text! Indeed. But if you want to actually force the server to encode some JSON, then make the reply include portions of the request. Then you don't have to have artificial rules about how it is implemented to get interesting/meaningful results: https://news.ycombinator.com/item?id=6650499 https://news.ycombinator.com/item?id=6650499
- emn13 13y ago
  No, you still need to, because you can otherwise do a trivial string concat in most cases, even though in reality that would be most unwise for security, robustness, and maintainability reasons.
  - haberman 13y ago
    Since you don't know the inputs ahead of time, the benchmark would be incorrect if it could not handle the corner cases like special characters that need to be escaped in JSON strings. If you wanted you could also make the request specify the key at which a value must be written. Be creative. :) But don't think that apps always have to be written the way they are now to be "legitimate."

ahoge 13y ago

> Why mandate how the benchmark is implemented? Otherwise you won't be able to compare the results. For example, the Benchmarks Game [1] specifies how the algorithms in those benchmarks should work. For some of the benchmarks, there are faster algorithms. However, it's not about comparing algorithms, it's about comparing language implementations. [1] http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/