3 ms·

> If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and r

> If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back. A better way to avoid this would be to make the output depend on the input. Require that the reply include some JSON that contains portions of the request. Likewise for databases, require that the reply contain contents of a previous request. If you want to, make the request contain an ID and make the reply return the data keyed under that ID. If you want to require that the data is persisted durably, require that the benchmark continue to function even if all processes are killed and restarted in the middle of a run. What really matters about a system are its inputs and outputs. If an innovative system comes along that works differently inside, it shouldn't be disqualified from a benchmark just for being innovative.

emn13 13y ago
That wouldn't work sufficiently well; unless the benchmark is quite complex it would almost always be better to implement an in-memory custom datastore rather than use a traditional backend. Even if you require persistance, since the intentionally small dataset means you can get away quite well with raw memory dumps or even an mmap variant. You wouldn't use a full-fledged JSON serializer but a limited string concatenation. In any case, the way I read it, the requirement doesn't mean you have to make anything particularly heavy, just that you need to use a representation that's general enough to be usable in other scenarios without rewriting the entire app. Of course, a little bit of extra encouragement in the form of a benchmark that's slightly harder to game wouldn't be bad.
- barrkel 13y ago
  Custom datastores (custom at the framework level, not necessarily at the application level) have been very useful for me in the past for getting high performance. For example, I wrote a little in-memory DB for use in a .NET framework to avoid having to deserialize an object graph per request. Instead, it navigated a byte array and plucked out just the data needed. Another time, I wrote a compact in-memory representation of a trie, using bit-packing and a few other tricks, to get an order of magnitude reduction in memory usage, making it possible to cache a lot more of a data set. Why go to the database and instantiate a classic object if you can avoid it?

emn13 13y ago

That wouldn't work sufficiently well; unless the benchmark is quite complex it would almost always be better to implement an in-memory custom datastore rather than use a traditional backend. Even if you require persistance, since the intentionally small dataset means you can get away quite well with raw memory dumps or even an mmap variant. You wouldn't use a full-fledged JSON serializer but a limited string concatenation. In any case, the way I read it, the requirement doesn't mean you have to make anything particularly heavy, just that you need to use a representation that's general enough to be usable in other scenarios without rewriting the entire app. Of course, a little bit of extra encouragement in the form of a benchmark that's slightly harder to game wouldn't be bad.

barrkel 13y ago
Custom datastores (custom at the framework level, not necessarily at the application level) have been very useful for me in the past for getting high performance. For example, I wrote a little in-memory DB for use in a .NET framework to avoid having to deserialize an object graph per request. Instead, it navigated a byte array and plucked out just the data needed. Another time, I wrote a compact in-memory representation of a trie, using bit-packing and a few other tricks, to get an order of magnitude reduction in memory usage, making it possible to cache a lot more of a data set. Why go to the database and instantiate a classic object if you can avoid it?