6 ms·
Wow... some of these tests are still pretty severely hobbled. Is there some reason that you use built in json serialization for some frameworks and not others?
by bilbo0s 14y ago
Wow... some of these tests are still pretty severely hobbled.
Is there some reason that you use built in json serialization for some frameworks and not others?
There is also a lot of heterogeneity in the implementation of the multiple queries test. For instance, even if I only look at... say... java frameworks, you seem to implement the exact same feature in very different ways between platforms. For instance, for servlets, you will store all of the results in a simple array... and then write them out when you are done. Like so:
final World[] worlds = new World[count];
final Random random = ThreadLocalRandom.current();
try (Connection conn = source.getConnection())
{
try (PreparedStatement statement = conn.prepareStatement(DB_QUERY,
ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY))
{
// Run the query the number of times requested.
for (int i = 0; i < count; i++)
{
final int id = random.nextInt(DB_ROWS) + 1;
statement.setInt(1, id);
try (ResultSet results = statement.executeQuery())
{
if (results.next())
{
worlds[i] = new World(id, results.getInt("randomNumber"));
}
}
}
}
}
catch (SQLException sqlex)
{
System.err.println("SQL Exception: " + sqlex);
}
// Write JSON encoded message to the response.
try
{
Common.MAPPER.writeValue(res.getOutputStream(), worlds);
}
catch (IOException ioe)
{
// do nothing
}
}
But for other frameworks, like Vert.x, you use CopyOnWriteArray to store all of the results... and then write them out when you are done. Like so:
private final HttpServerRequest req;
private final int queries;
private final List<Object> worlds = new CopyOnWriteArrayList<>();
.
.
.
@Override
public void handle(Message<JsonObject> reply)
{
final JsonObject body = reply.body;
if ("ok".equals(body.getString("status")))
{
this.worlds.add(body.getObject("result"));
}
if (this.worlds.size() == this.queries)
{
// All queries have completed; send the response.
// final JsonArray arr = new JsonArray(worlds);
try
{
final String result = mapper.writeValueAsString(worlds);
final int contentLength = result.getBytes(StandardCharsets.UTF_8).length;
this.req.response.putHeader("Content-Type", "application/json; charset=UTF-8");
this.req.response.putHeader("Content-Length", contentLength);
this.req.response.write(result);
this.req.response.end();
}
catch (IOException e)
{
req.response.statusCode = 500;
req.response.end();
}
}
}
In other words, you literally create a new array each time you add a result to that CopyOnWriteArray. In fact, not only are you creating a new array, but you are creating new copies of the data in the array as well. Seems a little strange??? DEFINITELY inefficient. Is there a reason that is implemented differently? It seems to me that, at the least, they should both use arrays... but maybe there is something more you guys are testing???
The Onion C based code is written in an even MORE efficient manner for the multiple queries test. It actually stores it's results in json format from the outset! Like so:
json_object *json=json_object_new_object();
json_object *array=json_object_new_array();
int i;
for (i=0;i<queries;i++){
json_object *obj=json_object_new_object();
snprintf(query,sizeof(query), "SELECT * FROM World WHERE id = %d", 1 + (rand()%10000));
mysql_query(db, query);
MYSQL_RES *sqlres = mysql_store_result(db);
MYSQL_ROW row = mysql_fetch_row(sqlres);
json_object_object_add(obj, "randomNumber", json_object_new_int( atoi(row[1]) ));
json_object_array_add(array, obj);
mysql_free_result(sqlres);
}
json_object_object_add(json,"json",array);
const char *str=json_object_to_json_string(json);
The equivalent java code would be something like:
private final HttpServerRequest req;
private final int queries;
// INSTEAD OF:
//private final List<Object> worlds = new CopyOnWriteArrayList<>();
// HAVE:
private final JsonArray worlds = new JsonArray();
.
.
.
@Override
public void handle(Message<JsonObject> reply)
{
final JsonObject body = reply.body;
if ("ok".equals(body.getString("status")))
{
// INSTEAD OF:
//this.worlds.add(body.getObject("result"));
// HAVE:
this.worlds.addObject(body.getObject("result"));
}
if (this.worlds.size() == this.queries)
{
// All queries have completed; send the response.
// final JsonArray arr = new JsonArray(worlds);
try
{
// INSTEAD OF:
//final String result = mapper.writeValueAsString(worlds);
// HAVE:
final String result = worlds.encode();
final int contentLength = result.getBytes(StandardCharsets.UTF_8).length;
this.req.response.putHeader("Content-Type", "application/json; charset=UTF-8");
this.req.response.putHeader("Content-Length", contentLength);
this.req.response.write(result);
this.req.response.end();
}
catch (IOException e)
{
req.response.statusCode = 500;
req.response.end();
}
}
}
With a similar change for Servlets. According to the benchmark results, Onion comes out on top. It's the fastest. But how much of that's because it seems to be written correctly and other tests seem to be written without taking advantage of the same efficiencies.
Is it the case here that some people have sent you test code optimized for their own frameworks?
If that is so, you should add some tests that would not be so amenable to optimization. I'm not picking on Onion here by the way. In fact, the argument could be made that Onion is not actually 'optimized', so much as just written correctly, and the other frameworks have tests written incorrectly. But I just wanted to know if you guys actually intended to use these different implementations for some reason that I am unaware of? Do they make the tests more fair somehow???
- henk53 14y agoContribute and we'll see what it does in round 5 ;)
- bilbo0s 14y agoDo you guys take Git Pulls???
- kainsavage 14y agoWe are accepting pull requests on the GitHub page, yes.
- bhauer 14y agoHello bilbo0s. Thanks for taking the time to dig in and provide some feedback. As much as possible, we want each test to be representative of idiomatic production-grade usage of the framework or platform. Furthermore, we have solicited contributions from fans of frameworks and the frameworks' authors. A side objective is that the code double as an example of how best to use the framework or platform. All of this means we fully expect that the implementation approaches will vary significantly. The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests. We are not Vert.x experts and we have not yet received a community contribution for the Vert.x test. However, it is our understanding that idiomatic Vert.x usage encourages the use of asynchronous queries. The question then is: how do we collect the results into a single List in a threadsafe manner? Is your JsonArray alternative threadsafe? Admittedly, using a CopyOnWriteArrayList gave us pause, but we are not (yet) aware of a better alternative. The Onion test was contributed by a reader and admittedly its compliance with the specification we've created is perhaps a bit dubious. We want a JSON serializer to process an in-memory object into JSON. I'm not certain if the Onion implementation matches that expectation, but the test implementation nevertheless seemed sufficiently idiomatic for his platform. We're certainly open to more opinions on that matter.
- 14y ago