3 ms·

In my last long post I said Retric was wrong about providing data that's accurate only to +/- .2 seconds, but maybe I'm the one who is wrong. It is possible th

by owkaye 18y ago

In my last long post I said Retric was wrong about providing data that's accurate only to +/- .2 seconds, but maybe I'm the one who is wrong. It is possible that I completely misinterpreted his meaning. I became aware of this possibility when I read lacker's post that said I can probably batch up data since I'm going to have latency issues anyways. He's right, I never thought of it this way. The issue I'm imagining with batching is that I cannot send any data back to visitors until the next batch is processed -- because if I do they won't see how their last post has changed their positions in the hierarchy, and they will interpret this as a system error. In other words, every time a member receives a response from the server, that response must indicate his last post's position in the hierarchy. Theoretically I could run a batch every second, and although this means holding up all the responses to requests that come in during the previous second, one second is also short enough that they will not interpret it as a problem with the system. On the other hand, 5 seconds in between batches would certainly be seen as a server problem especially by folks on high speed connections.

mseebach 18y ago
So you problem is not processing 1.000.000 requests pr. minutes, but rather how to sort up to 60.000.000 items 1.000.000 times pr. minute? Now that could prove to be quite problematic. If your sorting algorithm only requires one processing instruction pr. stored item, and is perfectly parallelizable, and there's zero overhead in the map-reduce, you'll need 300 3 ghz cores to do it.
- lallysingh 18y ago
  Hmm, nlogn(60,000,000) ~= 1,560,000,000 ops per sort, before your coefficient. If you want that done at 16.67 KHz (1 million times/min), you'll need some work. You can do it, in certain cases, with certain hardware & good skills, but it's tricky. I'm still not buying that the requirements are really that high.
anamax 18y ago
> The issue I'm imagining with batching is that I cannot send any data back to visitors until the next batch is processed -- because if I do they won't see how their last post has changed their positions in the hierarchy, and they will interpret this as a system error. In other words, every time a member receives a response from the server, that response must indicate his last post's position in the hierarchy. You're making this much harder than it has to be because you're assuming that users can see things that they can't. Suppose that the system had two users, A and B, and they each issued two commands, A1 and A2 in that order from A and B1 and B2 from B. Suppose further that system responds to each command with something that it says is the global state. To keep things simple, I'll use the command, such as A2, to refer to the response as well. (In other words, A2 is the response that A gets to command A2.) A knows that she issued A1 and A2 in that order, so the A2 response must reflect commands A1 and A2. However, the A2 response need not reflect B1 or B2 unless A has some way to know that B issued them before A2, regardless of when B1 and B2 were issued in relation to A2. In fact, it's probably okay for the A2 response to reflect command B2 and not command B1. Now let's throw C into the mix, issuing C1 and C2 in that order. As with B, the A2 response need not reflect C1 or C2 even if they actually occurred before the A2 command. Moreover, even if the C1 and C2 commands both occurred before B1, the A2 response can reflect B2 and not C1 or C2. The easist thing to do is to periodically produce consistent views of the data. This process will take a while so while it is happening, you'll have to use a past consistent view. Three views, call them X, Y, and Z that the system updates should be adequate. At all times, one view will be "current", one will be being generated, and the third will be a source for stragglers. Here's how the update process works. While X is being generated, from Z and all data received since Z's generation started, Y will be used to produce results to send to users. While Y is being generated, from X and all data received since X's generation started, X will be used to produce results to send to users. And similarly for Z (using Y). Suppose that user A sends a command while Y is being generated from X. The response to A need only reflect X and commands that A sent since X was generated. As long as that can be finished before the system finishes generating Y and Z, everything is copacetic.

by owkaye 18y ago

mseebach 18y ago

So you problem is not processing 1.000.000 requests pr. minutes, but rather how to sort up to 60.000.000 items 1.000.000 times pr. minute? Now that could prove to be quite problematic. If your sorting algorithm only requires one processing instruction pr. stored item, and is perfectly parallelizable, and there's zero overhead in the map-reduce, you'll need 300 3 ghz cores to do it.

lallysingh 18y ago
Hmm, nlogn(60,000,000) ~= 1,560,000,000 ops per sort, before your coefficient. If you want that done at 16.67 KHz (1 million times/min), you'll need some work. You can do it, in certain cases, with certain hardware & good skills, but it's tricky. I'm still not buying that the requirements are really that high.

anamax 18y ago

> The issue I'm imagining with batching is that I cannot send any data back to visitors until the next batch is processed -- because if I do they won't see how their last post has changed their positions in the hierarchy, and they will interpret this as a system error. In other words, every time a member receives a response from the server, that response must indicate his last post's position in the hierarchy. You're making this much harder than it has to be because you're assuming that users can see things that they can't. Suppose that the system had two users, A and B, and they each issued two commands, A1 and A2 in that order from A and B1 and B2 from B. Suppose further that system responds to each command with something that it says is the global state. To keep things simple, I'll use the command, such as A2, to refer to the response as well. (In other words, A2 is the response that A gets to command A2.) A knows that she issued A1 and A2 in that order, so the A2 response must reflect commands A1 and A2. However, the A2 response need not reflect B1 or B2 unless A has some way to know that B issued them before A2, regardless of when B1 and B2 were issued in relation to A2. In fact, it's probably okay for the A2 response to reflect command B2 and not command B1. Now let's throw C into the mix, issuing C1 and C2 in that order. As with B, the A2 response need not reflect C1 or C2 even if they actually occurred before the A2 command. Moreover, even if the C1 and C2 commands both occurred before B1, the A2 response can reflect B2 and not C1 or C2. The easist thing to do is to periodically produce consistent views of the data. This process will take a while so while it is happening, you'll have to use a past consistent view. Three views, call them X, Y, and Z that the system updates should be adequate. At all times, one view will be "current", one will be being generated, and the third will be a source for stragglers. Here's how the update process works. While X is being generated, from Z and all data received since Z's generation started, Y will be used to produce results to send to users. While Y is being generated, from X and all data received since X's generation started, X will be used to produce results to send to users. And similarly for Z (using Y). Suppose that user A sends a command while Y is being generated from X. The response to A need only reflect X and commands that A sent since X was generated. As long as that can be finished before the system finishes generating Y and Z, everything is copacetic.