2 ms·
Any reason why you don't send the data directly to the processing servers as a way to minimize complexity? If it fails to reach one, try the other. When you me
by halayli 12y ago
Any reason why you don't send the data directly to the processing servers as a way to minimize complexity? If it fails to reach one, try the other.
When you mention low latency, how low are we talking? ms, seconds, minutes? The reason I am asking is because you can use s3 as an intermediate storage where you ship your compressed logs/events at a rollover interval and the processing servers discover them there. Now of course this only works if latency is not a big deal.
- itaifrenkel 12y agoOur first architecture was indeed peer-to-peer where all components discovered each other. It is difficult to isolate failures on this kind of architecture. A reliable queue makes a big difference in zooming in on the root cause of the problem (a queue producer or a queue consumer problem). Regarding latency - we are aiming for a few ms tops. preferably less. So bulk events using S3 was not considered
- halayli 12y agoI'd argue that queues can introduce a lot of similar problems. Especially when there's slowness (backing up, slow disk etc...). Messages over tcp sent directly to the processors are not that hard to design for failures and high availability and probably your'll reduce the latency significantly. You can also get rid of auto discovery and use config files.