3 ms·

First of all, great work. It's been a long time since I've done a weekend hack like this myself. However, it seems to me that Kafka is unnecessary in this syst

First of all, great work. It's been a long time since I've done a weekend hack like this myself. However, it seems to me that Kafka is unnecessary in this system. It's clear that, at least in the final version, the system isn't designed to scale beyond one application server. For one thing, you're storing the ban list on the local filesystem. So it's definitely not 12-Factor compliant. And you're storing a local snapshot of the image. So why send the edits out to Kafka, only to have them come back in to the same process?

josephg 10y ago
Even if I'm only using a single machine, using kafka allows me to load balance across a local cluster of server processes. This is important for nodejs apps because node is single threaded. And if I skipped kafka I'd still need to store the edit log somewhere for catchup. I've written that code a few times now, and I think its arguably more complicated to implement than just using kafka directly. But right now to help deal with load the process is running across 2 machines. I just had to manually copy the ban list and snapshot database files across. When the server came up it pulled the snapshot version out of the database file, caught up from the kafka log and went to work. Having a nice solution to distribute those files would be lovely - but I made the whole project start to finish in 2 days. I'm not going for 12 factor compliance here.
- mwcampbell 10y ago
  Ah, makes sense.