4 ms·
The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka. It doesn't really matter what the "source of tru
by manigandham 9y ago
The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka.
It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll need. Add in an elasticsearch cluster on the side and problem solved.
Any database can serve as a log replay, as long as you save all the versions then it's just called a query.
- robbyt 9y agoWhy do you think it's over engineered? Don't you think the Times have considerable capacity requirements?
- manigandham 9y agoThey claim in the article to have 100GB of text data. Let's bump that up 2 magnitudes for all text and metadata ever (outside of media files) and you can still run the entire thing on a single rack of servers and meet any performance needs. Many industries and applications are leagues ahead in both data size and speed - this isn't an example of such.
- seanp2k2 9y ago100GB...let me go find the largest MicroSD card in my house to put that on. That would actually be a good fit, since it'd work in an rPi3 which could likely serve their data publishing needs (assuming only a few updates to articles per second..not mentioned in the article, but I'd be surprised if there's more than that given the data sources) vs what they've done. Honestly, what kind of RPS are they talking here? Requests per minute, if that, seems like.
- John23832 9y ago>The very definition of over-engineered. I agree, it is a lot. However, it is an interesting approach to their problem. The over engineering makes the flow much easier to handle. >Any database can serve as a log replay, as long as you save all the versions then it's just called a query. The article addressed their potential issues with snapshotting
- manigandham 9y ago> The article addressed their potential issues with snapshotting They say it would be outdated - but if you store all the versions like I said, what is getting outdated? Kafka consumers are essentially doing the same thing - it's a poll-based model that asks for more data from the current offset, no different than a SQL query with a where clause.
- vikiomega9 9y agoUnless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc? Intuitively, I guess kafka is "more realtime" and "more available" when compared to the home-brew event log? EDIT: obviously those constraints in my home-brew event log are relaxed when my problem domain is amenable to things like associative operators, idempotency, inverses etc.
- cookiecaper 9y agoThere's no reason not to make good use of Kafka or similar solutions. The issue is that people use it without understanding it. In this article, they say that Kafka is their system of record and their primary long-term storage. That's very silly.
- subsubsub 9y agoWhy is it silly to use as system of record or long term storage?
- qaq 9y agoit's exact opposite the main cost is gc of dead rows in MVCC RDBMS since you are never deleting or updating rows performance will be very decent for writes.
- manigandham 9y agoYou can use Kafka as the buffer/processing log before persisting to the database, but with such a small dataset it's just not necessary. It's a news publishing system, not high-frequency trading.
- vikiomega9 9y agoWell my point is that it's probably faster to get to production if I simply used Kafka _when modelling my work flow as an event processing system_ but it took them a year so I don't know now haha
- ma2rten 9y agoYou could use any database, but Kafka has apis specifically for that. Why would you reinvent the wheel? What makes you think that Kafka is unstable?
- manigandham 9y agoEvent sourcing is unnecessary here. What they want is a versioned history of their content with flexible schemas and arbitrary queries. Instead of using a strong fast graph database as a perfect fit, they chose to implement it poorly using Kafka which I see as much more time wasted on reinventing the wheel. The dataset is small and all of the "very different" use-cases are just downstream apps that query a database. Why use Kafka to then materialize several different databases when a single graph database can serve all these downstream apps? Remember Kafka consumers themselves are just polling queries against a log. A processing log is one thing but event-sourced source of truth in custom database logic in Kafka is borderline ridiculous. This project is more moving parts and less functionality while cleaning none of the existing mess. Effort would've been better spent consolidating all their systems instead (which they still have to do since these standard schemas need to be used somehow).
- puranjay 9y agoReminds me of the NYT "Snowfall" story website that NYT spent $100k+ on producing, and some startup remade within a couple of days on Wordpress
- josephg 9y agoThat doesn't prove that the NYT's money was poorly spent. Decisions are always slow and expensive, and typing is quick. Copying an existing design is almost always cheaper than making the original. And in many cases, copying is dramatically cheaper. My favorite example is the iOS game Threes[1], which took 14 months to make and then was cloned on android in ~20 days. It was cloned so quickly that the original developers were accused of copying the cloned version! But just because the cloners made a functionally identical product doesn't mean they did the same work as the original designers. They got to skip designing anything - which is usually the hardest and slowest part. [1] http://asherv.com/threes/threemails/ http://asherv.com/threes/threemails/
- neeleshs 9y agoI was about to comment that while any database can serve as a log replay and may even be feasible for these volumes, scaling it at high volume later would be extremely hard - I have experience with this in a multi-petabyte set up and its a nightmare using an RDBMS. But, the article says "In Apache Kafka, the Monolog is implemented as a single-partition topic" - losing all the goodness Kafka provides around scaling. This setup now is no better than an RDBMS with master/slave replication.