8 ms·
Many use cases don’t require Kafka (2019)
- thereare5lights 6y agoYAGNI
- nova22033 6y agoThis article is from Jun 21, 2019 and it describes WeWork's use of kafka. Not exactly the success story you want to get lessons from.
- doseofreality 6y agoMaybe it should have been an article on: "You Don't Need WeWork"
- nrmitchi 6y agoHonestly, I've only skimmed the article, but this comment rubbed me the wrong way right off the bat. The article (discussing use of Kafka) doesn't seem to have any direct relation to WeWork's failure. There is a lot that you can learn especially from businesses that failed. Just because the company wasn't successful does not mean you should discount everything related to that company, especially when the content and lessons are technical, and the business failed for non-technical reasons (shady business practices and self-dealing).
- Vaslo 6y agoJust getting in here to say someone should edit the HN title to have a date. Hearing about success at WeWork now is akin to being in accounting and finance in the late 90s doling out praise for Arthur Anderson.
- ashkankiani 6y agoSince this is going to get some attention and might attract some welcome naysayers, it might be a good time for me to ask for resources if I do want to use Kafka, specifically edge cases that I won't see as a newcomer until I actually try to use it in production. I've already done my due dilligence and I consider Kafka to be one of the potential solutions I could use for a service, and while I've gotten started using it fairly easily in a dev env, there's obviously things I could miss. Any pointers from experienced users would be appreciated.
- gkop 6y agoIn case you haven't already read it, do read the classic LinkedIn "Log" article: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... I just learned Kafka this year to augment an existing system at work. One thing I have found is, there's a good amount of smarts in the Kafka clients. Look carefully at the Kafka clients available for the runtimes you work with, and evaluate them for robustness, features, and development activity (sadly, for example, kafka-node seems to have lost all momentum).
- mr_tristan 6y agoYeah, I found the article to be a little annoying in that it's mostly attacking WeWork. My recommendation for anyone new to the area, is the streaming systems book: http://streamingbook.net/ http://streamingbook.net/, which came after two excellent blog posts: https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/ https://www.oreilly.com/radar/the-world-beyond-batch-streami..., and https://www.oreilly.com/radar/the-world-beyond-batch-streaming-102/ https://www.oreilly.com/radar/the-world-beyond-batch-streami... Basically, if you have an unbounded data problem, and you might have something that is _practically_ unbounded, that book really helped me gain insight to describe what we used Kinesis for (which is really similar to Kafka in a lot of ways). Thinking about event vs processing time, ordering and windowing requirements, etc, were really helpful in thinking about my own data problems. In the end, the author of this article claimed that WeWork just simply didn't have much of an unbounded data problem. Which I generally agree with. But simply saying "you can do this in PostgreSQL" isn't really a great takeaway, and, it really feels that was the message being reinforced here.
- kyrieeschaton 6y agoThe important question for any piece of analysis is "would we actually do anything different if this comes out one way or another?"
- manish_gill 6y agoThe general trend in these posts (and tweet storms) is the same - pick any technology and then preach the wisdom of not using it because "You're not Google", or "You won't need it". It's a great tactic as it puts the speaker in a position of judgement, and they can freely share their opinions on some architecture, without getting any pushback from the creators of said architecture. Quite a one-sided affair that makes you look like you know everything. It's, frankly, elitist. This entire article can be surmised by "You don't need Kafka because the throughput seems to be low". To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka. But of course, "You don't need Kafka" because <reasons>.
- hyperpape 6y ago> To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka. WeWork has very predictable scale because its data ingestion needs are tied to expensive investments in physical capital. They don't have to worry about getting slashdotted. Same thing at my company, where data needs are tied to expensive enterprise contracts. In fact, we're considering Kafka for something we do, but it didn't sneak up on us--we knew for years that a particular use case involved a lot of data and wasn't ideal for a relational database.
- kumarvvr 6y ago> It's, frankly, elitist Why not see the coin from the other side. Developers giving their experience when they are working sane ? Many of these articles, I have seen authors quoting from their experience building large systems. Moreover, as a system architect, you often have to analyze the requirements and choose a good stack for implementation. A young dev in a startup is biased against the architecture in one way, because they might be seeing Kafka on their resume as a shining star to greater riches.
- 6y ago
- zekrioca 6y agoThese types of articles seem to only serve a purpose of getting clicks..
- sk5t 6y agoHas Substack emerged as "new Medium" for ambitious but not-as-seasoned-as-they-would-appear tech diarists?
- alextheparrot 6y agoLooking forward to it coming full circle when Twitter fully fleshes out the “Tweetstorm” as a product. Long-form content, high intractability
- sk5t 6y agoKind of a lengthy ad hominem against WeWork, no?--and in an alternate reality where they did in fact have an IPO. I love Postgres but "just use Postgres for your small data" is IMHO a facile argument to an ambitious, naturally distributed IoT project. The fact that WeWork's IoT strategy is/was perhaps useless is rather beside the point.
- longcommonname 6y agoWhen you have a monolith database that everybody wants to access Kafka with CDC is a very good pattern for teams to adopt. The alternative is modifying your app for arbitrary data access and dealing with auth or giving direct data access. Giving direct access to your database is fast but leads to very strong data coupling. Exposing the data through the app layer is slower than cdc kafka integration but comes with a long term commitment to support that access. I have seen these type of data access queries easily hit performance edge cases. There are other pros and cons but those are the ones that end up being the most troubling. Often a single apps data can be incredibly valuable and many teams want to access it. So coming up with patterns that 1. Protect the app 2. Protect the data store 3. Decrease the maintenance cost of the app 4. Decrease maintenance cost of the datatype 5. Add sight complexities from kafka. Can be very valuable.
- derefr 6y ago> Giving direct access to your database is fast but leads to very strong data coupling. Or you could just give "direct" access to (a DB role that only has been granted SELECT on) a single schema within your database, entirely consisting of views. Then you've got a stable data-access API that you can iterate your data model behind, with nothing more than your DB itself. (And you don't need to put much thought into setting it up, if you don't want to; your stable schema-API-layer can just start off as a bunch of views with 1:1 relationships to your real tables, which each just "SELECT * FROM their_underlying_table".)
- philipkglass 6y agoKafka is (IME) pleasant to use for people writing application code. It can handle high message volumes. It has good documentation. Its core abstraction is easy to understand and use. OTOH, the author is correct that it's more work to set up and maintain than non-distributed software. Don't introduce it unless you have carefully considered alternatives. I became de facto responsible for running a small Kafka cluster at a small company that could have gotten away with something less shiny. That was not fun. (Especially since the company started with Kafka 0.8 and encountered many changes along the way to modern releases). Eventually the company ponied up for Confluent to take over running Kafka for us as a service. Now I'm back to enjoying the nice Kafka APIs without having to deal with the annoying Kafka servers. I like the Kafka API/capabilities but dislike running actual Kafka servers. If I were to start a one-man venture that needed event streaming I'd probably try Redpanda [1]. Its developer hangs out here, which is how I first learned of it. It's supposed to implement Kafka service but with no Zookeeper or JVM knowledge needed to run it. I wouldn't even consider running the original Apache Kafka for a solo/small team. [1] https://vectorized.io/ https://vectorized.io/
- rjgonza 6y ago100% agree on redpanda
- deleted 6y ago[deleted]
- agallego 6y agothat's correct - we actually don't manually write the kafka api, but we codegenerate it: https://github.com/vectorizedio/redpanda/blob/dev/src/v/kafka/protocol/schemata/generator.py#L576-L631 https://github.com/vectorizedio/redpanda/blob/dev/src/v/kafk... the tl;dr is that we wrote a pretty fast fragmented buffer parser (due to our thread-per-core design), we consume some json files and codegen it.... which basically means, if you connect an existing app that uses the kafka driver and it doesn't work it's a bug, and it's a matter of teaching the codegen to fix it most of the time. Transactions should land in the next month or so, which is the only thing really missing today. we have a ton of solo devs typically that come from the Go, Rust, JS & Python seem to like it quite a bit because of what you said.
- cortesoft 6y agoI always find these posts ironic, because I work at a company where we are often times dealing with the opposite problem; trying to redesign systems that worked fine when we were a small startup but have hit scaling issues now that we are many times the size. People are always so quick to comment, “don’t worry about scaling... if you are successful, you can redesign at that point.” However, it isn’t always easy to redesign later. Now people are relying on your systems, and you can’t have any down time. All redesigns have to be able to be implemented in a way that can transition from the previous system without breaking anything. As we often say, “we are changing the tires while driving 90mph down the road”. I wish we had spent more time on thinking about scale when we were smaller. Our new mantra is that everything we make now has to be designed to horizontally scale, even if it is something small right now.
- rntksi 6y agoI have had the same issue, and wishing we thought about that up the road rather than down the road is exactly the feeling when we huddle up for team meetings on optimisation. We could not have any down time for most days of the year.
- gfodor 6y agoEverybody at a company experiencing scaling issues feels the way you do. The argument however is if the past went the way you wish it did, arguably the company you currently work for would not exist, because they would have lacked the knowledge that you have now to know which areas were the ones that would matter. Absent that knowledge, they would have built “scalable” systems that still had bottlenecks they could not yet see and also would have adopted extraneous technology that ultimately would be thrown out as the product discarded the parts that ended up not being critical to the value proposition uncovered. And of course, this whole process would have burned time, energy, and cash, not to mention shifted the nature of hires and the composition of the team itself, such that in sum it may have been a sufficient opportunity cost to jeopardize the whole business.
- cortesoft 6y agoYeah, I know this is true, and we always try to remember to understand that the engineering decisions that were made at the time were made for good reasons given the circumstances we were facing. It helps that I was one of the people who made those decisions at the time, so I know we were making what we thought were the best decisions at the time. Still wish we had made some different ones, though!
- keyle 6y ago> Overarchitecture is real Amen. I've been through some interviews where I'm the oldest at the table and they tell me their stack and I try to keep a straight face. It's hard. Hard to keep a straight face because I'm a genuine honest person with a lot of experience. "We plan to hire 15 people within 3 months and double year after year". Well, yeah, with every acronyms AWS could come up with, using them all for no reasons at all (except S-C-A-L-I-N-G), you could build it all rock solid on a mid-size linode and do it with 5 dudes Expected: world domination, runway blast-off to the moooooon Actual: as many employees as concurrent users, grind for the next 2 years and run out of money Reality test: failed.
- MaxBarraclough 6y agoPlanning for scale that will probably never happen is a specific instance of the YAGNI principle, and no doubt a particularly expensive one.
- nicoburns 6y ago> I've been through some interviews where I'm the oldest at the table and they tell me their stack and I try to keep a straight face. > Well, yeah, with every acronyms AWS could come up with, using them all for no reasons at all (except S-C-A-L-I-N-G), you could build it all rock solid on a mid-size linode and do it with 5 dudes I've been through some interviews where I was the youngest person at the table, but it otherwise sounds like I had a very similar experience to you (they had ~1000 total users - their "scaling issues" were entirely due to poorly architected code). In my case, I politely questioned it during the interview and they actually ended up letting me take over and simplify things.
- keyle 6y agoI'm glad to hear that actually, cool ending to that story. I'm yet to see such experience first hand, they're usually too committed down the line.
- franklampard 6y agoWhy do you think it’s more cost effective to build a service on linode than just use AWS?
- marcinzm 6y agoWeWork had about 1500 engineers at its height (the rationality of that is a separate discussion). At that scale the limitation isn't having to maintain some technology but rather decoupling interfaces so communication and bureaucracy doesn't block engineers from getting work done. You can use Postgres but then getting access, schemas, resource management, coupling, and so on become the limiting factors on usage. Paying for 5 more engineers to maintain Kafka and unblocking 50 in the process is a net gain.
- breckenedge 6y agoI last used Kafka in 2015 and I really miss it. Used it as a message broker for loads of cellular IOT devices. It was critical for helping us extract services out from a Rails 2.3/PostGIS 1x monolith, especially its ability to store messages for a period of time while a consumer was offline.
- boredandroid 6y agoIn my opinion this is pre-cloud thinking. It used to be that distributed systems were a big trade off. They were operationally complex, they had limited apis (NoSQL), but they scaled. The best solution used to be to build things using a non-scalable but easy to use and run system, and then re-write it later if it needed to scale (often in a big hurry). This is just not the case any more, though. Why? Two reasons: 1. We’ve gotten much better at distributed systems so the apis aren’t nearly as limited. It’s no longer that you either choose hard-to-use things like Hadoop/NoSQL or elegant but unscalable single-server databases. You can have both good abstractions and scale. 2. The cloud makes it possible to get systems as a service so there should be way less ops than running a single node system yourself In the case of Kafka, I’m super biased as I’m one of the original authors, but I think the abstractions Kafka gives, stream processing capabilities, connectors, etc are just way better than a lot of the traditional solutions. Using something worse until you “need” Kafka might make sense on premise, but not in the cloud. Confluent offers a Kafka service which is fully managed so you don’t do any of the upgrades, security patches, midnight pages, etc you just use the APIs. This is super affordable for the kind of simple apps the article describes. The price varies by cloud, but e.g. on GCP it starts at $0.11/GB for reads and $0.10/GB stored. That is a lot cheaper than using a single node system and then rebuilding everything if you need to scale, but not only that, it is also lower operational overhead (effective none) and a better interface/abstraction. I think this isn’t unique to Kafka, either. There are great managed systems that are built to scale for most of the kinds of data systems you would use—-CockroachDB, Spanner, Aurora, Snowflake, Elasticsearch, Bigquery, etc. Basically, you can have nice things now, just like the big tech companies.
- dan_quixote 6y agoThe problem I see with kafka is that it was built before cloud architectures were commonly adopted (with distributed systems everywhere). Confluent has put a lot of effort dragging kafka's architecture to the present, but some major features are missing: - Auto-scaling: Confluent finally introduced "elastic scaling" a few months ago but it only allows you to scale up and must be triggered by the admin (no threshold-based auto-scaling). - Multi-tenancy: Planning for a multi-tenant kafka cluster is not for the faint of heart. Achieving isolation tends toward liberal usage of topics of which starts to become unmanageable in the low thousands. This isn't crazy when you've got a few hundred microservices and several tenants to keep isolated. - Decoupled brokers and storage: Any broker scaling or failure can lead to downtime while event storage is redistributed. Confluent's Cloud service reduces operational overhead but isn't always feasible due to cost, resource limits (like service accounts or schemas for instance), data controls, etc.
- jpgvm 6y agoUtter dribble. Kafka is an industry standard, it's benefits go well beyond its own qualities and extend into the ecosystem around it. Don't attempt to replace industry standard components with any home rolled systems unless you were already capable of building said system yourself and you understand why you are a special snowflake and why you need something special.
- josephg 6y agoCan you make your point without the appeal to a (missing) authority? There is no standards document I've ever seen which names kafka explicitly. Kafka has some benefits (lovely API) and some tradeoffs (hard to maintain, zookeeper is a bit of a mess). But there are obviously a lot of other ways to build successful, scalable software. WhatsApp was implemented on top of Erlang, and scaled to over 500m users when they had only 35 employees. I want to disagree with your claim that kafka should be considered the default option - but you haven't dignified your (controversial) claim with any actual arguments. There's nothing here to even disagree with.
- kirktrue 6y agoThere is work in progress to remove the Zookeeper dependency: https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/ https://www.confluent.io/blog/removing-zookeeper-dependency-...
- jpgvm 6y agoIt should be considered the default option because everything works with it out of the box and for pretty much all Big Data folk it's considered table stakes. All major streaming event processors are designed around it or work seamlessly with it (Spark, Beam, Flink, Storm, etc). There are pre-existing integrations for all long term storage options (S3, GCS, Swift, etc). Maybe it could have been elaborated in the original comment but it shouldn't need to be - all of this is about 10 minutes research away from anyone that wants to understand why Kafka is considered the core of modern data architectures. WhatsApp isn't really a relevant counter-example and if anything reveals the misconception yourself and many others hold about Kafka which is that it's a messaging solution. It's not. You can bastardize one on top of it but really, don't. WhatsApp was built around RabbitMQ which -is- a messaging solution (one of the more complex ones, crazy complicated compared to Kafka at least). Kafka -doesn't- compete with RabbitMQ any more than MongoDB competes with PostgreSQL. Apples and oranges if you are trying to do anything but the most basic stuff. Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place?
- yurielt 6y agoI really feel that FF VIwould be quite a different game without him tho
- fakecrusade 6y agoLMAO
- netik 6y agoAside from pissing all over WeWork, which given their immense losses they probably deserve, This article could have been summed up in a couple of short sentences. There's a lot of words and effort going on here to make Kafka sound immensely complex (and dare I say Kafka-esque?) The article continually chides Kafka as overweight, hard to setup, complex, blah blah blah. None of these things are true in modern day deployments. Three instances of Kafka, Two ZK instances, and you're off and running. All of that will run in docker and the docker images will stand themselves up if you let them. You need even less for developemnt. Postgres is a bit more complex given that you've got to manage machine sizes and database schemas. But even with that being said, the article gets one thing right. For average data use, most any system will do and some startups are over-engineering their stacks.
- barrkel 6y agoYou don't need Kafka if you don't know (and I mean really know, because it's hurting) where you need it. But if you do know where you need it, you wish you had it yesterday. This is the way with scaling in startups. Product-market fit and agility and lean operations is what you want early on, and scaling should come later. Big architectural decisions are hard to reverse and committing to some pattern that needs something like Kafka, like event sourcing, is probably too much risk for most products. Most event-sourced CQRS implementations fail. I say choose Kafka somewhere in the middle, when scaling needs are somewhat predictable and product market fit is good, but try and use Confluent's hosted service or something similarly light on ops. Do non-critical things with it first and ramp up. Don't build big from the get go - that rarely ends well.
- deleted 6y ago[deleted]
- lmm 6y agoSo you set up your Postgres instance, great. Then what, you hope it doesn't break? Master-master Postgres is significantly fiddlier and less well-tested than Kafka. And what do you gain? Fiddlier APIs and a query language that makes it easy to deadlock yourself with no warning or write queries that will run for months. Just use Kafka unless you've got a good reason not to. It gets a lot more of the basics right.
- prabhatjha 6y agoThis post is misleading. It's not about Kafka. It's about need for a messaging system or not which is a completely different question.
- geodel 6y agoAs someone who is currently working at enterprise where a migration of Kafka from old to a newer version is taking place. IMO this whole Kafka thing seems quite unstable compared to RDBMSs where migrations are relatively painless on application code. Kafka is causing huge churn in applications codebases. I know that "best practices" experts will tell our company that we are doing it wrong. However to me Kafka seems a clever algorithms circlejerking and answer to operational complexity is either go to managed cloud or go to hell.
- chmod775 6y agoTo really appreciate the ludicrousness of what WeWork is doing, you have to take that example even further and assume that WeWork needs to track every single working adult on the globe clocking in and out. That's about 100k events per second. You can handle that using TimescaleDB running on a single Azure DS4 Standard node.[1] [1]: https://blog.timescale.com/blog/timescaledb-vs-6a696248104e/ https://blog.timescale.com/blog/timescaledb-vs-6a696248104e/ Edit: Fixed my maths and language, and linking to hard data instead.
- wefarrell 6y agoMaybe this already exists, but it would be nice if there was a library that provided a uniform streaming interface that allowed you to use Postgres as your stream processor (as the author proposes) and then as you scale seamlessly migrate to Kafka (or another stream processor).
- kumarvvr 6y agoThis could be achieved by good architecture and system design. You should have various aspects of your application configurable and independent enough to enable you to develop an alternative system and make a small downtime migration. Many of scaling issues I see in startups, and stories told by devs in startups, seem to be because corners are cut to make the MVP and push it to the market as quickly as possible, and then suffer scaling woes due to poor design and architecture choices.
- kumarvvr 6y agoAre VCs really so incompetent to understand when a company is being hyper dramatic when it talks about "Kafka for IoT", when it is essentially a real-estate leasing entity ? Or do they think buzzwords will increase ROI?
- flowerlad 6y agoFrom the article: "The overhead in setting up Kafka versus a very hefty Postgres is enormous, because distributed systems are hard...". If you need a message queue what is the alternative to Kafka? In my application I needed to update my Lucene search index when data (stored in flat files) is updated. I need a queue in between, so I evaluated available solutions: RabbitMQ and Kafka, and I picked Kafka. Another place I need a message queue is for notifications. When an item is updated lots of users may need to be notified and I didn't want this processing to happen synchronously so I used Kafka in between. My application is running in a Kubernetes cluster, and I have Kafka running in a container. Never had any problems. Based on my experience I don't hesitate to recommend Kafka even for simple applications. The only negative I noticed that resource requirements are high compared to everything else I am running.
- redis_mlc 6y agoA lot of companies can push LOAD DATA INFILE or bulk copy programs, which require no additional servers and can ingest a t 1 million rows per second, pretty far before going Kafka. Also, Kafka has the problem that to keep rings under control, you end up expiring (losing) data after so many days, and most companies have just one expert to keep it running. https://mariadb.com/kb/en/load-data-infile/ https://mariadb.com/kb/en/load-data-infile/ https://mariadb.com/kb/en/mysqlimport/ https://mariadb.com/kb/en/mysqlimport/
- vemv 6y agoThe article fixates excessively in throughput, as if was the only factor that mattered for choosing a data store. It isn't, since Kafka isn't necessarily about scaling, but also focuses on immutability, architectural flexibility etc, all of which a RDBMS might not offer.
- karmakaze 6y agoThis post conflates 'big data' with what Kafka offers. Yes it can scale up, but what it gives you at any scale is a reliable ordered log that supports multiple producers and consumers. If this is not what you need, then by all means don't use it or similarly complex software. I've used AWS SQS and Kinesis at small and large (not big data) scales and even then in retrospect believe Kafka would have served much better.
- villgax 6y agoWhat people need is proxies between touch points such that scaling or changes/migrations are not a pain later on
- macmar 6y agoYou don't use Kafka just to move a large amount of data between systems. You can also make use of Kafka to decouple systems, make integration easier, and let distributed computing technology widely used by big companies like Facebook, Uber, Tinder, Via Varejo (Brazil), Banco Itau (Brazil), Banco Santander (Brazil), be used to act in what she is a specialist. The big problem is that the learning curve of the entire ecosystem is high, many people do not have the minimum semantic requirements that a distributed computing strategy requires and do not even want to study and understand, and thus generate this discomfort, and some statements erroneous that it is a cannon to kill a fly. Right now, I am using Apache Kafka as an integration strategy between systems to integrate large factories spread out geographically, there was a high curve for everyone involved (infrastructure, Developers, and Architects) to understand the ecosystem as a whole (Kafka Connect, Kafka Broker, Zookeeper, Streams strategy, MirrorMaker 2) we built the entire integration by moving files in just 45 days due to an emergency imposed by government regulations. We have almost 35 thousand messages a day (far from being high volume) being transported between different locations, in a stable manner, and with the delivery time (milliseconds) necessary for the demand. And now when we need to integrate a new factory that was bought by the company, we can do this integration in a short time because the entire base (the core) of the integration has been consolidated. Everything is a matter of strategy and the correct use of the solution.
- pachico 6y agoAlthough I see the point behind, the reasoning of the article is not great. Firstly, you don't only build for your current requirements but having growth in mind. And the calculations behind are those that might make someone get fired. WeWork users don't use the entrance evenly along the entire day but in peaks. It's a silly thing, probably, but the entire article is based on that silly calculation too.
- free652 6y agoUh whoever wrote this piece should just stop writing article about tech, it's unreadable and technologically immature. I had to scroll and scroll and I lost what was the point of the article?