6 ms·

Recently I watched a 50-engineer startup allocate more than 50% of their engineering time for about two years to trying to cope with the consequences of using K

Recently I watched a 50-engineer startup allocate more than 50% of their engineering time for about two years to trying to cope with the consequences of using Kafka as their database, and eventually try to migrate off of it. The whole time I was wondering "but how could anyone have started down this path?!?" Apparently the primary reason they went out of business was sales-related, not purely technical, but if they hadn't used Kafka, they could have had 2x the feature velocity, or better yet 2x the runway, which might have let them survive and eventually thrive. Imagine, thinking you want a message bus as your primary database.

steve_adams_86 6y ago
I worked with a short-lived startup which made this exact mistake. I suggested persisting important events to a postgres db, but I was shot down over and over. They were positive it would be fine to hold everything in kafka. There was this notion that kafka functioned fine as a long term data store with the correct configuration (I'm not arguing that), but there was no solution to the lack of... Well, everything postgres could offer. As you mentioned, the lack of velocity really dragged them down. Customers were constantly upset that x broke or y wasn't delivered yet, and each time it was clear that kafka wasn't helping them meet those needs. Kafka is awesome for what it does, but there are a ton of people out there using it for weird stuff.
- konradb 6y ago
  Do you happen to know what sort of thinking leads a team down this path? It seems a fundamental mistake. Resume driven development?
  - afarrell 6y ago
    Insufficient asking "why?" 1. Why do we want to use Kafka? [insert thinking] So that we can do x yet avoid y. 2. Why is the customer asking for these features? [insert listening] So that they can do z and think of themselves as w. And then failure to remember and revisit that "why?".
  - derrekl 6y ago
    My hypothesis: it’s hard to have a descent grasp of technologies without having actually used it. Tie that with “let’s not include too many moving parts” and it’s easy to end up in a situation where the edict is “Kafka”. Let’s say you have never used RDBMS, only used rethinkdb and that turned out to be problematic for whatever reason, next project the founder hires you on the premise that the system you build needs to scale to billions of requests per minute ASAP (eventhough currently there is 0 traffic).
    - steve_adams_86 6y ago
      Even though currently there is zero traffic is exactly right. Haha. When this company finally did get customers, the thing they thought would help them manage thousands of high volume customers ended up making it so they could barely retain a few very low volume customers.
      - derrekl 6y ago
        Yea, I agree. Having the endgame solution in place at the beginning is often a mistake and can actually foil your ability to get to the end game.
        steve_adams_86 6y ago
        I'll admit, I only recognized that mistake because I've made it myself, over and over. It's hard to push code knowing it'll need improvements later, or knowing how scopes will change. I find myself repeating "perfect is the enemy of good" because I struggle to just let a solution be good enough. It's tough to be consulted and watch people go against your advice like that, though.
        fomine3 6y ago
        Sometimes I'm annoying is that It seems that some cloud services (I imagine Firebase) only provides endgame solution for datastore.
  - steve_adams_86 6y ago
    In this case the application was commissioned by the founder, and the people developing it were given errant criteria (and not given other valid criteria) which admittedly made Kafka sound like it would make sense. The developers who adopted the commissioned application were way too inexperienced to know one way or another, and by the time I was asked to help out, no one had really learned anything about Kafka. My suggestions to replace it with a basic queue were rejected and my suggestions to store events in a database for easier recollection were rejected. They were sure learning Kafka better would fix everything instead. Ironically there's not even that much to learn about it, in the scheme of things. It's like learning more about your car so you can make it fly one day. It's just not the right tool for the job.
  - JamesBarney 6y ago
    In addition to what others have answered, I think many devs consistently underestimate the amount of heavy lifting that relational databases do. And even worse a lot of this heavy lifting isn't obvious until you are fairly far into a project and suddenly realize that to solve a bug you need to implement transactions or referential integrity.
    - commandlinefan 6y ago
      Not just Kafka, either. Mongo, Cassandra, couchbase, Redis, SOLR and elastic search are all mistaken as replacements for an RDBMS.
      - 411111111111111 6y ago
        Cassandra can replace an rdbms in a lot of cases if eventual consistency works for your data. It's not a drop in replacement though, that's for sure.
        commandlinefan 6y ago
        Better than the other suggestions, but my experience is that you run into Cassandra's limitations really quick: you can't query on non-primary columns, and you can't join tables without pulling all the data down to the client and merge manually.
        411111111111111 6y ago
        > can't query on non-primary columns, That's incorrect unless I'm misunderstanding what you mean with primary columns.. It's just not as efficient. And missing joins are by design and one of the reasons cassandra is as fast as it is. And as I said before: it's not a drop in replacement. You need to architect your application around it's strength to leverage it's performance. It is however usable as a rdbms replacement if you know what you're doing and your data is fine with eventual consistency. And knowing what cassandra does with your data is important as well. It's actually a key-value store on steroids. Once you get that, it's limitations and strengths become obvious
  - capkutay 6y ago
    First - you decide to build your data platforms with apache kafka. makes sense as it's a very widely used, durable message bus that does the job well. Then...you decide to attend a kafka developer conference like Kafka Summit to sharpen your skills and learn best practices. At that conference, you see the CEO of Confluent give a keynote speech about how Kafka is a transaction store; you 'turn your database inside out', run SQL queries on it, even run your BI dashboards on top of it! They make it seem like this is a best practice that all the smartest engineers are doing, so you better do it as well. At that point you're investing in a path that will make Confluent millions in revenue trying to consult you on implementing their experimental, up-sell architecture on top of Kafka. Maybe their sales rep will suggest you to buy a $100k/month subscription to Confluent Cloud to really patch up any holes and make it easier to maintain. Moral of the story is doing your best to separate out the business and the tech side of open source. Behind many open source projects, there's a company trying to make money on an 'enterprise' version. Ethically, I prefer data companies like Snowflake that are at least transparent about trying to make money off their product. Rather than companies that use open source as a ladder to drive adoption, then try to bait-and-switch you into the same 7-figure deals with software that is more brittle and still requires you to develop your own solutions with it.
  - dwoot 6y ago
    I think this is worth mentioning again: https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
- eternalban 6y ago
  > this exact mistake https://kafka-summit.org/sessions/source-truth-new-york-times-stores-every-piece-content-ever-published-kafka/ https://kafka-summit.org/sessions/source-truth-new-york-time... Both your comment and the GP are using examples of failed startups as basis of your criticism. One mentions "50 engineers" and I'd bet not a single one had an "engineer" bone or training in their body. So I read that as 50 blog-driven programmers fail to execute a sophisticated system design they read about on some blog. Surprise? Using Kafka as the distributed log of a deconstructed database is technically an order of magnitude more demanding than wiring MVC widgets, or using the latest paint by numbers framework, etc., because that is in fact developing a custom distributed database. Technical leadership and experience are forgotten things but they do matter. Let's not blame the stack.
  - simonh 6y ago
    Theyre not criticising Kafka, they’re criticising misuse of it in use cases it’s not designed for.
  - steve_adams_86 6y ago
    I said Kafka is great at what it's for - I'm not blaming it at all. I actually enjoyed working with it a lot, despite that it couldn't help my client very much. Fine tuning it to work better for them was probably one of the more fun projects I've worked on in the last year. It still wasn't the right tool for the job they were trying to do, though. They could have replaced their brokers with simple worker queues and fed the raw data and results into a db. That was all they needed. The fear was that when they got MASSIVE, that architecture wouldn't scale for them. In reality it absolutely would, but like you said, the blogs told them otherwise. Kafka is fine, using it for the wrong thing isn't.
    - JamesBarney 6y ago
      If I had a dollar for every project that couldn't make a 1000 users happy because they were too worried about making a million happy.
    - eternalban 6y ago
- deleted 6y ago
  [deleted]
devin 6y ago
I worked on a project recently which had Kafka in it. Our first order of business was to completely remove it. I would bet good money that many projects on Kafka would benefit tremendously from simply removing it. It's wildly unnecessary in 7/10 situations I see it employed.
- qlm 6y ago
  What was it replaced with?
  - StreamBright 6y ago
    Not the guy who posted above but I think I have relevant experience. After going through different options for message queuing we have decided that using S3 and object will be our best option for many reasons. Some of those reasons: - durable - cheap - scalable - we do not need to read messages in a strict order - using S3's API is super easy - no operations overhead - no compute resources (for the queue part) After doing it some time I think that I was drinking too much cool-aid with streaming services like Kinesis and Kafka and failed to see how easy it is to implement a simple analytics even service using purely S3.
joss82 6y ago
I worked for a company that made the same mistake but with AWS Kinesis instead of Kafka. So, much more costly mistake.
StreamBright 6y ago
Similar story here. One startup's engineer explained to me how they use Kafka (0.8ish) as they transactional database for financial transactions. When I explained to him that Kafka is(was) not transactional he did not believe me.
vishnugupta 6y ago
I’m curious, what makes Kafka to be picked in the first place. Perhaps a vast majority of developers are growing up using Kafka for everything for their learning projects as it makes it convenient for rapid development? Maybe they learn about it through blogs, video tutorials and so on, at the cost of other technologies? For instance, my nephew was heaping praises on Firebase and would use it for all his projects. It took me a while to understand that the convenience of using it masked some of the issues that would only get exposed at scale. For whatever reason a vast number of developers are entering the workforce believing in Kafka’s capability to solve all problems. And going against that majority will be very tiring and even costly.
jgraettinger1 6y ago
> Imagine, thinking you want a message bus as your primary database. I've built (and sold) an entire company around this architecture, and am working on the next. It can be incredibly powerful, and is what you _have_ to do if you want to mitigate the significant downsides of a Lambda architecture. We successfully used it to power graph building, reporting [1], and data exports, all in real-time and at scale. But, we didn't use Kafka for two key reasons (as evaluated back in 2014): * We wanted vanilla files in cloud storage to be the source-of-truth for historical log segments. This was really important, because it makes the data _accessible_. Applications that don't require streaming (Spark, Snowflake, etc) can trivially consume the log as a hierarchy of JSON/protobuf/etc files in cloud storage. Plus, who the heck wants to manage disk allocations these days? * We wanted read/write IO isolation of logs. This architecture is only useful if new applications can fearlessly back-fill at scale without putting your incoming writes at risk. Cloud storage is a great fit here: brokers keep an index and re-direct streaming readers to the next chunks of data, without moving any bytes themselves. Cloud read IOPs also scale elastically with stored data. Operational concerns were certainly a consideration: we were building in Go, targeting CoreOS and then Kubernetes, had no interest in Zookeeper, wanted broker instances to be ephemeral & disposable (cattle, not pets), etc, but those two requirements are pretty tough to bolt on after the fact. The result, Gazette, is now OSS (https://gazette.dev https://gazette.dev) and has blossomed to provide a bunch of other benefits, but serves those core needs really well. [1] https://github.com/liveramp/factable https://github.com/liveramp/factable

steve_adams_86 6y ago

I worked with a short-lived startup which made this exact mistake. I suggested persisting important events to a postgres db, but I was shot down over and over. They were positive it would be fine to hold everything in kafka. There was this notion that kafka functioned fine as a long term data store with the correct configuration (I'm not arguing that), but there was no solution to the lack of... Well, everything postgres could offer. As you mentioned, the lack of velocity really dragged them down. Customers were constantly upset that x broke or y wasn't delivered yet, and each time it was clear that kafka wasn't helping them meet those needs. Kafka is awesome for what it does, but there are a ton of people out there using it for weird stuff.

konradb 6y ago
Do you happen to know what sort of thinking leads a team down this path? It seems a fundamental mistake. Resume driven development?
- afarrell 6y ago
  Insufficient asking "why?" 1. Why do we want to use Kafka? [insert thinking] So that we can do x yet avoid y. 2. Why is the customer asking for these features? [insert listening] So that they can do z and think of themselves as w. And then failure to remember and revisit that "why?".
- derrekl 6y ago
  My hypothesis: it’s hard to have a descent grasp of technologies without having actually used it. Tie that with “let’s not include too many moving parts” and it’s easy to end up in a situation where the edict is “Kafka”. Let’s say you have never used RDBMS, only used rethinkdb and that turned out to be problematic for whatever reason, next project the founder hires you on the premise that the system you build needs to scale to billions of requests per minute ASAP (eventhough currently there is 0 traffic).
  - steve_adams_86 6y ago
    Even though currently there is zero traffic is exactly right. Haha. When this company finally did get customers, the thing they thought would help them manage thousands of high volume customers ended up making it so they could barely retain a few very low volume customers.
    - derrekl 6y ago
      Yea, I agree. Having the endgame solution in place at the beginning is often a mistake and can actually foil your ability to get to the end game.
      - steve_adams_86 6y ago
        I'll admit, I only recognized that mistake because I've made it myself, over and over. It's hard to push code knowing it'll need improvements later, or knowing how scopes will change. I find myself repeating "perfect is the enemy of good" because I struggle to just let a solution be good enough. It's tough to be consulted and watch people go against your advice like that, though.
      - fomine3 6y ago
        Sometimes I'm annoying is that It seems that some cloud services (I imagine Firebase) only provides endgame solution for datastore.
- steve_adams_86 6y ago
  In this case the application was commissioned by the founder, and the people developing it were given errant criteria (and not given other valid criteria) which admittedly made Kafka sound like it would make sense. The developers who adopted the commissioned application were way too inexperienced to know one way or another, and by the time I was asked to help out, no one had really learned anything about Kafka. My suggestions to replace it with a basic queue were rejected and my suggestions to store events in a database for easier recollection were rejected. They were sure learning Kafka better would fix everything instead. Ironically there's not even that much to learn about it, in the scheme of things. It's like learning more about your car so you can make it fly one day. It's just not the right tool for the job.
- JamesBarney 6y ago
  In addition to what others have answered, I think many devs consistently underestimate the amount of heavy lifting that relational databases do. And even worse a lot of this heavy lifting isn't obvious until you are fairly far into a project and suddenly realize that to solve a bug you need to implement transactions or referential integrity.
  - commandlinefan 6y ago
    Not just Kafka, either. Mongo, Cassandra, couchbase, Redis, SOLR and elastic search are all mistaken as replacements for an RDBMS.
    - 411111111111111 6y ago
      Cassandra can replace an rdbms in a lot of cases if eventual consistency works for your data. It's not a drop in replacement though, that's for sure.
      - commandlinefan 6y ago
        Better than the other suggestions, but my experience is that you run into Cassandra's limitations really quick: you can't query on non-primary columns, and you can't join tables without pulling all the data down to the client and merge manually.
        411111111111111 6y ago
        > can't query on non-primary columns, That's incorrect unless I'm misunderstanding what you mean with primary columns.. It's just not as efficient. And missing joins are by design and one of the reasons cassandra is as fast as it is. And as I said before: it's not a drop in replacement. You need to architect your application around it's strength to leverage it's performance. It is however usable as a rdbms replacement if you know what you're doing and your data is fine with eventual consistency. And knowing what cassandra does with your data is important as well. It's actually a key-value store on steroids. Once you get that, it's limitations and strengths become obvious
- capkutay 6y ago
  First - you decide to build your data platforms with apache kafka. makes sense as it's a very widely used, durable message bus that does the job well. Then...you decide to attend a kafka developer conference like Kafka Summit to sharpen your skills and learn best practices. At that conference, you see the CEO of Confluent give a keynote speech about how Kafka is a transaction store; you 'turn your database inside out', run SQL queries on it, even run your BI dashboards on top of it! They make it seem like this is a best practice that all the smartest engineers are doing, so you better do it as well. At that point you're investing in a path that will make Confluent millions in revenue trying to consult you on implementing their experimental, up-sell architecture on top of Kafka. Maybe their sales rep will suggest you to buy a $100k/month subscription to Confluent Cloud to really patch up any holes and make it easier to maintain. Moral of the story is doing your best to separate out the business and the tech side of open source. Behind many open source projects, there's a company trying to make money on an 'enterprise' version. Ethically, I prefer data companies like Snowflake that are at least transparent about trying to make money off their product. Rather than companies that use open source as a ladder to drive adoption, then try to bait-and-switch you into the same 7-figure deals with software that is more brittle and still requires you to develop your own solutions with it.
- dwoot 6y ago
  I think this is worth mentioning again: https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
eternalban 6y ago
> this exact mistake https://kafka-summit.org/sessions/source-truth-new-york-times-stores-every-piece-content-ever-published-kafka/ https://kafka-summit.org/sessions/source-truth-new-york-time... Both your comment and the GP are using examples of failed startups as basis of your criticism. One mentions "50 engineers" and I'd bet not a single one had an "engineer" bone or training in their body. So I read that as 50 blog-driven programmers fail to execute a sophisticated system design they read about on some blog. Surprise? Using Kafka as the distributed log of a deconstructed database is technically an order of magnitude more demanding than wiring MVC widgets, or using the latest paint by numbers framework, etc., because that is in fact developing a custom distributed database. Technical leadership and experience are forgotten things but they do matter. Let's not blame the stack.
- simonh 6y ago
  Theyre not criticising Kafka, they’re criticising misuse of it in use cases it’s not designed for.
- steve_adams_86 6y ago
  I said Kafka is great at what it's for - I'm not blaming it at all. I actually enjoyed working with it a lot, despite that it couldn't help my client very much. Fine tuning it to work better for them was probably one of the more fun projects I've worked on in the last year. It still wasn't the right tool for the job they were trying to do, though. They could have replaced their brokers with simple worker queues and fed the raw data and results into a db. That was all they needed. The fear was that when they got MASSIVE, that architecture wouldn't scale for them. In reality it absolutely would, but like you said, the blogs told them otherwise. Kafka is fine, using it for the wrong thing isn't.
  - JamesBarney 6y ago
    If I had a dollar for every project that couldn't make a 1000 users happy because they were too worried about making a million happy.
  - eternalban 6y ago
deleted 6y ago
[deleted]

devin 6y ago

I worked on a project recently which had Kafka in it. Our first order of business was to completely remove it. I would bet good money that many projects on Kafka would benefit tremendously from simply removing it. It's wildly unnecessary in 7/10 situations I see it employed.

qlm 6y ago
What was it replaced with?
- StreamBright 6y ago
  Not the guy who posted above but I think I have relevant experience. After going through different options for message queuing we have decided that using S3 and object will be our best option for many reasons. Some of those reasons: - durable - cheap - scalable - we do not need to read messages in a strict order - using S3's API is super easy - no operations overhead - no compute resources (for the queue part) After doing it some time I think that I was drinking too much cool-aid with streaming services like Kinesis and Kafka and failed to see how easy it is to implement a simple analytics even service using purely S3.

joss82 6y ago

I worked for a company that made the same mistake but with AWS Kinesis instead of Kafka. So, much more costly mistake.

StreamBright 6y ago

Similar story here. One startup's engineer explained to me how they use Kafka (0.8ish) as they transactional database for financial transactions. When I explained to him that Kafka is(was) not transactional he did not believe me.

vishnugupta 6y ago

I’m curious, what makes Kafka to be picked in the first place. Perhaps a vast majority of developers are growing up using Kafka for everything for their learning projects as it makes it convenient for rapid development? Maybe they learn about it through blogs, video tutorials and so on, at the cost of other technologies? For instance, my nephew was heaping praises on Firebase and would use it for all his projects. It took me a while to understand that the convenience of using it masked some of the issues that would only get exposed at scale. For whatever reason a vast number of developers are entering the workforce believing in Kafka’s capability to solve all problems. And going against that majority will be very tiring and even costly.

jgraettinger1 6y ago

> Imagine, thinking you want a message bus as your primary database. I've built (and sold) an entire company around this architecture, and am working on the next. It can be incredibly powerful, and is what you _have_ to do if you want to mitigate the significant downsides of a Lambda architecture. We successfully used it to power graph building, reporting [1], and data exports, all in real-time and at scale. But, we didn't use Kafka for two key reasons (as evaluated back in 2014): * We wanted vanilla files in cloud storage to be the source-of-truth for historical log segments. This was really important, because it makes the data _accessible_. Applications that don't require streaming (Spark, Snowflake, etc) can trivially consume the log as a hierarchy of JSON/protobuf/etc files in cloud storage. Plus, who the heck wants to manage disk allocations these days? * We wanted read/write IO isolation of logs. This architecture is only useful if new applications can fearlessly back-fill at scale without putting your incoming writes at risk. Cloud storage is a great fit here: brokers keep an index and re-direct streaming readers to the next chunks of data, without moving any bytes themselves. Cloud read IOPs also scale elastically with stored data. Operational concerns were certainly a consideration: we were building in Go, targeting CoreOS and then Kubernetes, had no interest in Zookeeper, wanted broker instances to be ephemeral & disposable (cattle, not pets), etc, but those two requirements are pretty tough to bolt on after the fact. The result, Gazette, is now OSS (https://gazette.dev https://gazette.dev) and has blossomed to provide a bunch of other benefits, but serves those core needs really well. [1] https://github.com/liveramp/factable https://github.com/liveramp/factable