3 ms·

Testability. Single biggest reason for us not going with the proprietary queue. We can create, reset, throw away queues as we want during testing, thousands tim

Testability. Single biggest reason for us not going with the proprietary queue. We can create, reset, throw away queues as we want during testing, thousands times a day. Even on a laptop.

sa46 6y ago
Do you have any advice for setting up a test Kafka environment? I'd love to be able setup a lightweight in-memory Kafka for unit-y tests without going through the whole Docker compose rigamarole.
- ndmeredian 6y ago
  Depending on complexity and specialisation of your use that's what may work for unit testing: 1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour. 2) Create simplest in-memory dummy implementation for that interface, basically a mock. 3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent. 4) Make code to load mock in local testing env 5) Use real kafka when running it on a testing pipeline Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.
- EdwardDiego 6y ago
  If you're using Kafka Streams it provides a TopologyTestRunner.
- gunnarmorling 6y ago
  Check out the Kafka support in Testcontainers: https://www.testcontainers.org/modules/kafka/ https://www.testcontainers.org/modules/kafka/. It is container-based, but with a very simple-to-use API for spinning up Kafka in tests. If you truly want something embedded, Debezium's KafkaCluster could be interesting: https://github.com/debezium/debezium/blob/master/debezium-core/src/test/java/io/debezium/kafka/KafkaCluster.java https://github.com/debezium/debezium/blob/master/debezium-co.... It's used four our own tests of Debezium, but I'm aware of several external users. It spins up AK and ZK embedded. Disclaimer: I work on Debezium

Testability. Single biggest reason for us not going with the proprietary queue. We can create, reset, throw away queues as we want during testing, thousands times a day. Even on a laptop.

sa46 6y ago

Do you have any advice for setting up a test Kafka environment? I'd love to be able setup a lightweight in-memory Kafka for unit-y tests without going through the whole Docker compose rigamarole.

ndmeredian 6y ago
Depending on complexity and specialisation of your use that's what may work for unit testing: 1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour. 2) Create simplest in-memory dummy implementation for that interface, basically a mock. 3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent. 4) Make code to load mock in local testing env 5) Use real kafka when running it on a testing pipeline Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.
EdwardDiego 6y ago
If you're using Kafka Streams it provides a TopologyTestRunner.
gunnarmorling 6y ago
Check out the Kafka support in Testcontainers: https://www.testcontainers.org/modules/kafka/ https://www.testcontainers.org/modules/kafka/. It is container-based, but with a very simple-to-use API for spinning up Kafka in tests. If you truly want something embedded, Debezium's KafkaCluster could be interesting: https://github.com/debezium/debezium/blob/master/debezium-core/src/test/java/io/debezium/kafka/KafkaCluster.java https://github.com/debezium/debezium/blob/master/debezium-co.... It's used four our own tests of Debezium, but I'm aware of several external users. It spins up AK and ZK embedded. Disclaimer: I work on Debezium