3 ms·

Do you have any advice for setting up a test Kafka environment? I'd love to be able setup a lightweight in-memory Kafka for unit-y tests without going through

by sa46 6y ago

Do you have any advice for setting up a test Kafka environment? I'd love to be able setup a lightweight in-memory Kafka for unit-y tests without going through the whole Docker compose rigamarole.

ndmeredian 6y ago
Depending on complexity and specialisation of your use that's what may work for unit testing: 1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour. 2) Create simplest in-memory dummy implementation for that interface, basically a mock. 3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent. 4) Make code to load mock in local testing env 5) Use real kafka when running it on a testing pipeline Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.
EdwardDiego 6y ago
If you're using Kafka Streams it provides a TopologyTestRunner.
gunnarmorling 6y ago
Check out the Kafka support in Testcontainers: https://www.testcontainers.org/modules/kafka/ https://www.testcontainers.org/modules/kafka/. It is container-based, but with a very simple-to-use API for spinning up Kafka in tests. If you truly want something embedded, Debezium's KafkaCluster could be interesting: https://github.com/debezium/debezium/blob/master/debezium-core/src/test/java/io/debezium/kafka/KafkaCluster.java https://github.com/debezium/debezium/blob/master/debezium-co.... It's used four our own tests of Debezium, but I'm aware of several external users. It spins up AK and ZK embedded. Disclaimer: I work on Debezium

by sa46 6y ago

Do you have any advice for setting up a test Kafka environment? I'd love to be able setup a lightweight in-memory Kafka for unit-y tests without going through the whole Docker compose rigamarole.

ndmeredian 6y ago

Depending on complexity and specialisation of your use that's what may work for unit testing: 1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour. 2) Create simplest in-memory dummy implementation for that interface, basically a mock. 3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent. 4) Make code to load mock in local testing env 5) Use real kafka when running it on a testing pipeline Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.

EdwardDiego 6y ago

If you're using Kafka Streams it provides a TopologyTestRunner.

gunnarmorling 6y ago

Check out the Kafka support in Testcontainers: https://www.testcontainers.org/modules/kafka/ https://www.testcontainers.org/modules/kafka/. It is container-based, but with a very simple-to-use API for spinning up Kafka in tests. If you truly want something embedded, Debezium's KafkaCluster could be interesting: https://github.com/debezium/debezium/blob/master/debezium-core/src/test/java/io/debezium/kafka/KafkaCluster.java https://github.com/debezium/debezium/blob/master/debezium-co.... It's used four our own tests of Debezium, but I'm aware of several external users. It spins up AK and ZK embedded. Disclaimer: I work on Debezium