3 ms·

If you really want to support Kafka, I think there are options. Besides the obvious option of forking the .NET client to add these missing features, you could a

If you really want to support Kafka, I think there are options. Besides the obvious option of forking the .NET client to add these missing features, you could also write some kind of "plug-in" in Java or Go or some other language with a fully featured client library, and communicate with it over some type of local RPC from your main application to query against Kafka. Whether it is technically feasible or not, it may not make business sense for you, which is fine, just interesting to think about the options. I'm slightly surprised that the .NET client isn't fully featured, but I've never tried to use Kafka from a .NET language before.

grujicd 5y ago
I suppose .Net client has everything that typical Kafka app needs. Just not everything that management app would need. Btw. that's also the problem with all other queuing systems we support. Their APIs are made to support Send/Receive scenarios, what these systems are built for in the first place. They rarely support "delete message in the middle of the queue", and never "replace message" or "insert message at arbitrary position". I guess API authors are worried that such functionality would lead to apps mistreating queues as a kind database, which is not intended use case. However that sort of functionality is often needed when you're troubleshooting and fixing a problem. It's not like everything always just works. In reality things go wrong, message format changes, invalid data goes into queue, etc. And then we have to abuse existing API and/or give only subset of useful features to users. About combining Java or something else - that might be possible from technical point of view, but remember this is a desktop app which ships as an installer. Not an environment you'd set up on the server. We would have to include another runtime and make sure it works correctly in different scenarios on end-user machines.
- mjul 5y ago
  It’s been quite a number of years ago, so I don’t know the current state, but back then we used IKVM to run some finance Java libs in .NET applications. Worked like a charm. Basically, it is a JVM implemented in .NET so you can easily interop to the missing management APIs from the Java client and there is no extra runtime install to worry about or calling out of process. Might be worth a try.

grujicd 5y ago

I suppose .Net client has everything that typical Kafka app needs. Just not everything that management app would need. Btw. that's also the problem with all other queuing systems we support. Their APIs are made to support Send/Receive scenarios, what these systems are built for in the first place. They rarely support "delete message in the middle of the queue", and never "replace message" or "insert message at arbitrary position". I guess API authors are worried that such functionality would lead to apps mistreating queues as a kind database, which is not intended use case. However that sort of functionality is often needed when you're troubleshooting and fixing a problem. It's not like everything always just works. In reality things go wrong, message format changes, invalid data goes into queue, etc. And then we have to abuse existing API and/or give only subset of useful features to users. About combining Java or something else - that might be possible from technical point of view, but remember this is a desktop app which ships as an installer. Not an environment you'd set up on the server. We would have to include another runtime and make sure it works correctly in different scenarios on end-user machines.

mjul 5y ago
It’s been quite a number of years ago, so I don’t know the current state, but back then we used IKVM to run some finance Java libs in .NET applications. Worked like a charm. Basically, it is a JVM implemented in .NET so you can easily interop to the missing management APIs from the Java client and there is no extra runtime install to worry about or calling out of process. Might be worth a try.