11 ms·
Why we wrote our Kafka Client in Pony
- hackmanytrades 9y agoHi, I'm the author of the blog post and the primary author of Pony Kafka. I'll be around to answer questions.
- adwhit 9y agoFirst - I spent a good amount of time with Pony attempting to make something similar to Wallaroo (on a much smaller scale), and it seems I hit the same problems as you re:FFI. Does it worry you that creating idiomatic C wrappers in Pony feels 'wrong'? Are there ways to make it better? After all, young languages especially are very dependent on wrapping C libs for basic functionality. Second - from what I can tell, Wallaroo Labs are now one of the main contributors to Ponylang. How are your experiences from writing Wallaroo affecting language design and direction?
- hackmanytrades 9y agoGreat questions! Re: FFI. In general, using the C FFI and creating C wrappers is not a major issue (aside from them possibly not being idiomatic) and, as you mentioned, required for a language as young as Pony. In fact, Pony Kafka internally relies on a number of C libraries via FFI. However, for the Pony Kafka use case, performance was a key driver. This meant that the risk of contention between the librdkafka thread pool and the Pony thread pool was one that had to be taken into account for long term performance goals. Same regarding the polling nature of getting data from librdkafka. Neither of these would have been a major issue had performance not been a high priority for Wallaroo. In regards to the idiomatic C wrappers, I think it's possible to wrap C libraries in an idiomatic way. It's not easy though because it's hard to figure out the right abstractions in Pony and map them to the C functionality. I don't think that's a problem though because not everything has to be idiomatic from day one. Over time, the C wrappers can be improved to be more idiomatic and/or phased out via native implementations. Re: Ponylang direction. My personal experience on this is that the Pony community has been very receptive to our ideas. I've mainly focused one the Pony runtime side of things and less on the compiler/language design side of things though.
- cs702 9y ago"Why we wrote our Kafka Client in Pony (wallaroolabs.com)" I love the fact that the words "Kafka," "Pony," and "Wallaroo" are all together in one sentence, and not as part of an elaborate joke. I love even more the fact that Serious Business Executives will have to use these words to discuss useful technology. Awesome names.
- hackmanytrades 9y agoJust one of the many perks of working here at Wallaroo Labs. 8*P
- AccountCreated 9y ago> I love the fact that the words " "Kafka," "Pony," and "Wallaroo" are all together in one sentence, and not as part of an elaborate joke Why are these two things mutually exclusive?
- DerBesserWisser 9y agoTL;DR Because it's more fun to learn a new programming language than deliver features to users. We're VC financed so no need to get money, also spend it before people talk about profitability, then the good times are over (see Etsy). We also can add Pony to our CV and move on in one year to the next company where we will introduce the next big thing to add to our CVs. Plus 10% more salary! Kaching! #LivingTheLife
- CapacitorSet 9y agoThat's not the impression I got from the article, which cites performance, API architecture and integration with existing codebases. Is the "dr" in "tl;dr" literal?
- sidlls 9y agoYour comment and the parent aren't mutually exclusive. It's fairly easy for coders to provide vast amounts of technical justification for resume driven development.
- dmix 9y agoIs a new immature language like Pony really that desirable in the marketplace? I highly doubt 'resume development' was ever a primary motivation. If it was anything beyond the strategic interests of the company it's that developers love a) clean code and possibilities of a fresh new project and b) playing with new modern toys.
- rgrieselhuber 9y agoI think there is a hipster element to language selection in situations like the parent refer to. I've certainly seen this at play.
- sidlls 9y agoI don't know about "hipster" unless you're using it as a synecdoche for "trend following." There is definitely a predilection in certain parts of the coder community to prefer newness and difference over tried and true. There isn't anything wrong with that necessarily: it's part of how progress is made. However I think it's often taken to extremes in the coder community.
- lmm 9y agoKafka is written in Scala, though it exposes a Java interface for ease of use.
- hackmanytrades 9y agoThanks! Blog post corrected.
- royjacobs 9y agoWhat was the reason for not contributing a "reactive" API to the existing C client? Especially because now you have a maintenance burden on your own client which, ironically, performs quite a lot slower than the C client although that was the major concern for not using it in the first place!
- vbernat 9y agoThe existing API already features ability to integrate into any event loop. See rd_kafka_queue_io_event_enable() to provide a file descriptor to be notified of incoming data.
- hackmanytrades 9y agoThat's a great question. Yes, Pony Kafka is currently slower than the C client. But it is also almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains. There is also the secondary concern regarding the thread pools internal to Pony and librdkafka. We've seen first hand how CPU cache invalidation can impact performance so we are very aware of the potential negatives if the Pony and librdkafka threads ever end up fighting with each other over the same CPU resources.
- royjacobs 9y agoSorry, that was my question-- was there a way to avoid using librdkafka's threadpool and essentially use it as a 'dumb' client, moving all the async stuff to your Pony actor layer?
- hackmanytrades 9y agoFrom what I understand of librdkafka, there's no easy way to disable the internal thread pool it uses. I'd imagine that the internal thread pool for sending/receiving data from Kafka is as core to librdkafka as the internal thread pool for running actors is to Pony and trying to remove or disable either of them would be a large undertaking.
- 9y ago
- manigandham 9y ago> Pony Kafka sends data to Kafka about 5% - 10% slower than librdkafka but reads data from Kafka about 75% slower than librdkafka. What? So what's the point? Wouldn't it better to just contribute and optimize the existing clients then? The Kafka/confluent team specifically chose to implement everything in librdkafka (because kafka is client-side logic heavy) and then make thin wrappers for every language so performance, bugs and stability can all be worked on in a single place.
- dmix 9y agoThat was for what I'd imagine is v0.1 quality code, they said there were plenty of optimizations available to make it at least parity with the C client. Secondly, the issue was the system performance between Wallaro and the Kafka client, not necessarily the raw performance of the library.
- hackmanytrades 9y agoYes, that's correct on both counts. Pony Kafka is almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains. And yes, we're concerned about the potential thread pool contention between Pony and librdkafka.
- manigandham 9y agoCopying data in memory between libraries is slower than actually getting the data across the network from the Kafka cluster itself? Seems like a strange bottleneck if so.
- fulafel 9y agoIf they worked on the C client, they then have to work in C, which is bad. (This is one of the things covered in their Tradeoffs section, along with the problems with serialization and the C client's thread pool).
- StreamBright 9y agoI really like why we wrote x in y sort of blog posts. It is almost always like this: because we could or because for our use case it works.
- sidlls 9y agoIt's actually almost always a post-hoc justification for scratching an itch.
- StreamBright 9y agoExactly. Pony looks interesting though. I am wondering how is it comparing to Erlang. Seems like they are trying to address the same problem (actor model with memory safety) with different approaches.
- princess-aslaug 9y agoEhm, pick a niche programming language, build your client for a complex system. All cool, but it's not the straight line to solve problems in a company in the simplest and most effective way. Good thing if you have some spare R&D time maybe but hardly a sane approach for most.
- fnord77 9y agoThe investors must be thrilled.
- ramchip 9y agoOut of curiosity, what pushed your team towards Pony rather than Erlang? It seems your team (or at least part of it) has experience with both languages which is interesting.
- spooneybarger 9y agoShort answer is in another thread here. After discussing our performance goals with folks we knew at Basho, they expressed a lot of skepticism that we could meet them with Erlang. How to plug-in potentially heavy user computations in non-Beam based languages is an incredibly tricky problem as well. If you'd be interested in chatting more, I'm happy to that. See: https://news.ycombinator.com/item?id=16266220 https://news.ycombinator.com/item?id=16266220 for more information.
- mac01021 9y agoAs someone who has worked on building a custom consumer API for kafka (in a distant past, before the 9.0 Java consumer came out) I am curious about how much engineering effort this required. I can see from github that the project is about 16k lines of code. I wonder how many developers worked on it, how much of each of their time it required, how many false starts in the architecture of the library had to be abandoned...
- hackmanytrades 9y ago99% of Pony Kafka is written by me (for better or worse). I've been working on it since about May 2017 off and on with it being my primary focus for the majority of that time. However, due to working arrangements and other commitments, I've only spent about 12 or so weeks of time on it (where 1 week is equal to 5 days and 1 day is equal to 8 hours). There have been a few iterations on the abstractions and API of the library but the majority of the architecture has been the same from the initial design sketch. I started by envisioning the features the library needed for end users and also internally in order to fully take advantage of Pony's actor concurrency model. From there I worked out the data and functionality ownership of the various bits (i.e. which actor does what and why). Lastly, I ran it by a couple of folks here at Wallaroo Labs to make sure I wasn't making any obvious mistakes. The biggest change so far has been caused by building in the leader failover handling in relation to the data/responsibility ownership transferring from one actor to another. That's not entirely completed yet but it has mostly been an internal change. The end user API has also changed, but that has mostly been about fixing abstractions and/or data ownership issues. I'm sure there will be additional changes as I have time to go through to fix abstractions and add in other features. Dynamic configuration changes, exactly once semantics, and group consumer functionality are all likely to impact the end user API along with requiring internal changes.
- krylon 9y agoMan, I really hope Pony grows a decent library ecosystem fast! The language itself is sooo nice, the type system is pure bliss (at least this side of Haskell), and I even managed to get beyond the first peak of "I have no clue what is going on here" to using the type system sensibly. I do not care about scalability so much (I do not complain, either), but the language and especially the type system was an eye-opener for me. Having a guarantee - proof! - that entire classes of bugs that haunt software written in C, C++, Java will be impossible to smuggle past the compiler sounds like a realistic approximation to the four-dimensional compiler I once envisioned, that would retroactively turn any and all runtime errors into compile time errors (without creating a paradox, of course!).
- reilly3000 9y ago4D compiler? That sounds amazing. I imagine that could be extended with a Kafka stream of errors from staging and production. I guess that would require a ‘system aware’ compiler.
- littlestymaar 9y agoPony is such a cool project, it's what Go could have been if it wasn't stuck in the middle of the eighties : it has a super expressive type-system, and a bit like in Rust, it gives you the data-race freedom. I really wish it gains traction.
- Avshalom 9y ago(this has nthing to do with wallaroo or kafka but:) go isn't stuck in the 80s it just stuck on Rob Pike's belief that him personally being lazy and jury rigging in general is "simple" and a virtue.
- acdha 9y agoI don’t think that making criticisms personal like that accomplished anything useful. The fact that a large number of other people agree with his decisions suggests that it’s not “being lazy” and more balancing different goals. It would be far more productive to understand those pressures rather than simply assuming the worst.
- Avshalom 9y agoAny criticism of any project Rob Pike has been attached to or Pike himself has over the last 40 years accomplished nothing no matter if they are technical or personal.
- lobster_johnson 9y agoI'm not overly familiar with Pony, but I'm curious, and the code looks nice and clean. One oddity though; so many of the identifiers have "Kafka" in them. Does Pony not have module namespacing?
- spooneybarger 9y agoIt does. But the point you raise... "Should this be HTTPLogger or Logger given that it is in the HTTP package" and variations thereof is something that has been a point of contention at almost every job I've been at. by default with Pony if you use a package, you'll have the classes imported directory into your namespace so... HTTPLogger is more clear in that case, but you could use a qualified import and then have something like http.Logger. It's a matter of preference.
- lobster_johnson 9y agoI understand, but the sheer amount of duplication is rather overwhelming. Also, a lot of it seems like implementation details related to the API/protocol and so on that don't need this kind of naming uniqueness. Go solves this by never dumping namespaces into another namespace: You have http.Request, and that's it, which is both unambiguous and self-explanatory. Name clashes can occur (e.g. packages have the same name, or a local variable has the same name), but that's rare.
- rurban 9y agoDipon, why not rewriting Kafka itself in Pony? The biggest problem is the synchronous API, and as pony service it would be much better, being controlled by async actors, without polling. Scala is very similar to pony.