14 ms·
Lessons learned porting 50k loc from Java to Go
- indogooner 8y agoThe article is more about the process and not the motivation for this port. What problem with Java/JVM caused the organization to commission this expensive porting exercise? And what are the benefits they have achieved after the port.
- cerberusss 8y agoAuthor states: "I was contracted to port a large Java code base to Go". That makes me think his focus is on earning the money, rather than questioning the motives of his client :)
- Svip 8y agoThe fact that he tried to port a lot of the Java code directly to Go, where Go had no suitable comparable feature. With generics, he mentions there are two different approaches in Go, but then only mention the one he calls the least preferred one, i.e. using interface{}. But he never talks about motivation of the original code. Like why it was generic in Java. Sometimes generic code in other languages can be ported to Go to two or three functions, because the general use of the generic code wasn't as generic as the designer may have envisioned, or it just became a nail to their hammer. Like was the inheritance really important, or could it be implemented another way in Go? But I guess if you're not questioning your client's motives, you may not be fully questioning how something ought to work, and instead just ensure it works as it does right now. It's possible they just needed a Go-library for their client code, and then the client code can be tweaked later on to be more idiomatic Go.
- kjksf 8y agoIf I was writing this code from scratch, I would make a few different decisions. That being said, from a business point of few, making Go code base be as close as possible to Java version makes a lot of sense. Java client is written the way it is. It already shipped and is used by people. Breaking its API was out of the question. RavenDB is 10 years old and will most likely be here for the next 10 years. In those future years, both Java client and Go client will have to be evolved to provide access to new capabilities of the server. The closer the two code bases are, the less effort it takes to maintain those code bases.
- kjksf 8y agoRavenDB is a database server, like PostgreSQL or MongoDB. It requires database drivers (client libraries) for as many languages as possible. More client libraries, more programmers can use RavenDB, more licenses for RavenDB sold. They already have C#/Java/Python/Node.js libraries. I ported Java client library to Go, so that people who program in Go can access RavenDB database.
- aidos 8y agoBonuses questions: do they other libraries stem from the java root client too or did they evolve separately (even out of house)? Why did you choose Java to port from? Are there interesting lessons in the commonality / differences between the language implementations?
- kjksf 8y agoRavenDB is written in C#. At first, it was Windows only but now is cross-platform database engine (Windows / Linux / Mac OS). As a result, the original and most featureful client is for C# / .NET. Java client is a port of C# client, done in house by Hibernating Rhinos (the creators of RavenDB). As far as I can tell, other clients (Python / Node.js and the Go client that I wrote) were contracted to outside people. The company suggested starting from Java code base. It makes sense because C# client heavily uses LINQ, which is unique to C# (neither Java nor Go has LINQ-like capabilities). I didn't dig much into non-Java clients so can't speak much to that. Overall, I was surprised how similar I was able to make Go code to Java code. Changing from exceptions to errors was pervasive but a simple, mechanical transformation. Porting functions using generics was the biggest hurdle. Porting functions that use overloading was easy but annoying. That being said, a Java code base that heavily used virtual functions and deep inheritance hierarchy would be more challenging to port to Go. Lucky for me, this code wasn't.
- pjmlp 8y ago> neither Java nor Go has LINQ-like capabilities). Not true since Java 8, with the introduction of streams and functional interfaces, which keep being improved with each release. What Java doesn't have are expression trees, which are convenient but not a requirement for LINQ like features.
- aidos 8y agoIt’s a client library by the looks. So presumably they have a Go codebase building up and they needed to be able to access RavenDB from there. Caveat, I haven’t finished the article and I’m not familiar with either the RavenDB or Go ecosystems.
- deleted 8y ago[deleted]
- severino 8y agoWith "porting" I guess he doesn't mean replacing one codebase with another one, but creating another client in a different language. In fact, he states The objective of the port was to keep it as close as possible to Java code base, as it needs to be kept in sync with Java changes in the future so it doesn't look like they are ditching their Java client.
- coldtea 8y ago>What problem with Java/JVM caused the organization to commission this expensive porting exercise? There needn't be any "problem with Java". This is a client library. You want to have it in as many languages as possible.
- lukeh 8y agoAlso, null is not the empty string...
- irth 8y agoDepending on how it's used in the Java code, it might be fine to replace it with an empty string.
- coldtea 8y agoHe knows it, that's why he mentions lack of null (nil) in Strings in Go as an issue when porting the code. But depending on how the Java client uses null, "" can do just as well. It's not like you have many other options (except to add your own composite struct on top of String, or to use a guard value that's still a string).
- fileeditview 8y agoIn most cases this would probably be bad but he could also use a pointer to a string.
- IloveHN84 8y agoI hate that everything has 5o be checked against null in Java. It sucks a lot
- hamdouni 8y agoThanks for sharing. I understood the contract was about replicating as much as possible the same structure or workflow as the original Java code but if you had to develop the same client in Go from scratch, where would be the main differences in your approach and how the Go code would benefit ?
- kjksf 8y agoI would split it into multiple packages. I would survey existing libraries for other JSON document databases (MongoDB, Google's Firestore etc.) and steal all their best ideas. But honestly, good API design is something that needs a lot of time to reach the polished state. Whatever I can come with today, I'm sure a year from now I would find ways to improve that. That has been my experience with much smaller libraries I wrote. The more time you get to spend thinking about the problem, the better solution you can come up with.
- raphael_kimmig 8y agoIt's interesting that the resulting go code has 43k lines of code, while the python client for raven only has 6k lines. I don't know whether they have equivalent feature sets - but I kind of wonder how it would have turned out if the go port would have been based on the python version.
- dxxvi 8y agoIf I'm paid to port it from Java to Scala, I'll write a blog like this after I'm done (maybe I'll never be done :-) ).
- ivan_gammel 8y agoIt is very interesting to see in such projects how they expose the weakness of certain Java idioms. The mentioned JavaBeans getters and setters are obsolete pattern, for which in most cases there’s no good reason to keep it in the code. Java ecosystem is probably the richest one on the design patterns, some of which receive „anti“ prefix over the years (e.g. self-contained singleton). At the same time new languages explore and verify the ideas and coding styles that might be worth to borrow and make mainstream. Porting exercise can be such source of new best practices pushing us to rethink what is worth keeping and what is pointless overengineering.
- deleted 8y ago[deleted]
- jaabe 8y agoI was struck by the lack of inheritance in go, that to me is brilliant. I’ve worked with C# for a decade, and I’ve yet to see a use of inheritance that wouldn’t have been easier to maintain in the long run without using inheritance. We’ve limited our own useage to override methods in the standard library, but even then it’s often used to implement things that are really just terrible practices. Like adding search functions for AD extension fields or increasing the timeout in one of the older web clients.
- codenesium 8y agoI'm also a c# dev but with less experience. Could you elaborate? Does the mean you copy paste shared code to all of the derived classes? I think I know why you say it's a maintenance nightmare because you start off with a method or properties that make sense in all of the derived classes but overtime the classes become less cohearant and start to develope warts and it would have been easier to modify the duplicated code.
- scanr 8y agoIt’s more that there are other ways to reuse code. Search for “composition over inheritance” in your favourite search engine for more information.
- MrBuddyCasino 8y agoCan anyone explain what’s happening here? I can’t make sense of the syntax: func (q Query) GroupBy(field string) Query {...}
- masklinn 8y agoThat's how you define a method in Go. The local to the left of the function name is the receiver (what `this` would be in Java). Note that there are a few odd bits to Go methods: the receiver can be a value or a pointer, and if the receiver is a pointer it might be null, because method calls on concrete types are statically dispatched so type S struct{} func (self *S) Foo() { fmt.Printf("%v\n", self) } func main() { var s *S s.Foo() } will print "<nil>".
- dewey 8y agoQuery is the type. So it's a function that works on the type Query, takes an input parameter of type string and returns a struct of type Query.
- Noumenon72 8y agoWhy is it "q Query"?
- ovao 8y agoIn Go, the type follows the identifier. To use a single letter or short sequence of letters is idiomatic Go for short-lived identifiers or ‘quintessential’ identifiers.
- bbbobbb 8y agoThat the "receiver" identifier. You can think of it as "this" or "self" in other languages -> reference to the current "instance" within the method. https://gobyexample.com/methods https://gobyexample.com/methods
- dewey 8y agoq of type Query. So in the function you can access the fields on q (Where you call the function on) via q.FieldName. If you do it with a pointer receiver (q *Query) like explained here (https://tour.golang.org/methods/4 https://tour.golang.org/methods/4) you can directly manipulate the object, not the copy like in this example.
- edem 8y agoWhy didn't you port it to Kotlin instead? Much better language than Go in every aspect and also comes with an auto-conversion tool.
- kjksf 8y agoKotlin can already use existing Java library, I presume. The goal for the company was to enable an estimated 1 million of Go programmers to use RavenDB database. This is Java AND Go (and Python and Node) scenario, not Java OR Go.
- deleted 8y ago[deleted]
- hactually 8y agoSingle executables yet without JVM? Static typing? Simplified toolchain without make/maven files? My big one after swearing off of Java was never needing an IDE to do even the simplest things - is Kotlin useable without an IDE?
- mustardo 8y ago> Single executables yet without JVM? Kotlin native is coming along, uber jars are popular so only other departments is the JRE, Docker can bundle that > Static typing? Yes with good type inference and "null safety" > Simplified toolchain without make/maven files? You can use the compiler directly? But for anything serious a build tool is pretty essential for any language? At least you have options > My big one after swearing off of Java was never needing an IDE to do even the simplest things - is Kotlin useable without an IDE? You could write it with a Morse Code Keyer if you really wanted
- eikenberry 8y ago> Single executables yet without JVM? Yes. The JVM is Java's biggest downside, so why would you want to just move to a different language on the same overly complex (to put it mildly) runtime.
- 8y ago
- networkimprov 8y agoGo has code coverage analysis in its testing facilities... why not use those?
- kjksf 8y agoI do use builtin coverage. It's just it's nice to be able to track code coverage over time, which is what codecov.io provides. Also, running all tests (to get full code coverage) takes 20 minutes and makes fan on my laptop unhappy. The way it works is that on every checkin the CI job runs all the tests with code coverage enabled and uploads the results to codecov. Codecov can then plot coverage over time. My gripe is with inaccurate accounting of empty lines (like comments or struct definitions) by codecov. Go's tool to visualize this count them properly. I don't know if it's codecov or maybe I'm not sending the data properly.
- jimsmart 8y agoIn the above linked article you say: > Codecov is barely adequate. For Go, they count non-code lines (comments etc.) as not executed. It's impossible to get 100% code coverage as reported by the tool. I have an open source Go project that has some comments in methods, but still achieves 100% coverage using Codecov.io — I'm not sure what I do differently to yourself? (Perhaps I'm not using any inline struct definitions?) Here's a link, in case there's anything useful to you in my .travis.yml ? https://github.com/jimsmart/store4 https://github.com/jimsmart/store4 HTH
- kjksf 8y agoIt looks like we're both doing the same thing. I don't know, maybe I'm not counting things right but for example https://codecov.io/gh/ravendb/ravendb-go-client/src/master/delete_attachment_command_data.go https://codecov.io/gh/ravendb/ravendb-go-client/src/master/d... shows less then 80% coverage and there are only 2 lines not exected out of at least 18, which should be at most 10% counted as not covered.
- jimsmart 8y ago
- tannhaeuser 8y agoI'm wondering if the goal of this porting project would've been achieved by Java's newly gained native/AOT compilation feature. Probably not, as a huge Java .so brings the JVM's infrastructure for gc etc., and too much overhead. Still, I hope we can get rid of the language-centric ecosystems we have in favour of established OS/POSIX ways for shared libs, or an evolution thereof.
- kjksf 8y agoThere are an estimated 1 million of Go programmers. The goal of the project was to enable those Go programmers to be able to use RavenDB database in projects written in Go. The company also maintains Python client library and Node.js client library. It's not about Go vs. Java as a technology but enabling as many programmers as possible to use RavenDB.
- tannhaeuser 8y agoI understand the goal, and my comment wasn't so much arguing in favour of one language vs another, but rather that developing the same thing for multiple languages over and over seems wasteful.
- Matthias247 8y agoIn those client things the amount of the code that can be moved into a cross-language shared library might be lower than expected. E.g. the wire protocol and socket handling might be an option - but it's often not that big. The bigger part might be transformation of the programming languages types into something useful for the client (e.g. through serialization), which has be be redone anyway for each language. And after that the question comes up whether sharing the remaining things yields enough benefits to justify the hassle of having a dependency which is less portable, requires another build system, etc. That's my general experience with those kinds of projects - I don't know enough about RavenDB in particular to tell if it's the same here.
- yxhuvud 8y agoLooking at the way the wind blows it rather is towards more language centric ecosystems, than towards less of it. That said I'm still waiting for when the language ecosystems start to standardize how they interact with each other. That is a very hard nut to solve, but it would also be a quantum leap forward.
- arendtio 8y agoI wonder if it would have been easier (in terms of complexity, not in hours spend coding) to rebuild the project in Go (in 100% idiomatic Go) and port it to Java afterward. Sounds weird but since Go has such a strong focus on simplicity and Java probably implements most things Go uses, a port from Go to Java is probably a lot easier.
- kjksf 8y agoIt's true that if those code bases were evolving together, it would be possible to make them more in sync by sticking closer to what Go provides. But RavenDB is 10 years old. Java client already exists and has 50 thousands lines of code. If I was trying to rewrite that from scratch, I'm afraid Joel Spolsky would find me and spank me.
- grey-area 8y agoDid you at any point dream of writing a translation tool which would port java to go automatically? (crazy idea I know)
- kjksf 8y agoYes I did and I don't think it's crazy. I'm not sure a tool could do 100% translation but I'm sure it could do a lot. A surprisingly large amount of time was just moving the order of variable declaration from Java's "type name" to Go's "name type" and renaming, say, "String" to "string". If a tool did that for me, it would save a ton of time. Unfortunately, the upfront time investment to learn enough to write even the simplest translator would probably be greater than time saved on one project.
- gdy 8y agoAnd also rewrite other client libraries to keep them in sync with the new Java/Go code.
- michaelcampbell 8y agoSome of these examples were obviously cherry picked for one side of the other. Comparing against getters/setters, which are written once, generally automatically by the IDE, and never looked at again and omitting the ubiquitious "if err, _ = f()" rote seems a bit disingenuous.
- farmdawgnation 8y agoThey also neglect to mention that there are active conversations in the Java community around how to effectively implement Data Classes. For example: https://cr.openjdk.java.net/~briangoetz/amber/datum.html https://cr.openjdk.java.net/~briangoetz/amber/datum.html
- eclipseo76 8y ago>Java allows circular imports between packages. > >Go does not. Huh? As a package maintainer for a lot of Go stuff, I had to deal with tricky cyclic dependencies several times, especially in Google own Go packages like golang.org/x/build and google cloud.
- rat9988 8y agoWhat do you mean by "deal with"?
- thanatos_dem 8y agoThey had to take it into an alley and kill it, mafia style. More seriously, they probably means find ways to restructure packages and extract shared dependencies into subpackages to avoid cyclical dependencies.
- rat9988 8y agoWhich in turn means that go doesn't allow it, hence the original point.
- eclipseo76 8y agoGo expects to have all dependencies available at any time, downloading them from Internet if necessary. Cyclic dependencies are not a problem in that case because all dependencies are built together. Packaging for a distro requires building Go packages isolated from the network in a chroot and having all the dependencies previously built in the same manner. So it is an iterative process, building blocks by building blocks. If you have a cyclic dependencies, you can't build iteratively, and you'll have to excise certain part of the code to eliminate the cycle.
- andonisus 8y agoI personally have ported about 20K LoC from a java application to go; the majority of the code was from an underlying library used to model the data structures being used. It was mostly a class-by-class, function-by-function process of porting the base data structure and then all of its implementations, and culminated with the port of the services which uses all of it. Regarding the author's frustration of moving POJOs over (and their variable declarations), I used sublime text to select all variables based on a shard token, then cut them and moved them over by word. You can then lower case the first letters of every word, and then find-and-replace by type using a shared token. I found this method very quick and effective.
- mykowebhn 8y ago"Both Java and Go have interfaces but they are different things, like apples and salami."
- rurban 8y ago> Today if I was settings this up from scratch, I would stick with just AppVeyor, as it can now do both Linux and Windows testing and the future of TravisCI is murky, after it was acquired by private equity firm and reportedly fired the original dev team. I would rather go with CirrusCI which has windows, macos, linux and freebsd support, is much faster and is easier to work with. appveyor being the slowest, and travis having the worst features.
- kjksf 8y agoThanks for the tip, CirrusCI looks interesting.
- sathishvj 8y agoOne of the most interesting parts for me was the time taken - 601 hours for about 50k loc. That could be a starting point estimate for other projects.
- archagon 8y agoI’d say that’s not nearly enough time to write 50k loc from scratch! (Unless I’m horribly slow and no one is telling me.) Porting is a very different kind of work from feature development, since you already know the inputs and outputs of your system.
- deleted 8y ago[deleted]