14 ms·

A little Golang way

201 points by vonsnowman 11y ago

ploxiln 11y ago
I'm all for replacing java bloat with a little bit of go, but it seems like the cause of the bloat becoming a real problem was hopping on the docker bandwagon, for a collection of services that always run together in an appliance and really benefited from being hosted in a shared JVM? Isn't the JVM + servlet container thing supposed to be able to isolate the different services? They get their own bunch of threads, and their faults probably shouldn't crash other servlets?
- markrages 11y ago
  Most of our problems are self-inflicted.
- eternalban 11y ago
  You should read Sun's J2EE specs. There is little semantic distinction between a stateless session bean and a container hosted micro-service. The specs were never fully grasped (imo) by the Java community and those who got it (per gossip I heard) -- appserver vendors e.g. IBM, JBoss, etc. -- effectively crippled the spec by resisting the completion of the APIs that would commoditize their containers. Sun was really ahead of the game in various fronts.
  - timtadh 11y ago
    So true. I had to read the entirety of the JSP spec (and a few others) when I was an intern to help the company do some static analysis things. I also read a good chunk of Tomcat. Sun had this grand vision of App servers which you could just push code to and it would run completely isolated. The app servers where supposed to be spec compliant meaning your app could run on any implementation. But, and this is what bit the company I was working for, all of the implementations add lots of non-standard bits and broke standard bits. This meant that if you had written your app to go on WebSphere there was essentially no way it was going to run on TomCat without modification. If Sun had more tightly controlled the marketplace for these things I think the whole ecosystem would have been more robust.
- Deinumite 11y ago
  Traditionally you'd deploy a bunch of apps to one container (Tomcat, or whatever) and they'd all share the same JVM. I'm not sure if jars are shared in this way or not. I think OSGI was supposed to allow deploy apps to reuse the same jars and resources but I don't know how commonly that is deployed. Now it is more common to run Tomcat or Jetty embedded for your app so that they are isolated.
colordrops 11y ago
Wow that pun in the title is painful.
- sumobob 11y ago
  Theres a pun in the title?
  - axyjo 11y ago
    It's from a saying... a little goes a long way.
    - deleted 11y ago
      [deleted]
    - BinaryIdiot 11y ago
      Thank you; I didn't understand it either.
- callum85 11y ago
  Enjoyably so.
- vonsnowman 11y ago
  Hehe, yes, I'm afraid I have a proclivity for terrible puns. I do think they're somewhat less cringe-worthy that the link-bait titles that are all the rage these days (or at least endearingly cringeworthy).
sz4kerto 11y ago
"Code size was reduced by almost half, from 175 lines down to 96." Hm, I how can we take a 175 LOC project as something relevant in any way?
- mkozlows 11y ago
  The relevance is that even a 175 LOC project in Java takes up enormous memory and disk footprint -- that the JVM is this super-heavyweight thing that's really just inherently inappropriate for a lot of applications.
  - xienze 11y ago
    > The relevance is that even a 175 LOC project in Java takes up enormous memory and disk footprint _This particular_ 175 LOC Java project takes up an enormous amount of memory. For all you know it was just coded poorly and the Go one is a bit more reasonable. > the JVM is this super-heavyweight thing that's really just inherently inappropriate for a lot of applications. The JVM introduces overhead but not so much that you can make these sorts of extrapolations.
    - mkozlows 11y ago
      34MB for Hello World, according to this SO post: http://stackoverflow.com/questions/13692206/high-java-memory-usage-even-for-small-programs http://stackoverflow.com/questions/13692206/high-java-memory... I'm not going to install the JDK just to verify, but feel free to report back if you get different results.
      - tracker1 11y ago
        For comparison... I did a similar Hello World test around the timeframe of .Net 2.0, and it was around 11mb to load IIRC. The latest iojs on windows seems to be just under 9mb. Not sure what the golang overhead is, by comparison.
        11y ago
jsnk 11y ago
More and more I hear about Go, I feel more convinced that it would be worth giving it a shot. Can you share what the microservices are doing? What are they for?
- vonsnowman 11y ago
  At this point we have 6 microservices written in go in the appliance: - team-server probe: already mentioned in the blog post. Determines if any installed Team Server is down. - ca: as mentioned in the blog post, as simple Certificate Authority - charlie: a checkin service. Desktop clients periodically post to it to signify they are up. This data is used in each user's device list to show if the device is up and which ip it was last seen from. - auditor: takes audit event in an HTTP endpoint and forwards them to a raw TCP connection as expected by splunk and co - valkyrie: a relay server used for data transfers when desktop clients cannot establish direct TCP connection (more about that in a future blog post) - lipwig: a messaging/pubsub server used for peer discovery and notifications (more about that in a future blog post)
aikah 11y ago
I'll bite. While it is absolutely true Go servers use less memory that classic servlet deployments, The question is what were you using at first place? where you using a big framework? with this or that big IoC container ? with a bloated ORM ? ... or where you using barebone jdbc and writing servlets without any framework? because essentially that's what you're doing with Go, Go has 0 big framework(and no Beego or Revel are not complex), 0 big ORM , 0 IoC container ... And while the Java culture is about heavy decoupling and reusability , the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't. So I'm curious. My point is didn't you reduce memory usage because Go forced you to write code a certain way ?
- jgalt212 11y ago
  > the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't. So I'm curious. That's a very strong statement. Do you have evidence to support this? I don't code in Go right now, but have been keeping an eye on it as it continues to gather momentum.
  - dcsommer 11y ago
    Lack of generics can make it harder to write cleanly reusable modules in Go. I've noticed gophers are often less allergic to copy pasting code with tweaks (as long as it isn't too many lines) and don't mind rewriting similar code if it is obviously correct. It was a bit of a culture shock for me at least.
    - deleted 11y ago
      [deleted]
    - _ak 11y ago
      You confuse parametric polymorphism with reusability. The former is merely one of several ways to achieve the latter. The Go way for reusability usually consists of interfaces.
melling 11y ago
I've got a little Swift Language "search engine" that I built with Go on App Engine. I haven't crawled any sites (yet). I simply created a Go data structure to do in memory searches because I didn't feel like using Google's data store. Here's the site: http://www.h4labs.com/dev/ios/swift.html http://www.h4labs.com/dev/ios/swift.html Here's the data: https://github.com/melling/SwiftResources/blob/master/swift_urls.tsv https://github.com/melling/SwiftResources/blob/master/swift_... I'm approaching 1400 URL's and it's still snappy. I was hoping Go's claimed "efficiency" would keep freely hosted a bit longer than Python.
- akhilcacharya 11y ago
  >I'm approaching 1400 URL's and it's still snappy. I was hoping Go's claimed "efficiency" would keep freely hosted a bit longer than Python. That is exactly why I picked my Go for my latest side project. I just hope the framework I'm using isn't the bottleneck, because I really hate writing pure Go servers.
stesch 11y ago
How tied is Go into Google? What if Google drops Go?
- shocks 11y ago
  Go is a programming language born out of Google for sure, but that does not mean it depends on Google. It's a community project after all.
  - aikah 11y ago
    > It's a community project after all. It is absolutely not a community project. Go governance is a 100% @ Google . There is no go committee or go working group outside Google. It's backed by one company that has full control over it. Sure it's open-source, but good luck with a fork. How can anybody be so misleading about that fact ? What did make you think Go is a "community project" ? > but that does not mean it depends on Google. There is a top down , vertical relationship between the Go team at Google and the rest of Go users, Go main goal is to fulfill the Go team needs, period. If you find it useful then it's a bonus. That's exactly how the Go team speaks and act, in fact the Go team makes it really hard to contribute to the core. Please, stop saying what you say, it is completely untrue and a total mis-characterization of how the Go project work. I still want to know what made you think Go is a "community project".
    - 0xdeadbeefbabe 11y ago
      What do you mean good luck with a fork? I know of one fork that will provide some lucky grad students with a degree. Can't find the link right now.
    - ansible 11y ago
      I still want to know what made you think Go is a "community project". Well, the main developers do listen to the community. Any project can fork, if there are enough people who are very dissatisfied with the current management. I'm not aware of any serious grumblings about such a fork though. Why? The core team is very good, and very narrowly focused. They've communicated clearly on nearly every issue as to what they're doing, and why. It seems clear that they are intent on making the best possible tool that fits with their particular vision. There are people who agree with this vision, and broadly support their efforts. And then there are people who really don't like their vision, and wander off to use Rust or something else. The core team has enormous respect from the existing golang community. If Google suddenly laid off the developers (or just switched them to something else) or otherwise dramatically changed direction in their support for the project, the community would move in quickly to help out the situation. I could see people getting together to form a non-profit foundation that could at least pay for a few guys to continue to work on Go full-time. But currently, there is no apparent need, so it hasn't been done. Google is willing to pay substantially for the development, and I don't see gophers complaining about that.
travjones 11y ago
There have been several blog posts and conference presentations with a similar theme -- "Go is efficient, especially compared to X." I know it seems like hype, but I encourage others to try Golang out, maybe even slap a web app together with it. What you'll find is that your Go app will be extremely efficient and performant. It's kind of unfair to compare Go to many of the other languages of the web because it is compiled, which is a huge reason why it is so performant. So rather than compare Go to other languages, I encourage you all to give it a try. If you already program, working on a small Go side project will get you up to speed quickly and you'll learn about some of the awesome packages individuals from the Go community have put together for us. Disclaimer: I do not work for Google. I write Go code and enjoy it. I think others will too.
- eropple 11y ago
  > What you'll find is that your Go app will be extremely efficient and performant. This is true, for sure. But my experience--and I've written plenty of Go, though I find it unpleasant to write and think about for all the usual reasons that Go partisans roll their eyes and so would rather deal with it as artifacts rather than actually writing it--has led to watching lots of developers make mudball codebases in the process (while blogging very heavily about how great it is three weeks in, and less a year-plus in). A pursuit of faux-simplicity has led to what I see as reinventing a lot of Java-1.4-era (because the language itself is essentially that) design patterns--and we should be reminded that design patterns exist to address defects in tooling--that more expressive languages have managed to avoid; the general desire for smaller applications helps to some extent, but software has this tendency to grow that I don't think Go gives you the tools to effectively manage and maintain. Reasonable minds can differ, of course. Past that, while I think it's an alright choice for company-internal deployable products--web servers, worker nodes, etc.--I have straight-up problems with it being the new devops tool of choice. Statically linking your SSL library makes you an asshole when it inevitably fails and now an application has to be regression-tested so that new features and new bugs don't hose you just because that's the only way to upgrade Heartbleed 2.0. (Go also discourages program extensibility through components and rather as recompilation; the stuff Packer and Terraform do to provide "plugins" that the same company's Vagrant just did with a `require` is gross and, to my mind, completely foolish.) So I wouldn't say it's hype, but I would say it's not all true, either.
lpsz 11y ago
Anecdotes like this makes me wonder how much funding money and electricity could be saved if people migrated en masse from Ruby/Python/Something else to Go. (Edit: not meant to be a political statement, more of a practical observation. I only recently started using Go.)
- tracker1 11y ago
  Also, the more complex a system is, the less the runtime overhead is by comparison. Hello world examples in .Net, node.js etc are 35-20% the memory overhead compared to Java. It's not like golang doesn't have some overhead of its' own... There's also Rust, and D to consider.
- jacquesm 11y ago
  That may be one of the bigger chances for go: mobile. After all being more efficient on mobile translates into longer battery life and go potentially has a huge advantage over the current java (ok, not java) based environment on Android. For google this would be a triple win, get out of the Oracle mess entirely, give their mobile developers a super fast toolchain and give the end users better battery life.
  - swah 11y ago
    I love Golang, but its ain't very expressive, so it cannot be a good match for writing UIs. And Swift is much closer to Rust than to Go, IIUC...
- melling 11y ago
  These guys went from 30 servers to 2 by switching to Go. http://www.iron.io/blog/2013/03/how-we-went-from-30-servers-to-2-go.html http://www.iron.io/blog/2013/03/how-we-went-from-30-servers-...
  - dougbarrett 11y ago
    the best part was, they had 2 only for redundancy, they could have gotten away with only having one.
jaytaylor 11y ago
Can anyone shed some light on how/why running the same Java apps in a docker container significantly increased the memory footprint? Is JVM overhead shared when multiple Java apps are being run on the same machine?
- haldean 11y ago
  A lot of it is, yes. The JVM loads a large number of largeish class files when it starts; these contain implementations of the standard library and whatever else is on your classpath that you've imported. These have the nice property that they're read-only, though, so multiple JVM processes can safely share them. When you move to sandboxing each JVM off by itself, you lose the ability for them to share memory (which is, in a sense, a _feature_ of sandboxing), so now each of them has to take the 50-100MB hit of those formerly-shared memory regions. (Note that the huge size of the classes is also the big reason why JVM startup time is so crap; another reason that multitenant JVM systems are great is that every process after the first starts much faster)
- lobster_johnson 11y ago
  Assuming this is referrring to the fact that separate containers will link in separate copies of all the binaries (executable + shared libraries), instead of sharing the pages across all instances of he JVM. There's no way for the kernel to know that they're all the same files. So a lot of code is duplicated.
_JamesA_ 11y ago
Java's main feature is compile once - run (almost) anywhere. From embedded to mainframe as long as a JVM is available. How does Go stack up for cross-platform development? Does every application and library have to be (re)compiled for the target platform? What about support for alternative architectures (ARM, PowerPC, etc)?
- aikah 11y ago
  AFAIK Go support cross-compilation for multiple targets... as long as one isn't linking C libraries.
- xienze 11y ago
  Go does require recompilation for every platform, but I'll give them this, it's very painless. Apparently in 1.5 you can just set an environment variable for your architecture and one for your OS and you're good to go.
findjashua 11y ago
the only thing that's preventing me from jumping on the Go bandwagon is lack of a nice collections library (a la lodash).
- politician 11y ago
  That's due to a lack of generics. There are some workarounds like gonerics [1] -- a clever abuse of import declarations. [1] http://bouk.co/blog/idiomatic-generics-in-go/ http://bouk.co/blog/idiomatic-generics-in-go/
davexunit 11y ago
And the cycle continues, from one crappy enterprise language to the next. I don't who said it, but whoever said "Go is a bold step backwards" is spot on.
- hwh 11y ago
  There are times when a step back is the mandatory step to get a new perspective on things. Just sayin'.
  - davexunit 11y ago
    But this is a step back to the cold, dead, static systems of UNIX. How about stepping back and taking a look at the dynamic, hackable environments of Lisp?
    - _ak 11y ago
      You should rather ask yourself why Lisp hasn't been able to attract the same large and active community in 5 decades that Go has been able to attract in 5 years.
      - davexunit 11y ago
        Argument ad populum.
        _ak 11y ago
        I'm not saying either of them is better because or the amount of people they attracted. I merely ask why Go managed to interest more people than Lisp in an order of magnitude of time less.
    - vezzy-fnord 11y ago
      I don't understand why Unix is equated to "cold, dead and static". Sure, that's the case for V7 Unix and descendents. Attempted Unix++ systems like Amoeba, Spring and Sprite were quite the opposite. Further, if I wanted maximum dynamism and runtime hackability I'd go for a Smalltalk.
umhan35 11y ago
I hope that there is a compiled version of Scala.
kentonv 11y ago
> Resident memory usage dropped from 87MB down to a mere 3MB, a 29x reduction! This isn't so much Java vs. Go as it is JIT/interpreted vs. AOT-compiled. The numbers are entirely typical across a wide range of such comparisons. With an interpreter or JIT, you need to load all your code at startup and process it. Generally, _all_ of your dependencies need to be loaded and parsed upfront, either converted to some internal in-memory representation or JIT'd directly to machine code. This will allocate a bunch of heap structures during the processing, and the end result is a bunch of data that needs to stay resident and can't easily be shared with other processes. With AOT, you mmap() in a file. The OS only pages in the code you execute, and can page it back out as needed. Pages are shared between all processes running the same executable. At sandstorm.io our rule of thumb is that an app written in Node, Ruby, Python, PHP, etc. will take 100MB of RAM while an app written in C++, Rust, or Go will take 2MB. Since Sandstorm runs per-user (and even per-document) app instances, this is a pretty big deal. The good news is that https://github.com/google/snappy-start https://github.com/google/snappy-start should fix this problem: by checkpointing the process after it finishes its parsing/JITing but before it starts handling requests, we can get an mmap-able starting state that is very much like an AOT-compiled binary. At least, in theory -- there's still a bunch of work to do for this to actually work in practice. > The resulting docker image shrunk from 668MB to 4.3MB While I would expect the Go image to be smaller (since Go builds static binaries, so literally all you need in the image is the binary), I suspect that the 668MB Java image was at least 90% unnecessary garbage that was not actually needed at runtime. Unfortunately the package managers we all use are not optimized for containers; instead they evolved targeting systems with dedicated disks that can easily absorb gigabytes of bloat. In Debian, for example, every package implicitly depends on coreutils. That's perfectly reasonable when installing an OS on a machine: you almost certainly need a working shell to boot and administer your machine. But a container can get by just fine without coreutils, and a typical web server probably (hopefully) doesn't need to call out to a shell. Even if a shell is needed, busybox/toybox is probably sufficient and will take a lot less space. Packages also often contain things like documentation, unit tests, etc. which obviously aren't needed in a container. For Sandstorm.io we deal with this problem by running the app in a mode where we trace all the files it actually uses, and then we build a package containing only those. It mostly works and manages to keep packages reasonably-sized, but it does lead to bugs of the form: "I forgot to test this feature while tracing, so the assets it requires didn't make it into the package." We're looking for better options.
s73v3r 11y ago
You know, as fun as it sounds like this was, wouldn't the natural solution to the problem be to roll back the Docker rollout?
skarap 11y ago
I guess nobody will be saying that Java is lightweight, but the actual results of porting from Java to Go will vary wildly from usecase to usecase. Both examples mentioned in the article (the 175LoC program and the CA) sound like very simple programs. E.g. I once wrote a C program which watched some directories with inotify and compressed new files using zlib. The memory footprint was 350KB. Obviously an empty JVM alone would use 100x more RAM. This static ~30MB overhead might be important in some cases and not relevant at all in others. The incremental (per-object) overhead is probably more relevant to almost all real-world usecases which are a bit more complex then the ones mentioned above. Also - care should be taken not to compare apples (no TM) to oranges: e.g. if you use a huge ORM in Java (which among other things also caches results of each query) and then do a simple SQL query in the new implementation, would be strange to expect those two to perform similar. This happens quite often with rewrites - they almost always aim for simplicity, do short cuts, get rid of "unused stuff". Basically a rewrite from Java to Java will also usually improve performance. Must-read about rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html . Though, I guess, everybody has already read it.
carloscarnero 11y ago
At my organization we have several Java-based services (some of them can even qualify as a form of microservices, if you squint your eyes). We have found that when you have very good developers and you write almost to the letter of the spec, Java can easily provide a stable base (which is probably true of any language/runtime/library). However, we've been eager to try Go in several places. Believe it or not, what has held us back is the lack of a solid LDAP library. We could/should scratch our own itch and be done with it, we lack the time... still so many things to do! In the mean time, for us, Java support for LDAP is nothing short of stellar; and has been for years.
bliti 11y ago
I've been writing Go for some time. Just finished writing a small CDN for the company I currently work for. It's It's a fast language that performs well in systems programming (what it was made for). But boy is it ugly. Not verbose like Java, but ugly to write and read. It does force a good coding convention because otherwise you end up with a pile of ugly code. People seem to be bent on writing it like 80s C. Full of single character variable names and odd function naming. Dunno if it's just my experience. I do like the fact that it resembles python in how it feels. Overall I'd say it's a nice language that is not for everybody. I am playing with Elixir these days. Go is good enougb, but I'm not happy with it being that.
mavelikara 11y ago
I have a question on Go's performance that I'd like someone here, who has experience writing programs in it, to help shed some light upon. 1) Go does not have a runtime. This means that there is no JIT to do any optimizations based on runtime profiling. 2) Go is also designed to compile fast. Since it compiles fast, the compiler's time budget to do compile-time optimizations is small and it probably can't do the best job possible. Are these two points accurate? If so, how does Go perform as well as it is claimed to do? Where is the catch?
- flippant 11y ago
  1) Go has a runtime that's included in the binary that you get when you run `go build`. It takes care of garbage collection, goroutines etc. It does not do any JIT compilation afaik. 2) Go compiles fast because it gives up a lot of modern features (namely generics).
  - deleted 11y ago
    [deleted]
anoother 11y ago
Something troubles me about this article. I hope I'm misunderstanding it. It's stems from the following quotes; knowing this information, why do AeroFS still think Docker is a good fit for their use-case? > However, after our move to Docker, we noticed a sharp increase of the appliance's memory footprint. > ... > We identified several major factors behind these symptoms: > 1. an increase in the number of running JVMs, as each tomcat servlet was placed into a separate container > 2. reduced opportunity for the many JVMs to share read-only memory: the JVM itself, all the shared libraries it depends on, and of course the many JARs used by multiple services > 3. memory isolation could in some cases confuse some sizing heuristics, which lead to larger caches being allocated by some services
- ambrice 11y ago
  Presumably Docker offers some benefit. It's weird that you and several others have jumped to the conclusion that it's obvious they should get rid of Docker. I would say: knowing this information, why do you still think java is a good fit for their use-case?
  - anoother 11y ago
    Because: a) Their service was already written in it b) Their service was already performing adequately before they brought docker into play But those aren't answers to your question. Java may have been a terrible fit for them, but their decision to move away from it was not motivated by that -- it was motivated by their already-written, already-performing Java code no longer performing as well, because of Docker. The only question is, does Docker bring them enough benefits to justify reimplementing large amounts code, in a new and unfamiliar language? Maybe it does, but somehow I doubt it.
djhworld 11y ago
I find this article confusing, in the first part the writer talks about Tomcat and servlets, in the second part he talks about reducing the LOC from 175 lines to 96 by using Go. To me it seems the Java solution was wildly over engineered and probably could have been refactored to not even need Tomcat.
themartorana 11y ago
I am not sure I understand where the issue is Java pure and simple, and where the issue is that Java services were impaired by being wrapped by Docker? "Everything was fine until we switched to Docker" makes me think the trouble may not be all Java. Anyone have educated thoughts on this?
nickpsecurity 11y ago
I appreciate you publishing this experiment as it will show Java's problems for what they are. However, a real test of Go's safety and efficiency would be a comparison to a similar language such as an optimized Wirth language, a subset of Free Pascal (Delphi-style), typed LISP (eg Racket), Julia, or Ada. People keep forgetting about the last one in safe, efficient, systems programming despite it doing for a long time what Go and Rust hope to do eventually. Its long-time use in embedded systems indicates it can be quite efficient. Regardless, Go certainly improves on the reliability of applications vs C++ while being much more efficient than .NET or Java.
tapirl 11y ago
I always think Golang is a Java killer, but I think Java itself is more a Java killer than Golang.

201 points by vonsnowman 11y ago

I'm all for replacing java bloat with a little bit of go, but it seems like the cause of the bloat becoming a real problem was hopping on the docker bandwagon, for a collection of services that always run together in an appliance and really benefited from being hosted in a shared JVM? Isn't the JVM + servlet container thing supposed to be able to isolate the different services? They get their own bunch of threads, and their faults probably shouldn't crash other servlets?

markrages 11y ago
Most of our problems are self-inflicted.
eternalban 11y ago
You should read Sun's J2EE specs. There is little semantic distinction between a stateless session bean and a container hosted micro-service. The specs were never fully grasped (imo) by the Java community and those who got it (per gossip I heard) -- appserver vendors e.g. IBM, JBoss, etc. -- effectively crippled the spec by resisting the completion of the APIs that would commoditize their containers. Sun was really ahead of the game in various fronts.
- timtadh 11y ago
  So true. I had to read the entirety of the JSP spec (and a few others) when I was an intern to help the company do some static analysis things. I also read a good chunk of Tomcat. Sun had this grand vision of App servers which you could just push code to and it would run completely isolated. The app servers where supposed to be spec compliant meaning your app could run on any implementation. But, and this is what bit the company I was working for, all of the implementations add lots of non-standard bits and broke standard bits. This meant that if you had written your app to go on WebSphere there was essentially no way it was going to run on TomCat without modification. If Sun had more tightly controlled the marketplace for these things I think the whole ecosystem would have been more robust.
Deinumite 11y ago
Traditionally you'd deploy a bunch of apps to one container (Tomcat, or whatever) and they'd all share the same JVM. I'm not sure if jars are shared in this way or not. I think OSGI was supposed to allow deploy apps to reuse the same jars and resources but I don't know how commonly that is deployed. Now it is more common to run Tomcat or Jetty embedded for your app so that they are isolated.

colordrops 11y ago

Wow that pun in the title is painful.

sumobob 11y ago
Theres a pun in the title?
- axyjo 11y ago
  It's from a saying... a little goes a long way.
  - deleted 11y ago
    [deleted]
  - BinaryIdiot 11y ago
    Thank you; I didn't understand it either.
callum85 11y ago
Enjoyably so.
vonsnowman 11y ago
Hehe, yes, I'm afraid I have a proclivity for terrible puns. I do think they're somewhat less cringe-worthy that the link-bait titles that are all the rage these days (or at least endearingly cringeworthy).

sz4kerto 11y ago

"Code size was reduced by almost half, from 175 lines down to 96." Hm, I how can we take a 175 LOC project as something relevant in any way?

mkozlows 11y ago
The relevance is that even a 175 LOC project in Java takes up enormous memory and disk footprint -- that the JVM is this super-heavyweight thing that's really just inherently inappropriate for a lot of applications.
- xienze 11y ago
  > The relevance is that even a 175 LOC project in Java takes up enormous memory and disk footprint _This particular_ 175 LOC Java project takes up an enormous amount of memory. For all you know it was just coded poorly and the Go one is a bit more reasonable. > the JVM is this super-heavyweight thing that's really just inherently inappropriate for a lot of applications. The JVM introduces overhead but not so much that you can make these sorts of extrapolations.
  - mkozlows 11y ago
    34MB for Hello World, according to this SO post: http://stackoverflow.com/questions/13692206/high-java-memory-usage-even-for-small-programs http://stackoverflow.com/questions/13692206/high-java-memory... I'm not going to install the JDK just to verify, but feel free to report back if you get different results.
    - tracker1 11y ago
      For comparison... I did a similar Hello World test around the timeframe of .Net 2.0, and it was around 11mb to load IIRC. The latest iojs on windows seems to be just under 9mb. Not sure what the golang overhead is, by comparison.
      - 11y ago

jsnk 11y ago

More and more I hear about Go, I feel more convinced that it would be worth giving it a shot. Can you share what the microservices are doing? What are they for?

vonsnowman 11y ago
At this point we have 6 microservices written in go in the appliance: - team-server probe: already mentioned in the blog post. Determines if any installed Team Server is down. - ca: as mentioned in the blog post, as simple Certificate Authority - charlie: a checkin service. Desktop clients periodically post to it to signify they are up. This data is used in each user's device list to show if the device is up and which ip it was last seen from. - auditor: takes audit event in an HTTP endpoint and forwards them to a raw TCP connection as expected by splunk and co - valkyrie: a relay server used for data transfers when desktop clients cannot establish direct TCP connection (more about that in a future blog post) - lipwig: a messaging/pubsub server used for peer discovery and notifications (more about that in a future blog post)

aikah 11y ago

I'll bite. While it is absolutely true Go servers use less memory that classic servlet deployments, The question is what were you using at first place? where you using a big framework? with this or that big IoC container ? with a bloated ORM ? ... or where you using barebone jdbc and writing servlets without any framework? because essentially that's what you're doing with Go, Go has 0 big framework(and no Beego or Revel are not complex), 0 big ORM , 0 IoC container ... And while the Java culture is about heavy decoupling and reusability , the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't. So I'm curious. My point is didn't you reduce memory usage because Go forced you to write code a certain way ?

jgalt212 11y ago
> the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't. So I'm curious. That's a very strong statement. Do you have evidence to support this? I don't code in Go right now, but have been keeping an eye on it as it continues to gather momentum.
- dcsommer 11y ago
  Lack of generics can make it harder to write cleanly reusable modules in Go. I've noticed gophers are often less allergic to copy pasting code with tweaks (as long as it isn't too many lines) and don't mind rewriting similar code if it is obviously correct. It was a bit of a culture shock for me at least.
  - deleted 11y ago
    [deleted]
  - _ak 11y ago
    You confuse parametric polymorphism with reusability. The former is merely one of several ways to achieve the latter. The Go way for reusability usually consists of interfaces.

melling 11y ago

I've got a little Swift Language "search engine" that I built with Go on App Engine. I haven't crawled any sites (yet). I simply created a Go data structure to do in memory searches because I didn't feel like using Google's data store. Here's the site: http://www.h4labs.com/dev/ios/swift.html http://www.h4labs.com/dev/ios/swift.html Here's the data: https://github.com/melling/SwiftResources/blob/master/swift_urls.tsv https://github.com/melling/SwiftResources/blob/master/swift_... I'm approaching 1400 URL's and it's still snappy. I was hoping Go's claimed "efficiency" would keep freely hosted a bit longer than Python.

akhilcacharya 11y ago
>I'm approaching 1400 URL's and it's still snappy. I was hoping Go's claimed "efficiency" would keep freely hosted a bit longer than Python. That is exactly why I picked my Go for my latest side project. I just hope the framework I'm using isn't the bottleneck, because I really hate writing pure Go servers.

stesch 11y ago

How tied is Go into Google? What if Google drops Go?

shocks 11y ago
Go is a programming language born out of Google for sure, but that does not mean it depends on Google. It's a community project after all.
- aikah 11y ago
  > It's a community project after all. It is absolutely not a community project. Go governance is a 100% @ Google . There is no go committee or go working group outside Google. It's backed by one company that has full control over it. Sure it's open-source, but good luck with a fork. How can anybody be so misleading about that fact ? What did make you think Go is a "community project" ? > but that does not mean it depends on Google. There is a top down , vertical relationship between the Go team at Google and the rest of Go users, Go main goal is to fulfill the Go team needs, period. If you find it useful then it's a bonus. That's exactly how the Go team speaks and act, in fact the Go team makes it really hard to contribute to the core. Please, stop saying what you say, it is completely untrue and a total mis-characterization of how the Go project work. I still want to know what made you think Go is a "community project".
  - 0xdeadbeefbabe 11y ago
    What do you mean good luck with a fork? I know of one fork that will provide some lucky grad students with a degree. Can't find the link right now.
  - ansible 11y ago
    I still want to know what made you think Go is a "community project". Well, the main developers do listen to the community. Any project can fork, if there are enough people who are very dissatisfied with the current management. I'm not aware of any serious grumblings about such a fork though. Why? The core team is very good, and very narrowly focused. They've communicated clearly on nearly every issue as to what they're doing, and why. It seems clear that they are intent on making the best possible tool that fits with their particular vision. There are people who agree with this vision, and broadly support their efforts. And then there are people who really don't like their vision, and wander off to use Rust or something else. The core team has enormous respect from the existing golang community. If Google suddenly laid off the developers (or just switched them to something else) or otherwise dramatically changed direction in their support for the project, the community would move in quickly to help out the situation. I could see people getting together to form a non-profit foundation that could at least pay for a few guys to continue to work on Go full-time. But currently, there is no apparent need, so it hasn't been done. Google is willing to pay substantially for the development, and I don't see gophers complaining about that.

travjones 11y ago

There have been several blog posts and conference presentations with a similar theme -- "Go is efficient, especially compared to X." I know it seems like hype, but I encourage others to try Golang out, maybe even slap a web app together with it. What you'll find is that your Go app will be extremely efficient and performant. It's kind of unfair to compare Go to many of the other languages of the web because it is compiled, which is a huge reason why it is so performant. So rather than compare Go to other languages, I encourage you all to give it a try. If you already program, working on a small Go side project will get you up to speed quickly and you'll learn about some of the awesome packages individuals from the Go community have put together for us. Disclaimer: I do not work for Google. I write Go code and enjoy it. I think others will too.

eropple 11y ago
> What you'll find is that your Go app will be extremely efficient and performant. This is true, for sure. But my experience--and I've written plenty of Go, though I find it unpleasant to write and think about for all the usual reasons that Go partisans roll their eyes and so would rather deal with it as artifacts rather than actually writing it--has led to watching lots of developers make mudball codebases in the process (while blogging very heavily about how great it is three weeks in, and less a year-plus in). A pursuit of faux-simplicity has led to what I see as reinventing a lot of Java-1.4-era (because the language itself is essentially that) design patterns--and we should be reminded that design patterns exist to address defects in tooling--that more expressive languages have managed to avoid; the general desire for smaller applications helps to some extent, but software has this tendency to grow that I don't think Go gives you the tools to effectively manage and maintain. Reasonable minds can differ, of course. Past that, while I think it's an alright choice for company-internal deployable products--web servers, worker nodes, etc.--I have straight-up problems with it being the new devops tool of choice. Statically linking your SSL library makes you an asshole when it inevitably fails and now an application has to be regression-tested so that new features and new bugs don't hose you just because that's the only way to upgrade Heartbleed 2.0. (Go also discourages program extensibility through components and rather as recompilation; the stuff Packer and Terraform do to provide "plugins" that the same company's Vagrant just did with a `require` is gross and, to my mind, completely foolish.) So I wouldn't say it's hype, but I would say it's not all true, either.

lpsz 11y ago

Anecdotes like this makes me wonder how much funding money and electricity could be saved if people migrated en masse from Ruby/Python/Something else to Go. (Edit: not meant to be a political statement, more of a practical observation. I only recently started using Go.)

tracker1 11y ago
Also, the more complex a system is, the less the runtime overhead is by comparison. Hello world examples in .Net, node.js etc are 35-20% the memory overhead compared to Java. It's not like golang doesn't have some overhead of its' own... There's also Rust, and D to consider.
jacquesm 11y ago
That may be one of the bigger chances for go: mobile. After all being more efficient on mobile translates into longer battery life and go potentially has a huge advantage over the current java (ok, not java) based environment on Android. For google this would be a triple win, get out of the Oracle mess entirely, give their mobile developers a super fast toolchain and give the end users better battery life.
- swah 11y ago
  I love Golang, but its ain't very expressive, so it cannot be a good match for writing UIs. And Swift is much closer to Rust than to Go, IIUC...
melling 11y ago
These guys went from 30 servers to 2 by switching to Go. http://www.iron.io/blog/2013/03/how-we-went-from-30-servers-to-2-go.html http://www.iron.io/blog/2013/03/how-we-went-from-30-servers-...
- dougbarrett 11y ago
  the best part was, they had 2 only for redundancy, they could have gotten away with only having one.

jaytaylor 11y ago

Can anyone shed some light on how/why running the same Java apps in a docker container significantly increased the memory footprint? Is JVM overhead shared when multiple Java apps are being run on the same machine?

haldean 11y ago
A lot of it is, yes. The JVM loads a large number of largeish class files when it starts; these contain implementations of the standard library and whatever else is on your classpath that you've imported. These have the nice property that they're read-only, though, so multiple JVM processes can safely share them. When you move to sandboxing each JVM off by itself, you lose the ability for them to share memory (which is, in a sense, a _feature_ of sandboxing), so now each of them has to take the 50-100MB hit of those formerly-shared memory regions. (Note that the huge size of the classes is also the big reason why JVM startup time is so crap; another reason that multitenant JVM systems are great is that every process after the first starts much faster)
lobster_johnson 11y ago
Assuming this is referrring to the fact that separate containers will link in separate copies of all the binaries (executable + shared libraries), instead of sharing the pages across all instances of he JVM. There's no way for the kernel to know that they're all the same files. So a lot of code is duplicated.

_JamesA_ 11y ago

Java's main feature is compile once - run (almost) anywhere. From embedded to mainframe as long as a JVM is available. How does Go stack up for cross-platform development? Does every application and library have to be (re)compiled for the target platform? What about support for alternative architectures (ARM, PowerPC, etc)?

aikah 11y ago
AFAIK Go support cross-compilation for multiple targets... as long as one isn't linking C libraries.
xienze 11y ago
Go does require recompilation for every platform, but I'll give them this, it's very painless. Apparently in 1.5 you can just set an environment variable for your architecture and one for your OS and you're good to go.

findjashua 11y ago

the only thing that's preventing me from jumping on the Go bandwagon is lack of a nice collections library (a la lodash).

politician 11y ago
That's due to a lack of generics. There are some workarounds like gonerics [1] -- a clever abuse of import declarations. [1] http://bouk.co/blog/idiomatic-generics-in-go/ http://bouk.co/blog/idiomatic-generics-in-go/

davexunit 11y ago

And the cycle continues, from one crappy enterprise language to the next. I don't who said it, but whoever said "Go is a bold step backwards" is spot on.

hwh 11y ago
There are times when a step back is the mandatory step to get a new perspective on things. Just sayin'.
- davexunit 11y ago
  But this is a step back to the cold, dead, static systems of UNIX. How about stepping back and taking a look at the dynamic, hackable environments of Lisp?
  - _ak 11y ago
    You should rather ask yourself why Lisp hasn't been able to attract the same large and active community in 5 decades that Go has been able to attract in 5 years.
    - davexunit 11y ago
      Argument ad populum.
      - _ak 11y ago
        I'm not saying either of them is better because or the amount of people they attracted. I merely ask why Go managed to interest more people than Lisp in an order of magnitude of time less.
  - vezzy-fnord 11y ago
    I don't understand why Unix is equated to "cold, dead and static". Sure, that's the case for V7 Unix and descendents. Attempted Unix++ systems like Amoeba, Spring and Sprite were quite the opposite. Further, if I wanted maximum dynamism and runtime hackability I'd go for a Smalltalk.

umhan35 11y ago

I hope that there is a compiled version of Scala.

kentonv 11y ago

> Resident memory usage dropped from 87MB down to a mere 3MB, a 29x reduction! This isn't so much Java vs. Go as it is JIT/interpreted vs. AOT-compiled. The numbers are entirely typical across a wide range of such comparisons. With an interpreter or JIT, you need to load all your code at startup and process it. Generally, _all_ of your dependencies need to be loaded and parsed upfront, either converted to some internal in-memory representation or JIT'd directly to machine code. This will allocate a bunch of heap structures during the processing, and the end result is a bunch of data that needs to stay resident and can't easily be shared with other processes. With AOT, you mmap() in a file. The OS only pages in the code you execute, and can page it back out as needed. Pages are shared between all processes running the same executable. At sandstorm.io our rule of thumb is that an app written in Node, Ruby, Python, PHP, etc. will take 100MB of RAM while an app written in C++, Rust, or Go will take 2MB. Since Sandstorm runs per-user (and even per-document) app instances, this is a pretty big deal. The good news is that https://github.com/google/snappy-start https://github.com/google/snappy-start should fix this problem: by checkpointing the process after it finishes its parsing/JITing but before it starts handling requests, we can get an mmap-able starting state that is very much like an AOT-compiled binary. At least, in theory -- there's still a bunch of work to do for this to actually work in practice. > The resulting docker image shrunk from 668MB to 4.3MB While I would expect the Go image to be smaller (since Go builds static binaries, so literally all you need in the image is the binary), I suspect that the 668MB Java image was at least 90% unnecessary garbage that was not actually needed at runtime. Unfortunately the package managers we all use are not optimized for containers; instead they evolved targeting systems with dedicated disks that can easily absorb gigabytes of bloat. In Debian, for example, every package implicitly depends on coreutils. That's perfectly reasonable when installing an OS on a machine: you almost certainly need a working shell to boot and administer your machine. But a container can get by just fine without coreutils, and a typical web server probably (hopefully) doesn't need to call out to a shell. Even if a shell is needed, busybox/toybox is probably sufficient and will take a lot less space. Packages also often contain things like documentation, unit tests, etc. which obviously aren't needed in a container. For Sandstorm.io we deal with this problem by running the app in a mode where we trace all the files it actually uses, and then we build a package containing only those. It mostly works and manages to keep packages reasonably-sized, but it does lead to bugs of the form: "I forgot to test this feature while tracing, so the assets it requires didn't make it into the package." We're looking for better options.

s73v3r 11y ago

You know, as fun as it sounds like this was, wouldn't the natural solution to the problem be to roll back the Docker rollout?

skarap 11y ago

I guess nobody will be saying that Java is lightweight, but the actual results of porting from Java to Go will vary wildly from usecase to usecase. Both examples mentioned in the article (the 175LoC program and the CA) sound like very simple programs. E.g. I once wrote a C program which watched some directories with inotify and compressed new files using zlib. The memory footprint was 350KB. Obviously an empty JVM alone would use 100x more RAM. This static ~30MB overhead might be important in some cases and not relevant at all in others. The incremental (per-object) overhead is probably more relevant to almost all real-world usecases which are a bit more complex then the ones mentioned above. Also - care should be taken not to compare apples (no TM) to oranges: e.g. if you use a huge ORM in Java (which among other things also caches results of each query) and then do a simple SQL query in the new implementation, would be strange to expect those two to perform similar. This happens quite often with rewrites - they almost always aim for simplicity, do short cuts, get rid of "unused stuff". Basically a rewrite from Java to Java will also usually improve performance. Must-read about rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html . Though, I guess, everybody has already read it.

carloscarnero 11y ago

At my organization we have several Java-based services (some of them can even qualify as a form of microservices, if you squint your eyes). We have found that when you have very good developers and you write almost to the letter of the spec, Java can easily provide a stable base (which is probably true of any language/runtime/library). However, we've been eager to try Go in several places. Believe it or not, what has held us back is the lack of a solid LDAP library. We could/should scratch our own itch and be done with it, we lack the time... still so many things to do! In the mean time, for us, Java support for LDAP is nothing short of stellar; and has been for years.

bliti 11y ago

I've been writing Go for some time. Just finished writing a small CDN for the company I currently work for. It's It's a fast language that performs well in systems programming (what it was made for). But boy is it ugly. Not verbose like Java, but ugly to write and read. It does force a good coding convention because otherwise you end up with a pile of ugly code. People seem to be bent on writing it like 80s C. Full of single character variable names and odd function naming. Dunno if it's just my experience. I do like the fact that it resembles python in how it feels. Overall I'd say it's a nice language that is not for everybody. I am playing with Elixir these days. Go is good enougb, but I'm not happy with it being that.

mavelikara 11y ago

I have a question on Go's performance that I'd like someone here, who has experience writing programs in it, to help shed some light upon. 1) Go does not have a runtime. This means that there is no JIT to do any optimizations based on runtime profiling. 2) Go is also designed to compile fast. Since it compiles fast, the compiler's time budget to do compile-time optimizations is small and it probably can't do the best job possible. Are these two points accurate? If so, how does Go perform as well as it is claimed to do? Where is the catch?

flippant 11y ago
1) Go has a runtime that's included in the binary that you get when you run `go build`. It takes care of garbage collection, goroutines etc. It does not do any JIT compilation afaik. 2) Go compiles fast because it gives up a lot of modern features (namely generics).
- deleted 11y ago
  [deleted]

anoother 11y ago

Something troubles me about this article. I hope I'm misunderstanding it. It's stems from the following quotes; knowing this information, why do AeroFS still think Docker is a good fit for their use-case? > However, after our move to Docker, we noticed a sharp increase of the appliance's memory footprint. > ... > We identified several major factors behind these symptoms: > 1. an increase in the number of running JVMs, as each tomcat servlet was placed into a separate container > 2. reduced opportunity for the many JVMs to share read-only memory: the JVM itself, all the shared libraries it depends on, and of course the many JARs used by multiple services > 3. memory isolation could in some cases confuse some sizing heuristics, which lead to larger caches being allocated by some services

ambrice 11y ago
Presumably Docker offers some benefit. It's weird that you and several others have jumped to the conclusion that it's obvious they should get rid of Docker. I would say: knowing this information, why do you still think java is a good fit for their use-case?
- anoother 11y ago
  Because: a) Their service was already written in it b) Their service was already performing adequately before they brought docker into play But those aren't answers to your question. Java may have been a terrible fit for them, but their decision to move away from it was not motivated by that -- it was motivated by their already-written, already-performing Java code no longer performing as well, because of Docker. The only question is, does Docker bring them enough benefits to justify reimplementing large amounts code, in a new and unfamiliar language? Maybe it does, but somehow I doubt it.

djhworld 11y ago

I find this article confusing, in the first part the writer talks about Tomcat and servlets, in the second part he talks about reducing the LOC from 175 lines to 96 by using Go. To me it seems the Java solution was wildly over engineered and probably could have been refactored to not even need Tomcat.

themartorana 11y ago

I am not sure I understand where the issue is Java pure and simple, and where the issue is that Java services were impaired by being wrapped by Docker? "Everything was fine until we switched to Docker" makes me think the trouble may not be all Java. Anyone have educated thoughts on this?

nickpsecurity 11y ago

I appreciate you publishing this experiment as it will show Java's problems for what they are. However, a real test of Go's safety and efficiency would be a comparison to a similar language such as an optimized Wirth language, a subset of Free Pascal (Delphi-style), typed LISP (eg Racket), Julia, or Ada. People keep forgetting about the last one in safe, efficient, systems programming despite it doing for a long time what Go and Rust hope to do eventually. Its long-time use in embedded systems indicates it can be quite efficient. Regardless, Go certainly improves on the reliability of applications vs C++ while being much more efficient than .NET or Java.

tapirl 11y ago

I always think Golang is a Java killer, but I think Java itself is more a Java killer than Golang.