10 ms·

Go kit: Go in the modern enterprise

140 points by sagichmal 12y ago

tekacs 12y ago
This is well written. I gave a talk at the (unaffiliated) Hacker News London Meetup on composing a microservice architecture and why I think a framework to bind today's disparate components needs to exist: http://vimeo.com/118895501 http://vimeo.com/118895501 I don't so much believe that any language is perfectly poised to implement this, however - we're working on a framework[1] that focuses on making every language a first-class participant. Furthermore, it's self-hosting, with most of the core itself implemented as communicating microservices. It's possible to reimplement bits of the core in different languages! We've been waiting to finish a polished implementation before releasing anything (and talking to individual companies about their needs), but I'm starting to think that's a bad idea. :) [1]: http://wym.io/ http://wym.io/
- aikah 12y ago
  What's the difference between a spec and a "multi-language" framework? I mean there is none. You can't have a multi language framework without a strict interface therefore a spec. Now what you provide seems to be the actual framework and just not a spec,which is great, but how is it different from Rest or SOAP, in essence? Also,since languages are vastly different(in features), don't you think it will limit what your framework can do? Great talk anyway. I believe however,that having a central authority that orchestrates services is better than micro services that can talk directly to each other.In my opinion,some kind of mediator is mandatory,one that registers services and manage service communication.
  - tekacs 12y ago
    Addressing each of your points in turn: - What's the difference from a spec? There used to be a FAQ on our site emphasising that at its core _wym_ is a spec. Our reference implementation, itself made up of multiple pieces (wymd, wymsy, oneweb, ...) provides a way to actually make use of that spec and to validate all the guarantees of the strict interfaces. Also, I'm not a fan of entities who would define a spec in isolation without a reference implementation alongside it - where's the proof that you're making the right decisions? We provide that implementation to keep ourselves grounded. - Won't multi-language limit features? Not really. For one thing, since we're self-hosting we can't afford to lack power and expressivity. For example everyone has the ability to refer to types (as per generics) else we wouldn't be able to implement the core. ;) Our interface language - it's quite simple, to make the addition of new language bindings trivial, but we expose built-in services to use more powerful features simply by switching which nodes you're holding on to - there's a data-implementation divide, where a node needn't understand a feature in order to deal in its data. The net effect is that language bindings can be created trivially and evolved to support features as a good way of dealing with them is found (e.g. in the simplest case, Javascript Int64 support). - Having a central authority is better. This is an interesting one :D - I agree, in a way. A wymd cluster self-starts a consensus using a lock service, performs master election and then uses this consensus service for registration and discovery. We have an elected distributed authority. :P
- touristtam 12y ago
  sorry about the off topic, but wym.io w/o js enabled on the page shows this: > function (){ return React.renderComponentToString(Application(null)); } Just looks a bit messy tbh. ;)
  - tekacs 12y ago
    Hm, that's interesting. I went to great lengths to ensure it rendered without JS (since it's all React)! :P I'll look into it. Thanks. :)
    - e12e 12y ago
      Looking at the source-code sent to the browser, I'm afraid there's scant evidence of that. I take it the idea was that the snippet in the body (as mentioned above) was to be rendered on the server? If you figure it out, please let us know/write it up -- being able to use react and have it work without javascript would be a very nice feature.
      - tekacs 12y ago
        Ask and I shall deliver. Fixed. :P I'll upload our build setup right about when I write a blog response to the OP's post. :) The chief point to note is that the components need to render as you want non-JS viewers to see them when they're passed the default props/state.
valevk 12y ago
Golang just needs more database drivers for enterprise products (Informix...), and some libraries that make office products and the windows enironment (active direcory...) more assimilable.
- Zikes 12y ago
  I would really like to have a Golang Teradata driver, however I've thus far made do with routing it through ODBC and using that driver instead. Not ideal, but functional.
- aikah 12y ago
  You mean Go needs libraries for the "enterprise" world, because devs actually need languages that integrate well with their client's infrastructure.And you're totally right.
- CatDevURandom 12y ago
  I completely agree. Active directory. Better database drivers have blocked me from using Go on several projects at work.
wiremine 12y ago
This is a great start. I'm actually in the middle of a writing a Golang service, and this list checks off everything I've been dealing with. A lot of these things aren't golang-specific, though. Ideally it would be easy to swap in services written in different languages, and the conventions stay the same.
knodi 12y ago
The problem is in enterprise Go is really hard to push currently due to management no having proper understanding of it. I believe Go is the future and I recently tried to push Go in using it to write a new semi-real time system to handle many little messages with complex business logic but nope CTO went with node.js and have a callback hell. Sorry didn't mean to rant... Love Go its the future, use it.
- 0xdeadbeefbabe 12y ago
  So why is node, with it's callback hell and single thread, worse? Is it syntax? That single thread assumption can free you from some significant problems, you know. golang's own http lib is event driven and uses callbacks, but perhaps less hellishly. I agree it is hard to push go if you don't properly understand it.
  - frowaway001 12y ago
    Comparing Go and Node is like asking whether someone wants his paster served with vomit or feces. Yes, some people might prefer one over the another, but asking how they ended up with such a ridiculous issue would be a much better question.
  - lebek 12y ago
    Go's http package doesn't use callbacks at the API level. See here http://golang.org/pkg/net/http/ http://golang.org/pkg/net/http/
- codygman 12y ago
  > I believe Go is the future and I recently tried to push Go in using it to write a new semi-real time system to handle many little messages with complex business logic but nope CTO went with node.js and have a callback hell. I used to think and hope Go was the future, and I agree it's better than node.js's callback hell, but I no longer hope Go is the future. Something like Nim[0] is an improvement in every way. Not my main point, but I personally moved from Go to Haskell. So, If you don't mind educating your developers Haskell, Scala, or Ocaml offers you true type safety and compile time correctness. 0: http://nim-lang.org/ http://nim-lang.org/
jbooth 12y ago
Any efforts towards a version-numbering standard for dependencies? Godeps is ok as a workaround, but published versions of things like logging packages, etc, would make me feel a lot safer when depending on them. And no, I'd rather not 'vendor' source-snapshots of a logging package into every project I ever work on.
- threeseed 12y ago
  That's an interesting point. Does Go have an equivalent of Sonatype's Nexus ? This is basically mandatory in most enterprises since often you can't just go into the wild to pickup new dependencies. Plus keeping build artefacts managed is pretty important.
  - StevePerkins 12y ago
    Not really. Moreover, as far as I know there really isn't a good story around shipping dependencies in binary format rather than source code (as Maven repositories do by default). It is fine to assume the availability of source code when you're working on tiny web microservices... and you're only depending on things like Gorilla, or whatever the leading Sinatra-clone is this week. However, a typical "enterprise" project has numerous proprietary dependencies for which the vendor doesn't want to ship source code. It will be interesting to see whether the Go community eventually adapts to fit enterprise culture, whether enterprise culture adapts to fit Go's model, or whether the two worlds never really come together at all.
  - NateDad 12y ago
    Man, that page needs a "What is Nexus?" link. I can't really tell what it is.. I guess some sort of central repo for packages? What go has is Godep (https://github.com/tools/godep https://github.com/tools/godep) which with a single command can copy all your project's dependencies into your repo. from there on out, the dependencies are just part of source control like anything else, so you don't have to open up your build machine to the external internet, for example.
    - threeseed 12y ago
threeseed 12y ago
I wonder if this guy has actually worked in the "enterprise" before because it sure isn't what he is talking about. Companies like Facebook, Soundcloud etc are nothing more than bigger startups. Go work for a bank sometime and then tell me they are comparable to Spotify. As for Go. What it really needs is a great story around Java integration. JDBC drivers and libraries need to just work out of the box. Then you can talk about building cute little micro services. And Scala's toolchain is pretty simple at least with something like Play.
- frowaway001 12y ago
  Agree. I also wonder what this inferiority complex "hurr durr, $X is not written in Go, therefore we need to duplicate it in Go!!!" is about ...
  - slimsag 12y ago
    I don't think it's _always an_ inferiority complex. I've personally run into many situations in which porting a lib to Go is easier -- and more efficient -- than my brain's context switching between multiple languages in order to fix issues with the original lib.
  - Zikes 12y ago
    Look, it's pretty obvious from your comments that you're not a fan of Go, but if you can come to terms with the fact that some people enjoy languages you do not (or their use is mandated by their work environment) then you'll also be able to understand that they would also like certain tools and libraries available to them in that language. It's not about recreating everything in Go "just because", it's about being a Go developer and wanting those tools to be available to you.
    - Retra 12y ago
      This highlights a bigger problem: why should you have to reimplement the same idea hundreds of times if what you already want to do already exists?
krenoten 12y ago
I'd be pretty nervous about any attempt to recreate Finagle in Go. Scala provides many ways of creating composable complexity. Finagle leverages pretty much every feature of Scala (and there are many) to maximize this composability in a sane way, but because there is simply so much complexity in Finagle, a lot of opportunities for composability have been skipped due to lack of need within Twitter (maybe finagle-http has significantly improved in the last year, which is mostly what I'm thinking about when I say this) but it's a LOT of complexity, sort of like the RoR of distributed systems, and when you want to diverge from Twitter's chosen trade-offs, it may be a hellish experience. It's pretty sweet for when your problem space aligns with Twitter's, however. Go has basically none of the mechanisms for sanely creating composable complexity that Scala has. It makes me really grossed out to think about something with the complexity that Finagle exposes as a Go library. Go is all about small, sturdy tools that work for most of the cases most of the time and that compose well with each other - IF - they are built into the standard library. And if it's not, have fun spending your time writing uni-typed interfaces for any kind of interop with desired complexity. Go is great for writing dead simple byte slingers, pipelines that will be rigid in the most dynamic possibility, guns that shoot JS and interact with services, etc. That's simply not Finagle.
- frowaway001 12y ago
  Go has basically none of the mechanisms for sanely creating composable complexity that Scala has. It makes me really grossed out to think about something with the complexity that Finagle exposes as a Go library. Go is all about small, sturdy tools that work for most of the cases most of the time and that compose well with each other - IF - they are built into the standard library. And if it's not, have fun spending your time writing uni-typed interfaces for any kind of interop with desired complexity. Go is great for writing dead simple byte slingers, pipelines that will be rigid in the most dynamic possibility, guns that shoot JS and interact with services, etc. That's simply not Finagle. A thousand times this.
- tekacs 12y ago
  I didn't mention in it my sibling post here, but we started writing the core of wym (our microservices framework) in Go, then Node.js and finally settled on Scala for exactly the reasons you outline. Writing in those other languages was indeed a hellish experience - it was very clear that they lacked the expressivity to model half of even just this very core piece.
deferpanic 12y ago
As for metrics - we've been working precisely on that @ https://deferpanic.com https://deferpanic.com . To some extent logging as well. This means (exactly as OP mentioned) - memory, gc, go-routines, request durations, db query latency, logging errors/panics, etc. We are extremely open to suggestions/feedback. One key point I'd like to point out though, is that OP specifically mentioned the go kit being targeted towards a company with "100–1000+ engineers". I think this is highly on point - if you have less engineers than this you probably don't have enough resources to maintain these sorts of systems yourselves and this is where we think we can help out.
smegel 12y ago
> In this environment, a language like Go has the opportunity to shine. Go has another advantage over Scala (and any other JVM langauge), which also happens to be it's killer feature - green threads, the ability to write non-blocking concurrent code in a synchronous style without callbacks. Yes, I am sure Scala has some "leaky abstraction" over what are essentially just callbacks, as does Java, but it's not quite the same.
- pjmlp 12y ago
  I guess you haven't read the JLS. There isn't any requirement for red threads in certified JVMs and a few of them do offer green threads, which is actually how the first versions used to do threading.
  - smegel 12y ago
    Well I meant that in regard to which JVMs people actually use...not what are theoretically possible.
    - jbooth 12y ago
      Even then, Java's had ExecutorService, Callable<T> and BlockingQueue<T> for a long time. Effectively the same thing as Go's green threads from a programmer's perspective, with the only real difference being that Go swaps out an executing goroutine on I/O or other syscalls (or calling any C code).
      - Matthias247 12y ago
        Newer Go ersions will also preempt green threads on normal functions calls and not only on syscalls. Both is not possible with the JVMs primitives where you need "cooperative code". However there exists Quasar for the JVM which seems to be able to emulate something similar to green threads with the help of bytecode weaving (and inserting preemption points).
        12y ago
qooleot 12y ago
Hey just curious, why did you pick spacemonkeygo/monitor over mozilla-services/heka for data collection? I was thinking of using heka for a logstash-like project and just wanted to see if you had found several know issues, or that monitor really shined, or that maybe the feature set you needed just didn't overlap to the point they were comparable for your needs?
- sagichmal 12y ago
  It wasn't chosen for anything, it was just a related project. And I guess not for its data collection functionality -- that's orthogonal to the goals of the toolkit -- but presumably for the instrumentation and request tracing bits.
bmurphy1976 12y ago
This is great. I hope something like this is built. Go needs it. However, not only did we need something like this 5 years ago, we needed it to be language/platform agnostic then. We need it more so now. We won't be able to function without it 5 years from now. The reality is Pandora's box has been opened there is no going back. We have software we need to run in production at scale that is written in a multitude of languages. We have our Python/Perl monitoring systems talking to our Java search APIs talking to our NodeJS frontend APIs talking to our legacy C#/Java/C++ applications over messaging systems written in Erlang/Ruby/Java/Go. Every single one of these platforms need this functionality and everything needs to run in a cutthroat environment with fewer people, resources, and money to manage it. We need performance, discoverability, security, scalability, deployability, testability, reliability and most importantly EASY. Not enterprisey over-engineered bullshit. I mean real easy to use plug and play, simple APIs, minimal JSON/YAML, small focused tools simple. We need to do it in a way that is portable to every cloud provider so we can get the best bang for our buck and keep the cloud providers honest and avoid lock in. We need to do it in a way that makes the "hardware" vanish. We don't want to manage hardware anymore. We want to deploy our software and we want to know it will work and we don't want to compromise functionality (ala Google App Engine or being forced into the "Tomcat 7" box). I applaud this line of thinking, but I don't think it goes far enough. In 5 years time, I don't think we'll deploy servers anymore. We'll deploy services and those services will be in whatever language and those services will find each other and communicate with each other over the network and will be reliable, secure, and scalable. Most importantly, it needs to be open. Truly open. We don't want to trade one master for another. That's the software I would like to see. This could be a foundational component of something like this, but it's only the beginning. I know about CoreOS/Docker/Mesos/Marathon, etc. Tools like those are an important piece of this puzzle, but not the only piece. They are a vast improvement over past initiatives (cough openstack cough), but they too are only a step in the right direction. We still have a lot of work to do to realize this vision of the future.
- sagichmal 12y ago
  I agree in the broad strokes. But the only way to this utopia is stepwise. I hope this Go kit is one of those steps.
zobzu 12y ago
Sounds like a package manager with s/glibc/go packages/ and s/otherlibs/go packages/ Not necessarily bad but. In fact this is similar to how Go replaces the OS scheduling and message passing. Maybe its a necessary path to evolution. Til the OS itself can get fixed without "it doesnt run X or Z" problems.
telecoda 12y ago
I´m all for in increased uptake of go. But I don want it getting dragged into Enterprise with a big ¨E¨ and going the way of java with its monolithic enterprise frameworks. Somehow we will end up losing the features of the language that make is currently so endearing.
zwieback 12y ago
When we read the word enterprise we probably think of older, slow-moving bureaucracies, like IBM, HP, or even Red Hat. But it’s been a long time since companies like those have been technical leaders. Decades, in some cases. Now, companies like Google, Amazon, Twitter, Netflix, Facebook, Spotify, or even SoundCloud set the tone for our industry. These modern enterprises tend to share a few characteristics... Ouch! I work for HP and while I agree that we're not exactly a nimble, fast moving startup I think it's not a good comparison. IBM and HP still build a lot of physical things while the listed "modern" enterprises are primarily SW. Google would be the exception here but from an engineering point of view Google uses it's vast war chest to play around with very cool HW but isn't much concerned with running a business based on that. So, I hope Go has a good run of it but it's not going to build your next printer.
- enneff 12y ago
  He's talking about software.
codygman 12y ago
A related musing about Go I wrote somewhere else before... Option types as well as lack of support for things like either and generics are why Go quickly became unappealing to me for real world work. Check out this haskell example (note that the exceptions are values as well, so this is more flexible than most languages try or panic in Go): failureExample :: IO (Either SomeException (Response LBS.ByteString)) failureExample = try $ do get "http://www.httpbin.org/status/200" get "http://www.httpbin.org/status/200" get "http://www.httpbin.org/status/200" get "http://www.httpbin.org/status/404" main = failureExample >>= \case Right r -> putStrLn $ "The successful pages status was (spoiler: it's 200!): " ++ show (r ^. responseStatus) Left e -> putStrLn ("error: " ++ show e) The equivalent Go is much longer and needlessly verbose: func failureExample()(*http.Response) { // 1st get, do nothing if success else print exception and exit response, err := http.Get("http://httpbin.org/status/200") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response.Body.Close() } // 2nd get, do nothing if success else print exception and exit response2, err := http.Get("http://httpbin.org/status/200") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response2.Body.Close() } // 3rd get, do nothing if success else print exception and exit response3, err := http.Get("http://httpbin.org/status/200") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response3.Body.Close() } // 4th get, return response if success else print exception and exit response4, err := http.Get("http://httpbin.org/status/404") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response4.Body.Close() } return response4 } func main() { fmt.Println("A failure.") failure := failureExample(); fmt.Println(failure); } This code is all tested/runs and I might be putting this along with other issues I've had in a blog post, but no promises. I think this makes it very apparent why some people have an issue with Go, namely that it feels like going backwards and ignores lots of prior research at the altar of some deluded notion of pragmatism. I say that despite a Go Gopher staring at me on my desk.
- Zikes 12y ago
  I'm not familiar with Haskell so I can't exactly follow what you're trying to do, but I can take a stab at improving what your Go code is doing: package main import ( "fmt" "net/http" ) func main() { for _, url := range []string{ "http://httpbin.org/status/200", "http://httpbin.org/status/200", "http://httpbin.org/status/200", "http://httpbin.org/status/404", } { fmt.Printf("%s: %d\n", url, checkUrl(url)) } } func checkUrl(url string) int { response, err := http.Get(url) if err != nil { panic(err) } defer response.Body.Close() return response.StatusCode } You say of your Haskell code that errors are values, and Go is no different in this regard. checkUrl could easily be modified to return (int, error) and return both values, rather than panicking when an error occurs.
reality_czech 12y ago
Uh oh. The "framework people" found us. Hopefully they'll be too busy writing AbstractFactoryBuilderServiceDispatcherProxies to actually do any damage to production systems...
- enneff 12y ago
  I've talked to the author extensively about these ideas, and let me assure you he is firmly in the "no monolithic frameworks please" camp. (He's also the author of some great Go code, btw.) What he proposes here looks more to me like a standard library for building distributed systems, which makes a lot of sense to me.
  - reality_czech 12y ago
    I don't see the difference between "a standard libary for building systems" and a framework. Just write libraries for what you need and forget the idea of some overarching framework.
    - enneff 12y ago
      When we built the Go standard library it was valuable for us to think of it as a whole. What would most of our users need? Are there any gaps? I think the gokit project is a good opportunity to focus the energies of likeminded people in the Go community, to close the gaps and work together.

140 points by sagichmal 12y ago

This is well written. I gave a talk at the (unaffiliated) Hacker News London Meetup on composing a microservice architecture and why I think a framework to bind today's disparate components needs to exist: http://vimeo.com/118895501 http://vimeo.com/118895501 I don't so much believe that any language is perfectly poised to implement this, however - we're working on a framework[1] that focuses on making every language a first-class participant. Furthermore, it's self-hosting, with most of the core itself implemented as communicating microservices. It's possible to reimplement bits of the core in different languages! We've been waiting to finish a polished implementation before releasing anything (and talking to individual companies about their needs), but I'm starting to think that's a bad idea. :) [1]: http://wym.io/ http://wym.io/

aikah 12y ago
What's the difference between a spec and a "multi-language" framework? I mean there is none. You can't have a multi language framework without a strict interface therefore a spec. Now what you provide seems to be the actual framework and just not a spec,which is great, but how is it different from Rest or SOAP, in essence? Also,since languages are vastly different(in features), don't you think it will limit what your framework can do? Great talk anyway. I believe however,that having a central authority that orchestrates services is better than micro services that can talk directly to each other.In my opinion,some kind of mediator is mandatory,one that registers services and manage service communication.
- tekacs 12y ago
  Addressing each of your points in turn: - What's the difference from a spec? There used to be a FAQ on our site emphasising that at its core _wym_ is a spec. Our reference implementation, itself made up of multiple pieces (wymd, wymsy, oneweb, ...) provides a way to actually make use of that spec and to validate all the guarantees of the strict interfaces. Also, I'm not a fan of entities who would define a spec in isolation without a reference implementation alongside it - where's the proof that you're making the right decisions? We provide that implementation to keep ourselves grounded. - Won't multi-language limit features? Not really. For one thing, since we're self-hosting we can't afford to lack power and expressivity. For example everyone has the ability to refer to types (as per generics) else we wouldn't be able to implement the core. ;) Our interface language - it's quite simple, to make the addition of new language bindings trivial, but we expose built-in services to use more powerful features simply by switching which nodes you're holding on to - there's a data-implementation divide, where a node needn't understand a feature in order to deal in its data. The net effect is that language bindings can be created trivially and evolved to support features as a good way of dealing with them is found (e.g. in the simplest case, Javascript Int64 support). - Having a central authority is better. This is an interesting one :D - I agree, in a way. A wymd cluster self-starts a consensus using a lock service, performs master election and then uses this consensus service for registration and discovery. We have an elected distributed authority. :P
touristtam 12y ago
sorry about the off topic, but wym.io w/o js enabled on the page shows this: > function (){ return React.renderComponentToString(Application(null)); } Just looks a bit messy tbh. ;)
- tekacs 12y ago
  Hm, that's interesting. I went to great lengths to ensure it rendered without JS (since it's all React)! :P I'll look into it. Thanks. :)
  - e12e 12y ago
    Looking at the source-code sent to the browser, I'm afraid there's scant evidence of that. I take it the idea was that the snippet in the body (as mentioned above) was to be rendered on the server? If you figure it out, please let us know/write it up -- being able to use react and have it work without javascript would be a very nice feature.
    - tekacs 12y ago
      Ask and I shall deliver. Fixed. :P I'll upload our build setup right about when I write a blog response to the OP's post. :) The chief point to note is that the components need to render as you want non-JS viewers to see them when they're passed the default props/state.

valevk 12y ago

Golang just needs more database drivers for enterprise products (Informix...), and some libraries that make office products and the windows enironment (active direcory...) more assimilable.

Zikes 12y ago
I would really like to have a Golang Teradata driver, however I've thus far made do with routing it through ODBC and using that driver instead. Not ideal, but functional.
aikah 12y ago
You mean Go needs libraries for the "enterprise" world, because devs actually need languages that integrate well with their client's infrastructure.And you're totally right.
CatDevURandom 12y ago
I completely agree. Active directory. Better database drivers have blocked me from using Go on several projects at work.

wiremine 12y ago

This is a great start. I'm actually in the middle of a writing a Golang service, and this list checks off everything I've been dealing with. A lot of these things aren't golang-specific, though. Ideally it would be easy to swap in services written in different languages, and the conventions stay the same.

knodi 12y ago

The problem is in enterprise Go is really hard to push currently due to management no having proper understanding of it. I believe Go is the future and I recently tried to push Go in using it to write a new semi-real time system to handle many little messages with complex business logic but nope CTO went with node.js and have a callback hell. Sorry didn't mean to rant... Love Go its the future, use it.

0xdeadbeefbabe 12y ago
So why is node, with it's callback hell and single thread, worse? Is it syntax? That single thread assumption can free you from some significant problems, you know. golang's own http lib is event driven and uses callbacks, but perhaps less hellishly. I agree it is hard to push go if you don't properly understand it.
- frowaway001 12y ago
  Comparing Go and Node is like asking whether someone wants his paster served with vomit or feces. Yes, some people might prefer one over the another, but asking how they ended up with such a ridiculous issue would be a much better question.
- lebek 12y ago
  Go's http package doesn't use callbacks at the API level. See here http://golang.org/pkg/net/http/ http://golang.org/pkg/net/http/
codygman 12y ago
> I believe Go is the future and I recently tried to push Go in using it to write a new semi-real time system to handle many little messages with complex business logic but nope CTO went with node.js and have a callback hell. I used to think and hope Go was the future, and I agree it's better than node.js's callback hell, but I no longer hope Go is the future. Something like Nim[0] is an improvement in every way. Not my main point, but I personally moved from Go to Haskell. So, If you don't mind educating your developers Haskell, Scala, or Ocaml offers you true type safety and compile time correctness. 0: http://nim-lang.org/ http://nim-lang.org/

jbooth 12y ago

Any efforts towards a version-numbering standard for dependencies? Godeps is ok as a workaround, but published versions of things like logging packages, etc, would make me feel a lot safer when depending on them. And no, I'd rather not 'vendor' source-snapshots of a logging package into every project I ever work on.

threeseed 12y ago
That's an interesting point. Does Go have an equivalent of Sonatype's Nexus ? This is basically mandatory in most enterprises since often you can't just go into the wild to pickup new dependencies. Plus keeping build artefacts managed is pretty important.
- StevePerkins 12y ago
  Not really. Moreover, as far as I know there really isn't a good story around shipping dependencies in binary format rather than source code (as Maven repositories do by default). It is fine to assume the availability of source code when you're working on tiny web microservices... and you're only depending on things like Gorilla, or whatever the leading Sinatra-clone is this week. However, a typical "enterprise" project has numerous proprietary dependencies for which the vendor doesn't want to ship source code. It will be interesting to see whether the Go community eventually adapts to fit enterprise culture, whether enterprise culture adapts to fit Go's model, or whether the two worlds never really come together at all.
- NateDad 12y ago
  Man, that page needs a "What is Nexus?" link. I can't really tell what it is.. I guess some sort of central repo for packages? What go has is Godep (https://github.com/tools/godep https://github.com/tools/godep) which with a single command can copy all your project's dependencies into your repo. from there on out, the dependencies are just part of source control like anything else, so you don't have to open up your build machine to the external internet, for example.
  - threeseed 12y ago

threeseed 12y ago

I wonder if this guy has actually worked in the "enterprise" before because it sure isn't what he is talking about. Companies like Facebook, Soundcloud etc are nothing more than bigger startups. Go work for a bank sometime and then tell me they are comparable to Spotify. As for Go. What it really needs is a great story around Java integration. JDBC drivers and libraries need to just work out of the box. Then you can talk about building cute little micro services. And Scala's toolchain is pretty simple at least with something like Play.

frowaway001 12y ago
Agree. I also wonder what this inferiority complex "hurr durr, $X is not written in Go, therefore we need to duplicate it in Go!!!" is about ...
- slimsag 12y ago
  I don't think it's _always an_ inferiority complex. I've personally run into many situations in which porting a lib to Go is easier -- and more efficient -- than my brain's context switching between multiple languages in order to fix issues with the original lib.
- Zikes 12y ago
  Look, it's pretty obvious from your comments that you're not a fan of Go, but if you can come to terms with the fact that some people enjoy languages you do not (or their use is mandated by their work environment) then you'll also be able to understand that they would also like certain tools and libraries available to them in that language. It's not about recreating everything in Go "just because", it's about being a Go developer and wanting those tools to be available to you.
  - Retra 12y ago
    This highlights a bigger problem: why should you have to reimplement the same idea hundreds of times if what you already want to do already exists?

krenoten 12y ago

I'd be pretty nervous about any attempt to recreate Finagle in Go. Scala provides many ways of creating composable complexity. Finagle leverages pretty much every feature of Scala (and there are many) to maximize this composability in a sane way, but because there is simply so much complexity in Finagle, a lot of opportunities for composability have been skipped due to lack of need within Twitter (maybe finagle-http has significantly improved in the last year, which is mostly what I'm thinking about when I say this) but it's a LOT of complexity, sort of like the RoR of distributed systems, and when you want to diverge from Twitter's chosen trade-offs, it may be a hellish experience. It's pretty sweet for when your problem space aligns with Twitter's, however. Go has basically none of the mechanisms for sanely creating composable complexity that Scala has. It makes me really grossed out to think about something with the complexity that Finagle exposes as a Go library. Go is all about small, sturdy tools that work for most of the cases most of the time and that compose well with each other - IF - they are built into the standard library. And if it's not, have fun spending your time writing uni-typed interfaces for any kind of interop with desired complexity. Go is great for writing dead simple byte slingers, pipelines that will be rigid in the most dynamic possibility, guns that shoot JS and interact with services, etc. That's simply not Finagle.

frowaway001 12y ago
Go has basically none of the mechanisms for sanely creating composable complexity that Scala has. It makes me really grossed out to think about something with the complexity that Finagle exposes as a Go library. Go is all about small, sturdy tools that work for most of the cases most of the time and that compose well with each other - IF - they are built into the standard library. And if it's not, have fun spending your time writing uni-typed interfaces for any kind of interop with desired complexity. Go is great for writing dead simple byte slingers, pipelines that will be rigid in the most dynamic possibility, guns that shoot JS and interact with services, etc. That's simply not Finagle. A thousand times this.
tekacs 12y ago
I didn't mention in it my sibling post here, but we started writing the core of wym (our microservices framework) in Go, then Node.js and finally settled on Scala for exactly the reasons you outline. Writing in those other languages was indeed a hellish experience - it was very clear that they lacked the expressivity to model half of even just this very core piece.

deferpanic 12y ago

As for metrics - we've been working precisely on that @ https://deferpanic.com https://deferpanic.com . To some extent logging as well. This means (exactly as OP mentioned) - memory, gc, go-routines, request durations, db query latency, logging errors/panics, etc. We are extremely open to suggestions/feedback. One key point I'd like to point out though, is that OP specifically mentioned the go kit being targeted towards a company with "100–1000+ engineers". I think this is highly on point - if you have less engineers than this you probably don't have enough resources to maintain these sorts of systems yourselves and this is where we think we can help out.

smegel 12y ago

> In this environment, a language like Go has the opportunity to shine. Go has another advantage over Scala (and any other JVM langauge), which also happens to be it's killer feature - green threads, the ability to write non-blocking concurrent code in a synchronous style without callbacks. Yes, I am sure Scala has some "leaky abstraction" over what are essentially just callbacks, as does Java, but it's not quite the same.

pjmlp 12y ago
I guess you haven't read the JLS. There isn't any requirement for red threads in certified JVMs and a few of them do offer green threads, which is actually how the first versions used to do threading.
- smegel 12y ago
  Well I meant that in regard to which JVMs people actually use...not what are theoretically possible.
  - jbooth 12y ago
    Even then, Java's had ExecutorService, Callable<T> and BlockingQueue<T> for a long time. Effectively the same thing as Go's green threads from a programmer's perspective, with the only real difference being that Go swaps out an executing goroutine on I/O or other syscalls (or calling any C code).
    - Matthias247 12y ago
      Newer Go ersions will also preempt green threads on normal functions calls and not only on syscalls. Both is not possible with the JVMs primitives where you need "cooperative code". However there exists Quasar for the JVM which seems to be able to emulate something similar to green threads with the help of bytecode weaving (and inserting preemption points).
      - 12y ago

qooleot 12y ago

Hey just curious, why did you pick spacemonkeygo/monitor over mozilla-services/heka for data collection? I was thinking of using heka for a logstash-like project and just wanted to see if you had found several know issues, or that monitor really shined, or that maybe the feature set you needed just didn't overlap to the point they were comparable for your needs?

sagichmal 12y ago
It wasn't chosen for anything, it was just a related project. And I guess not for its data collection functionality -- that's orthogonal to the goals of the toolkit -- but presumably for the instrumentation and request tracing bits.

bmurphy1976 12y ago

This is great. I hope something like this is built. Go needs it. However, not only did we need something like this 5 years ago, we needed it to be language/platform agnostic then. We need it more so now. We won't be able to function without it 5 years from now. The reality is Pandora's box has been opened there is no going back. We have software we need to run in production at scale that is written in a multitude of languages. We have our Python/Perl monitoring systems talking to our Java search APIs talking to our NodeJS frontend APIs talking to our legacy C#/Java/C++ applications over messaging systems written in Erlang/Ruby/Java/Go. Every single one of these platforms need this functionality and everything needs to run in a cutthroat environment with fewer people, resources, and money to manage it. We need performance, discoverability, security, scalability, deployability, testability, reliability and most importantly EASY. Not enterprisey over-engineered bullshit. I mean real easy to use plug and play, simple APIs, minimal JSON/YAML, small focused tools simple. We need to do it in a way that is portable to every cloud provider so we can get the best bang for our buck and keep the cloud providers honest and avoid lock in. We need to do it in a way that makes the "hardware" vanish. We don't want to manage hardware anymore. We want to deploy our software and we want to know it will work and we don't want to compromise functionality (ala Google App Engine or being forced into the "Tomcat 7" box). I applaud this line of thinking, but I don't think it goes far enough. In 5 years time, I don't think we'll deploy servers anymore. We'll deploy services and those services will be in whatever language and those services will find each other and communicate with each other over the network and will be reliable, secure, and scalable. Most importantly, it needs to be open. Truly open. We don't want to trade one master for another. That's the software I would like to see. This could be a foundational component of something like this, but it's only the beginning. I know about CoreOS/Docker/Mesos/Marathon, etc. Tools like those are an important piece of this puzzle, but not the only piece. They are a vast improvement over past initiatives (cough openstack cough), but they too are only a step in the right direction. We still have a lot of work to do to realize this vision of the future.

sagichmal 12y ago
I agree in the broad strokes. But the only way to this utopia is stepwise. I hope this Go kit is one of those steps.

zobzu 12y ago

Sounds like a package manager with s/glibc/go packages/ and s/otherlibs/go packages/ Not necessarily bad but. In fact this is similar to how Go replaces the OS scheduling and message passing. Maybe its a necessary path to evolution. Til the OS itself can get fixed without "it doesnt run X or Z" problems.

telecoda 12y ago

I´m all for in increased uptake of go. But I don want it getting dragged into Enterprise with a big ¨E¨ and going the way of java with its monolithic enterprise frameworks. Somehow we will end up losing the features of the language that make is currently so endearing.

zwieback 12y ago

When we read the word enterprise we probably think of older, slow-moving bureaucracies, like IBM, HP, or even Red Hat. But it’s been a long time since companies like those have been technical leaders. Decades, in some cases. Now, companies like Google, Amazon, Twitter, Netflix, Facebook, Spotify, or even SoundCloud set the tone for our industry. These modern enterprises tend to share a few characteristics... Ouch! I work for HP and while I agree that we're not exactly a nimble, fast moving startup I think it's not a good comparison. IBM and HP still build a lot of physical things while the listed "modern" enterprises are primarily SW. Google would be the exception here but from an engineering point of view Google uses it's vast war chest to play around with very cool HW but isn't much concerned with running a business based on that. So, I hope Go has a good run of it but it's not going to build your next printer.

enneff 12y ago
He's talking about software.

codygman 12y ago

A related musing about Go I wrote somewhere else before... Option types as well as lack of support for things like either and generics are why Go quickly became unappealing to me for real world work. Check out this haskell example (note that the exceptions are values as well, so this is more flexible than most languages try or panic in Go): failureExample :: IO (Either SomeException (Response LBS.ByteString)) failureExample = try $ do get "http://www.httpbin.org/status/200" get "http://www.httpbin.org/status/200" get "http://www.httpbin.org/status/200" get "http://www.httpbin.org/status/404" main = failureExample >>= \case Right r -> putStrLn $ "The successful pages status was (spoiler: it's 200!): " ++ show (r ^. responseStatus) Left e -> putStrLn ("error: " ++ show e) The equivalent Go is much longer and needlessly verbose: func failureExample()(*http.Response) { // 1st get, do nothing if success else print exception and exit response, err := http.Get("http://httpbin.org/status/200") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response.Body.Close() } // 2nd get, do nothing if success else print exception and exit response2, err := http.Get("http://httpbin.org/status/200") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response2.Body.Close() } // 3rd get, do nothing if success else print exception and exit response3, err := http.Get("http://httpbin.org/status/200") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response3.Body.Close() } // 4th get, return response if success else print exception and exit response4, err := http.Get("http://httpbin.org/status/404") if err != nil { fmt.Printf("%s", err) os.Exit(1) } else { defer response4.Body.Close() } return response4 } func main() { fmt.Println("A failure.") failure := failureExample(); fmt.Println(failure); } This code is all tested/runs and I might be putting this along with other issues I've had in a blog post, but no promises. I think this makes it very apparent why some people have an issue with Go, namely that it feels like going backwards and ignores lots of prior research at the altar of some deluded notion of pragmatism. I say that despite a Go Gopher staring at me on my desk.

Zikes 12y ago
I'm not familiar with Haskell so I can't exactly follow what you're trying to do, but I can take a stab at improving what your Go code is doing: package main import ( "fmt" "net/http" ) func main() { for _, url := range []string{ "http://httpbin.org/status/200", "http://httpbin.org/status/200", "http://httpbin.org/status/200", "http://httpbin.org/status/404", } { fmt.Printf("%s: %d\n", url, checkUrl(url)) } } func checkUrl(url string) int { response, err := http.Get(url) if err != nil { panic(err) } defer response.Body.Close() return response.StatusCode } You say of your Haskell code that errors are values, and Go is no different in this regard. checkUrl could easily be modified to return (int, error) and return both values, rather than panicking when an error occurs.

reality_czech 12y ago

Uh oh. The "framework people" found us. Hopefully they'll be too busy writing AbstractFactoryBuilderServiceDispatcherProxies to actually do any damage to production systems...

enneff 12y ago
I've talked to the author extensively about these ideas, and let me assure you he is firmly in the "no monolithic frameworks please" camp. (He's also the author of some great Go code, btw.) What he proposes here looks more to me like a standard library for building distributed systems, which makes a lot of sense to me.
- reality_czech 12y ago
  I don't see the difference between "a standard libary for building systems" and a framework. Just write libraries for what you need and forget the idea of some overarching framework.
  - enneff 12y ago
    When we built the Go standard library it was valuable for us to think of it as a whole. What would most of our users need? Are there any gaps? I think the gokit project is a good opportunity to focus the energies of likeminded people in the Go community, to close the gaps and work together.