10 ms·

Choosing Go at American Express (2019)

62 points by veqq 3y ago

sgt 3y ago
Whether they had chosen Go or Java, or anything else, what I see here is an internal culture that really takes ownership of their language choice. That is its weight worth in gold. That is what matters in the end ("can we ship?") - not potential issues with the language itself.
bob1029 3y ago
This is from 2019. I'd be curious how all of this worked out for AMEX. We've got over 4 years behind this decision now. Edit: I found a followup from 2021 - https://youtu.be/e7PtBOsTpXE https://youtu.be/e7PtBOsTpXE
evanm 3y ago
Curious what other languages were evaluated.
- vlod 3y ago
  From [0] "Based on these criteria, we narrowed our testing languages to C++17, Java, Node.js, and Go. The first three languages were already in use at American Express, whereas Go was not." [0]: https://americanexpress.io/choosing-go/ https://americanexpress.io/choosing-go/
Ancalagon 3y ago
I actually worked with Ben for a couple of my formative years as a software engineer - he's an extremely smart guy. I'm no longer with Amex (still up for debate in my mind if that was a good idea to leave or not), but last I heard the department was very successful in onboarding a lot of their Java engineers to Go and transitioning to a Golang microservice architecture.
- kylehotchkiss 3y ago
  I find as an end user/consumer that Amex has one of the best web/app experiences of all the traditional financial companies I've interacted with. How did it feel working on it? I get the sense a lot of banking software is makeup on an ancient mainframe but not with Amex. Was it a fairly good environment for developers?
  - bob1029 3y ago
    Are you sure that isn't the case for Amex as well? I recall being forced to use a very particular set of legal characters when defining my password for use with Amex online account services. The exact same constraints I have with several other financial institutions. I suspect that somewhere in the heart of the beast there still lurks an IBM-branded system of record. Just because a machine from the 90s is running the database doesn't mean you can't put a webGL frontend on top of it.
    - cempaka 3y ago
      I used to write for a service that submitted transactions to AmexDirect. It's still done in EBCDIC, an encoding which famously led to a lawsuit against a European bank by a customer who couldn't create an account with his real name.
  - Ancalagon 3y ago
    Amex is probably the one company of my three previous jobs I'd gladly work for again.
matrix87 3y ago
After using both java and go, I prefer go. The tooling is simpler and more intuitive, the feature set is less bloated (fewer useless things reinventing the wheel like streams api), it also isn't affiliated with Oracle In general java feels more enshittified in comparison
- CobrastanJorji 3y ago
  As someone who's got a lot of Java experience but almost no Go experience, I worry that I've been coddled with a humongous standard library for pretty much everything, from collections of every sort to handling of time, units, strings, I/O buffers, and pretty much every data structure I might need. Do you find that the Go ecosystem is rich enough that you can find whatever you require?
  - rlawson 3y ago
    Yes I had the exact same question. Philosophically I like Go better but I worry in practice that Java has me spoiled
    - rad_gruchalski 3y ago
      The Go ecosystem has everything one needs. The cryptography stuff is nicer than any Java library (sorry BouncyCastle), first class tls everywhere, very decent http client/server support and tons community routers, third-party collections are easy to come by, ssh in standard library, any package from the internet can be directly pulled as a dependency, packages can be renamed without the need of shadowing, kafka clients, any cloud sdk, everything what one can imagine. I was a heavy java snd scala user before 2018 and 5 years later I’m not missing anything. For what it’s worth, life got easier when I moved to Go.
  - holografix 3y ago
    Go introduced generics as of recently therefore you won’t find many of the more sophisticated collection types like trees and sorted lists etc. However there is an effort to include the usual suspects in the standard lib.
7e 3y ago
How did they ensure their service benchmark wasn’t I/O bound?
dang 3y ago
Discussed at the time: Choosing Go at American Express - https://news.ycombinator.com/item?id=21637916 https://news.ycombinator.com/item?id=21637916 - Nov 2019 (15 comments)
deleted 3y ago
[deleted]
EdwardDiego 3y ago
Go is a decent industrial programming language alternative to Java, only things I'm not super enthused about is the logging experience (for the love of Christ, please create a standardised logging facade along the lines of slf4j, and allow the person running the application to set logging levels for individual loggers and choose whether stack-traces are emitted or not! Bonus points if I can change logging conf for a running app without restarting it) and libs that use C bindings, I have enough problems with that in Python already. E.g., my manager is really interested in Dapr as a way to make basic Kafka usage easier for teams across our org. Dapr's written in Go, and have made the reasonable decision to not use any libs that have C bindings for the same reason I don't like them. However, this means that they can't use the Confluent Go Kafka client, as it's just a wrapper around librdkafka. So they use Sarama, as it's written in pure Go, and is widely used, but is one of the most problematic Kafka clients I've had the misfortune to support. Although now that IBM has taken ownership of the project from Shopify, some of the more egregious bugs and oddities are finally being fixed. But yeah, while you _can_ bind to C libs in Java, I've never encountered a library that did, so cross-platform lib compatibility wasn't something I ever had to worry about when working with Java.
- liampulles 3y ago
  You may rejoice on the logging front: https://go.dev/blog/slog https://go.dev/blog/slog
  - aaomidi 3y ago
    This is nice but it’s not a simple library to use and I think that’s going to hurt it.
    - i_am_programmer 3y ago
      It's as easy to use as other 3rd party logging libs. I frequently make use and it's as easy as slog.Info("text", "field", value, "field", value)
    - jen20 3y ago
aslilac 3y ago
not to be the “but rust” person, but I’m curious why Rust wasn’t in the test group. it seems to meet all their criteria, though the community was a bit smaller back then. Go is a perfectly reasonable choice though, and easier to learn on average.
- steveklabnik 3y ago
  https://news.ycombinator.com/item?id=21638455 https://news.ycombinator.com/item?id=21638455
- greatgib 3y ago
  Python is not in the list also, despite them using nodejs. But when I read the following: Based on these criteria, we narrowed our testing languages to C++17, Java, Node.js, and Go. The first three languages were already in use at American Express, whereas Go was not. What I understand is that they did not really wanted to do a real benchmark but they decided to use Go first, and then they had to setup a kind of "bullshit neutral study" to pretend to leadership that Go was the best choice for all the languages that could fit the job. Maybe it is the case in the end for them, but it is another subject.
  - sturmdev 3y ago
    My understanding is that JS running on Node is insanely fast compared to most other interpreted languages. This is because of how much work has been put into the V8 engine, especially the JIT compiler. Based on this, it’s fair to put Node into a league above languages like Python and Ruby when considering performance. This might have played a role in their decision to include Node.
- aaomidi 3y ago
  Rust crypto libraries were definitely not ready in 2019. I think the story is a bit better today.
liampulles 3y ago
I love how stubbornly plain and lacking in magic Go is. There is almost no hidden control flow, there's no aspect oriented programming, or operator overloading, etc., etc. - pretty much everything the code actually does is in front of you. I think this means you have a bigger upfront cost in writing code the first time, with the benefit of easier debugging and easy maintenance later (which is a tradeoff I'm very happy to take).
- aaomidi 3y ago
  TBH go does have a lot of weird magic stuff. For example: - types vs type aliases - the init function - how you define a struct can have significant impact on performance. There are lints that align struct better. - changing a returned value in a defer function. Very useful but can easily hide complexity. - interfaces are.. weird. They can also easily hide weirdness.
  - richbell 3y ago
    Build directives are magic comments. Embed is a magic comment but you also need to import it. Weird magic syntax to declare enums. Etc.
  - liampulles 3y ago
    I was not very explicit on what magic means to me (so fair do's). By magic I mean logic of a program which is not plain function calls, conditionals, etc. or unexpected/hidden logic. In that light, I agree init calls and defers are a bit magic. I'll even add panics in there. But I don't use these much personally, so I'm hesitant to say it is a lot of magic - especially compared to something like Java for example.
wizwit999 3y ago
I say this as a personal rust fan, but practically if a language like a combination of Kotlin+Go existed, that'd be like an awesome standard business language. Kotlin is good but just very tied to the JVM, Go has a hostile syntax/developer experience. Another way to frame it is like Rust with a garbage collector. Most biz APIs don't really need zero cost abstractions of Rust.
- mdwagner 3y ago
  If you're interested, take a look at Crystal (https://crystal-lang.org/ https://crystal-lang.org/)!
- bmikaili 3y ago
  This is Swift. It‘s just not being used a lot outside of iOS / Macos dev.
- cempaka 3y ago
  I see OCaml described as "Rust with a GC" pretty often, but maybe it's too heavily functional for what you're seeking.
- Shakahs 3y ago
  Is being tied to the JVM that much of a problem these days? Developer workstations and servers have much more compute and RAM these days, and native compilation with GraalVM seems to eliminate high JVM startup times and memory overhead. Quarkus for example is known for a snappy developer experience.
itsdrewmiller 3y ago
This seems a lot more like someone deciding on Go up front and then pushing a process through to pick it - it wasn't the fastest in their tests and it was the only language without existing usage at the company, and basically all the qualitative advantages are either not very unique or not quantifiable or both. No other new languages besides Go were even considered, despite some that easily meet the criteria for consideration.
- officialchicken 3y ago
  Oracle's lawsuit against Google's use of Java API's started in 2019[0]. The writing was on the wall; many Java shops started looking at other options. [0] https://en.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_Inc https://en.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_....
  - scarface_74 3y ago
    The lawsuit started in 2009 https://en.m.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_Inc https://en.m.wikipedia.org/wiki/Google_LLC_v._Oracle_America....
  - Larrikin 3y ago
    So if you're a Java shop you pick Scala because it seems amazing until you read about real world usage/find out for yourself, and then you pick Kotlin.
- bob1029 3y ago
  If you forced me to conduct best language olympics, the process I would use might look something like this: 1. Each team develops the same vertical slice of the business in their ecosystem of choice. 2. Test officials will make an edit to each codebase (somehow invisible to git history) that introduces a subtle bug. Alternatively, they may propose a small feature enhancement. 3. Time how long it takes each team to get a corrected/updated build deployed & re-tested successfully. 4. Go to 2 until statisticians are satisfied with the results.
  - 3y ago
xyst 3y ago
Lately I have been more interested in rust. Similar trade offs but “fearless concurrency”, access to low level APIs, and no garbage collector sold it for me.
aaomidi 3y ago
I need Go to take one lesson from rust and that’s specifying if a function will mutate values passed through it or not. This can technically be done as a breaking change spread over several versions of go and I think the cost would be worth what we’d get out of it. Utilities can also be made to edit code to support this automatically without having to have some user go fix it all. Basically, I want an introduction of either a function level or value level ‘nut’ keyword like Rust where the function can not mutate a field unless it has explicitly stated mut. This would fundamentally solve a whole class of bugs and errors that happen in go where some library ends up modifying a struct (pointer or a struct with a pointer field) without the caller intending that.
- aatd86 3y ago
  I'm not sure I fully understand what you're proposing? Is it about function purity or immutable values? I think the way to do the latter in Go is to not export the field and only have it modifiable by a dedicated method. (or even enforce copy-on-write). And passing values instead of pointers where needed. I don't think that breaking compatibility to add a mut keyword would fit Go. It makes more sense for rust. (especially wrt interior mutability... That would make it a very pervasive change, probably too much of a change) Another thing is that wrapping values in an empty interface essentially make those immutable. It doesn't necessarily have mechanical sympathy but as long as you don't wrap pointers, it would be difficult to modify (no aliasing). Then again, it's more defensive and pragmatic coding than something that proves functional purity. Seems that it is enough.

62 points by veqq 3y ago

sgt 3y ago

Whether they had chosen Go or Java, or anything else, what I see here is an internal culture that really takes ownership of their language choice. That is its weight worth in gold. That is what matters in the end ("can we ship?") - not potential issues with the language itself.

bob1029 3y ago

This is from 2019. I'd be curious how all of this worked out for AMEX. We've got over 4 years behind this decision now. Edit: I found a followup from 2021 - https://youtu.be/e7PtBOsTpXE https://youtu.be/e7PtBOsTpXE

evanm 3y ago

Curious what other languages were evaluated.

vlod 3y ago
From [0] "Based on these criteria, we narrowed our testing languages to C++17, Java, Node.js, and Go. The first three languages were already in use at American Express, whereas Go was not." [0]: https://americanexpress.io/choosing-go/ https://americanexpress.io/choosing-go/

Ancalagon 3y ago

I actually worked with Ben for a couple of my formative years as a software engineer - he's an extremely smart guy. I'm no longer with Amex (still up for debate in my mind if that was a good idea to leave or not), but last I heard the department was very successful in onboarding a lot of their Java engineers to Go and transitioning to a Golang microservice architecture.

kylehotchkiss 3y ago
I find as an end user/consumer that Amex has one of the best web/app experiences of all the traditional financial companies I've interacted with. How did it feel working on it? I get the sense a lot of banking software is makeup on an ancient mainframe but not with Amex. Was it a fairly good environment for developers?
- bob1029 3y ago
  Are you sure that isn't the case for Amex as well? I recall being forced to use a very particular set of legal characters when defining my password for use with Amex online account services. The exact same constraints I have with several other financial institutions. I suspect that somewhere in the heart of the beast there still lurks an IBM-branded system of record. Just because a machine from the 90s is running the database doesn't mean you can't put a webGL frontend on top of it.
  - cempaka 3y ago
    I used to write for a service that submitted transactions to AmexDirect. It's still done in EBCDIC, an encoding which famously led to a lawsuit against a European bank by a customer who couldn't create an account with his real name.
- Ancalagon 3y ago
  Amex is probably the one company of my three previous jobs I'd gladly work for again.

matrix87 3y ago

After using both java and go, I prefer go. The tooling is simpler and more intuitive, the feature set is less bloated (fewer useless things reinventing the wheel like streams api), it also isn't affiliated with Oracle In general java feels more enshittified in comparison

CobrastanJorji 3y ago
As someone who's got a lot of Java experience but almost no Go experience, I worry that I've been coddled with a humongous standard library for pretty much everything, from collections of every sort to handling of time, units, strings, I/O buffers, and pretty much every data structure I might need. Do you find that the Go ecosystem is rich enough that you can find whatever you require?
- rlawson 3y ago
  Yes I had the exact same question. Philosophically I like Go better but I worry in practice that Java has me spoiled
  - rad_gruchalski 3y ago
    The Go ecosystem has everything one needs. The cryptography stuff is nicer than any Java library (sorry BouncyCastle), first class tls everywhere, very decent http client/server support and tons community routers, third-party collections are easy to come by, ssh in standard library, any package from the internet can be directly pulled as a dependency, packages can be renamed without the need of shadowing, kafka clients, any cloud sdk, everything what one can imagine. I was a heavy java snd scala user before 2018 and 5 years later I’m not missing anything. For what it’s worth, life got easier when I moved to Go.
- holografix 3y ago
  Go introduced generics as of recently therefore you won’t find many of the more sophisticated collection types like trees and sorted lists etc. However there is an effort to include the usual suspects in the standard lib.

7e 3y ago

How did they ensure their service benchmark wasn’t I/O bound?

dang 3y ago

Discussed at the time: Choosing Go at American Express - https://news.ycombinator.com/item?id=21637916 https://news.ycombinator.com/item?id=21637916 - Nov 2019 (15 comments)

deleted 3y ago

[deleted]

EdwardDiego 3y ago

Go is a decent industrial programming language alternative to Java, only things I'm not super enthused about is the logging experience (for the love of Christ, please create a standardised logging facade along the lines of slf4j, and allow the person running the application to set logging levels for individual loggers and choose whether stack-traces are emitted or not! Bonus points if I can change logging conf for a running app without restarting it) and libs that use C bindings, I have enough problems with that in Python already. E.g., my manager is really interested in Dapr as a way to make basic Kafka usage easier for teams across our org. Dapr's written in Go, and have made the reasonable decision to not use any libs that have C bindings for the same reason I don't like them. However, this means that they can't use the Confluent Go Kafka client, as it's just a wrapper around librdkafka. So they use Sarama, as it's written in pure Go, and is widely used, but is one of the most problematic Kafka clients I've had the misfortune to support. Although now that IBM has taken ownership of the project from Shopify, some of the more egregious bugs and oddities are finally being fixed. But yeah, while you _can_ bind to C libs in Java, I've never encountered a library that did, so cross-platform lib compatibility wasn't something I ever had to worry about when working with Java.

liampulles 3y ago
You may rejoice on the logging front: https://go.dev/blog/slog https://go.dev/blog/slog
- aaomidi 3y ago
  This is nice but it’s not a simple library to use and I think that’s going to hurt it.
  - i_am_programmer 3y ago
    It's as easy to use as other 3rd party logging libs. I frequently make use and it's as easy as slog.Info("text", "field", value, "field", value)
  - jen20 3y ago

aslilac 3y ago

not to be the “but rust” person, but I’m curious why Rust wasn’t in the test group. it seems to meet all their criteria, though the community was a bit smaller back then. Go is a perfectly reasonable choice though, and easier to learn on average.

steveklabnik 3y ago
https://news.ycombinator.com/item?id=21638455 https://news.ycombinator.com/item?id=21638455
greatgib 3y ago
Python is not in the list also, despite them using nodejs. But when I read the following: Based on these criteria, we narrowed our testing languages to C++17, Java, Node.js, and Go. The first three languages were already in use at American Express, whereas Go was not. What I understand is that they did not really wanted to do a real benchmark but they decided to use Go first, and then they had to setup a kind of "bullshit neutral study" to pretend to leadership that Go was the best choice for all the languages that could fit the job. Maybe it is the case in the end for them, but it is another subject.
- sturmdev 3y ago
  My understanding is that JS running on Node is insanely fast compared to most other interpreted languages. This is because of how much work has been put into the V8 engine, especially the JIT compiler. Based on this, it’s fair to put Node into a league above languages like Python and Ruby when considering performance. This might have played a role in their decision to include Node.
aaomidi 3y ago
Rust crypto libraries were definitely not ready in 2019. I think the story is a bit better today.

liampulles 3y ago

I love how stubbornly plain and lacking in magic Go is. There is almost no hidden control flow, there's no aspect oriented programming, or operator overloading, etc., etc. - pretty much everything the code actually does is in front of you. I think this means you have a bigger upfront cost in writing code the first time, with the benefit of easier debugging and easy maintenance later (which is a tradeoff I'm very happy to take).

aaomidi 3y ago
TBH go does have a lot of weird magic stuff. For example: - types vs type aliases - the init function - how you define a struct can have significant impact on performance. There are lints that align struct better. - changing a returned value in a defer function. Very useful but can easily hide complexity. - interfaces are.. weird. They can also easily hide weirdness.
- richbell 3y ago
  Build directives are magic comments. Embed is a magic comment but you also need to import it. Weird magic syntax to declare enums. Etc.
- liampulles 3y ago
  I was not very explicit on what magic means to me (so fair do's). By magic I mean logic of a program which is not plain function calls, conditionals, etc. or unexpected/hidden logic. In that light, I agree init calls and defers are a bit magic. I'll even add panics in there. But I don't use these much personally, so I'm hesitant to say it is a lot of magic - especially compared to something like Java for example.

wizwit999 3y ago

I say this as a personal rust fan, but practically if a language like a combination of Kotlin+Go existed, that'd be like an awesome standard business language. Kotlin is good but just very tied to the JVM, Go has a hostile syntax/developer experience. Another way to frame it is like Rust with a garbage collector. Most biz APIs don't really need zero cost abstractions of Rust.

mdwagner 3y ago
If you're interested, take a look at Crystal (https://crystal-lang.org/ https://crystal-lang.org/)!
bmikaili 3y ago
This is Swift. It‘s just not being used a lot outside of iOS / Macos dev.
cempaka 3y ago
I see OCaml described as "Rust with a GC" pretty often, but maybe it's too heavily functional for what you're seeking.
Shakahs 3y ago
Is being tied to the JVM that much of a problem these days? Developer workstations and servers have much more compute and RAM these days, and native compilation with GraalVM seems to eliminate high JVM startup times and memory overhead. Quarkus for example is known for a snappy developer experience.

itsdrewmiller 3y ago

This seems a lot more like someone deciding on Go up front and then pushing a process through to pick it - it wasn't the fastest in their tests and it was the only language without existing usage at the company, and basically all the qualitative advantages are either not very unique or not quantifiable or both. No other new languages besides Go were even considered, despite some that easily meet the criteria for consideration.

officialchicken 3y ago
Oracle's lawsuit against Google's use of Java API's started in 2019[0]. The writing was on the wall; many Java shops started looking at other options. [0] https://en.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_Inc https://en.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_....
- scarface_74 3y ago
  The lawsuit started in 2009 https://en.m.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_Inc https://en.m.wikipedia.org/wiki/Google_LLC_v._Oracle_America....
- Larrikin 3y ago
  So if you're a Java shop you pick Scala because it seems amazing until you read about real world usage/find out for yourself, and then you pick Kotlin.
bob1029 3y ago
If you forced me to conduct best language olympics, the process I would use might look something like this: 1. Each team develops the same vertical slice of the business in their ecosystem of choice. 2. Test officials will make an edit to each codebase (somehow invisible to git history) that introduces a subtle bug. Alternatively, they may propose a small feature enhancement. 3. Time how long it takes each team to get a corrected/updated build deployed & re-tested successfully. 4. Go to 2 until statisticians are satisfied with the results.
- 3y ago

xyst 3y ago

Lately I have been more interested in rust. Similar trade offs but “fearless concurrency”, access to low level APIs, and no garbage collector sold it for me.

aaomidi 3y ago

I need Go to take one lesson from rust and that’s specifying if a function will mutate values passed through it or not. This can technically be done as a breaking change spread over several versions of go and I think the cost would be worth what we’d get out of it. Utilities can also be made to edit code to support this automatically without having to have some user go fix it all. Basically, I want an introduction of either a function level or value level ‘nut’ keyword like Rust where the function can not mutate a field unless it has explicitly stated mut. This would fundamentally solve a whole class of bugs and errors that happen in go where some library ends up modifying a struct (pointer or a struct with a pointer field) without the caller intending that.

aatd86 3y ago
I'm not sure I fully understand what you're proposing? Is it about function purity or immutable values? I think the way to do the latter in Go is to not export the field and only have it modifiable by a dedicated method. (or even enforce copy-on-write). And passing values instead of pointers where needed. I don't think that breaking compatibility to add a mut keyword would fit Go. It makes more sense for rust. (especially wrt interior mutability... That would make it a very pervasive change, probably too much of a change) Another thing is that wrapping values in an empty interface essentially make those immutable. It doesn't necessarily have mechanical sympathy but as long as you don't wrap pointers, it would be difficult to modify (no aliasing). Then again, it's more defensive and pragmatic coding than something that proves functional purity. Seems that it is enough.