11 ms·

Einstein Analytics and Go

55 points by bsg75 7y ago

moksly 7y ago
GO is always listed as popular, but if I search my entire country for GO jobs there isn’t a single one. Well, there is one single listing, but that’s for a C/C++ job at Google, and it’s listed under “Nice to know” along with Rust and a few others. Are we behind the curve?
- weberc2 7y ago
  What is your country, out of curiosity?
  - moksly 7y ago
    Denmark.
    - simonh 7y ago
      The homeland of the inventors of both C++ and Python. I'm sure there's no favouritism in play at all...
      - gerad 7y ago
        Guido is Dutch, not Danish.
      - moksly 7y ago
        Well not Python, that’s Dutch. Danes did have a hand in Ruby, PHP and a few others, but it doesn’t really show around here. The most listed language is JAVA.
      - IceDane 7y ago
        The inventor of PHP is Danish, and Anders Hejlsberg is also Danish, so we had a hand in C# and TypeScript as well. However, this doesn't mean there is a nationally endorsed programming language.
- hderms 7y ago
  Go is certainly somewhat popular in the US, but imo the language isn't well suited for all tasks.
  - marcrosoft 7y ago
    It is well suited for most tasks though.
  - cwbrandsma 7y ago
    I would argue that no language is suited for all tasks. Some can handle more tasks than others tho.
- wil421 7y ago
  Just did the same thing for my US city. There were a total of 5 job postings from scummy head hunting companies. Try to search Golang.
- peterwwillis 7y ago
  It's a trendy language. It will increase in popularity (e.g. developers want to learn it because it's new) until it eventually wanes. Remember when everybody wanted to write Ruby? This page shows job postings per programming language from searches on Indeed.com: https://www.codeplatoon.org/the-best-paying-and-most-in-demand-programming-languages-in-2019/ https://www.codeplatoon.org/the-best-paying-and-most-in-dema... Go isn't on the list. The TIOBE index has Go at #17: https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/
  - moksly 7y ago
    I remember Ruby, and it’s a similar story really. There is one Ruby shop in my region.
  - asdfman123 7y ago
    People really need to break down salary analysis by region. If a skill commands a higher salary, does that mean I can actually get paid for it more in Houston, or does it mean that it's only used in tech hubs which pay way more to keep up with cost of living?
  - fjp 7y ago
    > Remember when everybody wanted to write Ruby? There is an absolutely mind-boggling amount of Ruby jobs in the US.
- davzie 7y ago
  In the UK Go jobs are so prevalent it's insane. But then again we have a lot of fintech, AI and blockchain startups that are knocking about so that could be a contributing factor.
- deleted 7y ago
  [deleted]
- TrackerFF 7y ago
  Norwegian here. Searched for "Golang", which yielded a whooping 4 results. "Go" gave me 8, but of those only two were the main language, the rest lumped it in the nice to know category. Python on the other hand yielded almost 200.
- robbrit 7y ago
  Having Go experience on my resume, I do get a lot of recruiters reaching out to me for Go jobs. It's quite popular for infrastructure work. This is in the US, not sure if it's the same in other countries.
- kjeetgill 7y ago
  Most companies will hire targeting the language that the vast majority of the company is using: often Java or JavaScript depending on the role. They will also have often have projects in "support languages" for internal tools and services. Often in python atleast, and more and more in Go. Those are often unlisted. Edit: Just noticed you said in your country. I suspect this is true broadly but what do I know?
- rb808 7y ago
  Yeah I'd love to work in Go, but there are so few jobs it either isn't worth it or makes it risky to specialize in it.
- jhoechtl 7y ago
  What is your job search engine?
trimbo 7y ago
Java was not considered?
- blub 7y ago
  One one hand one would be tempted to dismiss Java as uncool, but on the other hand we're talking about Salesforce, so this is a very legitimate question. The truth is probably rather ordinary: even dull corporations want to look cool.
elisharobinson 7y ago
what is this strange obsession with type , it improves performance yeah sure . but to go around claiming that it is a panacea for all things poorly planned is pure unadulterated stupidity. i can concede that some immutable data structures are needed , but types not for me . Python doesnt handle threads well this can be true. also if you are straining python threading system you ought to take a look at your own abstractions and design. chances are you wont go very far before you hit the same bottle neck in a different language if you assumed python gave up at X you could achive 100X with go but you will almost certainly hit the same issue when your data grows 100X.( i am aware of computational complexity is a factor. but the author claims that part is handled in C , from what is described it looks like they replaced flask with go)
- wyldfire 7y ago
  > what is this strange obsession with type , it improves performance yeah sure Static typing allows for a lot of static decisions which in general brings better performance for a language. CPython pushes so many decisions to runtime that it is a very significant performance impact. But the article cites more significant challenges related to type. Not performance, but design: > "First, Python uses loose typing, which was great for a small team rapidly developing new ideas and putting them into production –but less great for an enterprise-scale application that some customers were paying millions of dollars for," he writes. Cue the reminder that in fact Python is strongly (but dynamically) typed. But LeStum's point stands: dynamic type hurts developers trying to read/write unfamiliar python code. The mypy static typing should help out a lot but I don't think it's very popular yet. For performance, IMO in this order you should consider (1) PyPy, (2) multiprocessing, (3) cython and/or c-extensions. (and I suppose implicitly (0) analyze your algorithm, exploit numpy where possible). If you exhaust those, Go seems like a great alternative.
  - elisharobinson 7y ago
    i agree . i might clarify my original post, im half asleep. but i agree with the correction you pointed out in my original post.
  - 7y ago
derivagral 7y ago
I found an interesting subtlety to the message here. They had a Python wrapper around a C library, and effectively never updated the library. I wonder if this was due to engineering decisions, hiring/expertise of their staff (or future hires), or something else (risk? etc). I imagine it is easier these days to find Go devs than C devs who want to work at a company like this? Also, for a sense of scale, this rewrite took about 4 years to complete from first concepts.
- sho 7y ago
  For 9 out of 10 articles like this - "We rewrote our old code in tired old language FOO with hot new language BAR and it's SO MUCH BETTER!" - I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned, and achieved basically the same result, probably faster. But that doesn't scratch the "new hotness" itch, does it? That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Unlike certain other projects I know about where they HAD to write everything in golang/microservices/k8s/etc because it HAD TO SCALE, took 18 months instead of the 3 months it would have taken with rails but credit where credit is due - those 2 or 3 requests a minute (peak) are handled very, very quickly.
  - Barrin92 7y ago
    > That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Yes, in this particular case I don't think 'the new hotness syndrome' applies. The performance limits of python given that it's dynamic and the multithreading problems are fundamental to the language.
    - pjmlp 7y ago
      The same old theme. Common Lisp, Dylan, Smalltalk, JavaScript are just as dynamic as Python and yet run circles around its performance. The only thing lacking is more community love for PyPy and similar endeavours.
jbergens 7y ago
With the current upwards trend for python I wonder how many companies will come to the same conclusion the next 5 years? Maybe ending up rewriting large parts of their system. If not to Golang then to some other fast and statically typed language like c# or java.
thanatropism 7y ago
I'm reminded of the trend of starting systems in Lisp (I believe this included reddit) and then rewriting in Python. Maybe the Blub paradox has been hitting its limits. The point of using leftfield expressive languages was that developer time is orders of magnitude more expensive than machine chugging time. But maybe web-scale-big-data-etc. needs enough (latency) juice that the scales tip and the comparative advantage is clearly on the side of adopting "industrial-strength" enterprisey technologies again.
- robbrit 7y ago
  They touched on this in the article, but I'll go into more detail here since it's been fashionable for years to bash on "Blub" languages like Java or Go. I myself was guilty of this for a long time until I started using these languages in settings where they shine, and developed an appreciation for them. The argument is nothing to do with machine chugging time, and is entirely towards developer time. The problem with expressive languages like Lisp, Ruby, Python, etc. is that the language ends up varying from person to person - the more expressive the language, the more variance there is. This is a feature when you're a small team because the abstractions you build let you move quickly, but it is a bug when you're a large team maintaining a piece of software over years, where developers have come and gone. The ramp-up time to learn and understand the various abstractions that people have built over the years ends up accumulating and cancelling out the gains that those abstractions gave earlier on. Blub languages on the other hand tend to be more uniform, so it's easier for someone who isn't very familiar with the code to dive in and understand what is going on.
  - blub 7y ago
    Java is no longer uniform, especially in the latest versions they're adding more and more features. And there's also Kotlin and project Lombok for people that want even more excitement. And yet Java somehow manages to be both a boring language and still have too many things to lean. This achievement is probably not appreciated enough by those that criticize it.
crimsonalucard 7y ago
Analytics needs unstructured data in order to function well as a modular service. Golangs lack of coproducts makes such data much harder to deal with. As much as I hate js. Nodejs is actually better for one of these servers. Then again it's mostly the database that determines the schemas.
erokar 7y ago
Of all new programming languages that have received a certain amount of hype, Go is the only one I never felt inclined to play with after reading about it. It just seems too dumbed-down and backwards. It's like someone wished CS stopped evolving after 1970 and designed a language to pretend it did. The simplest of abstractions are removed from the language, like operations on collections that abstracts away for loops (e.g. filter, map, reduce, etc.). I don't doubt that it has good CPU performance and concurrency, the developer experience just seems so frustrating.
- eitland 7y ago
  I belive it when other people say they like it, but I agree with you. (That said I really didn't like Java either, and it later became my favorite language for a decade or so.)
  - winter_blue 7y ago
    Why did Java later become your favorite language? And why only for a decade? (I started liking Java a lot more after Java 8 came out.)
    - eitland 7y ago
      My comment became so long I moved it to my blog: https://erik.itland.no/questions-about-why-i-like-java https://erik.itland.no/questions-about-why-i-like-java
      - winter_blue 7y ago
        Thank you for the long response! I can relate (and feel the same) about several of the points you made in your blog post.
        eitland 7y ago
        Thanks a lot for coming back so long after to say that! You actually put a smile on my face :-)
scythe 7y ago
Previous discussions here: https://news.ycombinator.com/item?id=21203651 https://news.ycombinator.com/item?id=21203651 I worked on this project; my comment is here: https://news.ycombinator.com/item?id=21205504 https://news.ycombinator.com/item?id=21205504
weberc2 7y ago
> After these ports, our team has built up some expertise with Go and its compiler quirks. But you can still get burned. For example, you can very easily write data that you want to place on the cheaper stack instead onto the much more expensive heap. You won’t even know this is happening by reading over your code. That’s why, as with any new language that you require high performance from, you need to monitor processes closely and create benchmarks around CPU and memory use. And then share what you learn with the community so that this knowledge becomes less tribal. While Go doesn't have formal semantics that let you specify whether a value is allocated on the heap or stack, it seems like it would be easy enough to create your own. The compiler has flags that cause it to output the sites where it heap-allocates--if you add comments above a function or an allocation site such as `// noalloc`, then you could write a "linter" that compares the allocation sites against those comments and errors if one of those noalloc sites allocates. In lieu of allocation semantics, this seems like a better approach than writing performance tests for each of these sites.
- twotwotwo 7y ago
  I see why you'd do perf tests as well (you might fail to annotate everything that could make your function run slowly), but a static analyzer that looks at -gcflags -m output does sound like a super cool idea--direct way to check if your intuitions about the code and its compilation are right.
  - weberc2 7y ago
    Yeah, I envision fewer performance tests at a higher level of granularity. Not only might you miss annotations here and there, but allocations aren't the only thing (although they are probably the most significant) that would affect your performance and my proposal won't cover you there.
- hinkley 7y ago
  There are at least two ways to fail. You can get something wrong from the start, or you can get it right and fail to keep it that way. Perf sensitive code can often be that way. Some innocent coworker comes in to add a feature and they don't get why two idioms are not equivalent from the compiler's perspective, and so they change things. Meanwhile, you're busy doing something else. The code is still correct from a testing standpoint. It still satisfies your code standards. But it no longer meets your response time expectations. It's a lot of work to maintain the sort of toolchain that lets you reliably spot this sort of thing at build time or even in pre-prod environments, and it's difficult to identify places where those tools are failing to detect problems until there's been a problem.
SomaticPirate 7y ago
I’m curious to what tribal knowledge they are referring to? What are these incantations to make my Go code faster? Does this mean just stripping debug symbols or are there flags I should know about?
- weberc2 7y ago
  They were talking about writing code that doesn't allocate in the hot path. Mostly this is pretty easy to eye-ball, but you can make sure it doesn't allocate by passing `-gcflags -m` to the `go build` command. This is almost always the initial bottleneck in naive Go code.
twotwotwo 7y ago
The post dates the rewrite effort as happening in 2016-7. In the past couple years escape analysis has gotten a lot of effort (https://github.com/golang/go/issues/23109 https://github.com/golang/go/issues/23109) and there's also now mid-stack inlining (https://github.com/golang/go/issues/19348 https://github.com/golang/go/issues/19348) which obviously can be useful in itself but also help escape analysis and other optimizations work. Does anyone know if there's any big picture work on GC throughput in progress? The idea of the ROC (fast freeing of memory never shared between threads on goroutine exit) was canned since it regressed on some workloads and I'm having trouble finding signs Austin's idea with hashing memory landed either. A moving GC or a read barrier seem mostly out because they can hurt other things (C interop, performance of code that doesn't allocate much). Though they seem to have addressed pauses quite well, and lots of code out there isn't GC-heavy, it still seems like a key area where it might be possible to give a "free" throughput win to existing code.
SnarkAsh 7y ago
Why is this on the Stack Overflow blog? When I replied to their survey that I wanted "tech articles written by other developers", I was imagining a platform for Stack Overflow authors to contribute longer-form work -- an idea that's been floated by staff for most of the life of the site! I wasn't expecting random cross-promotional content.
thumbsdownnow 7y ago
Simplicity is king.
lugg 7y ago
> First, Python uses loose typing, which was great for a small team rapidly developing new ideas and putting them into production—but less great for an enterprise-scale application that some customers were paying millions of dollars for. Code review is supposed to catch people trying to mix return types, mess with or overload parameters in unexpected ways. Static typing removes this need but adds a million more. If your people aren't catching this stuff in review, they're probably not catching other stuff, that is, static typing isn't going to save you like you think it is.

55 points by bsg75 7y ago

GO is always listed as popular, but if I search my entire country for GO jobs there isn’t a single one. Well, there is one single listing, but that’s for a C/C++ job at Google, and it’s listed under “Nice to know” along with Rust and a few others. Are we behind the curve?

weberc2 7y ago
What is your country, out of curiosity?
- moksly 7y ago
  Denmark.
  - simonh 7y ago
    The homeland of the inventors of both C++ and Python. I'm sure there's no favouritism in play at all...
    - gerad 7y ago
      Guido is Dutch, not Danish.
    - moksly 7y ago
      Well not Python, that’s Dutch. Danes did have a hand in Ruby, PHP and a few others, but it doesn’t really show around here. The most listed language is JAVA.
    - IceDane 7y ago
      The inventor of PHP is Danish, and Anders Hejlsberg is also Danish, so we had a hand in C# and TypeScript as well. However, this doesn't mean there is a nationally endorsed programming language.
hderms 7y ago
Go is certainly somewhat popular in the US, but imo the language isn't well suited for all tasks.
- marcrosoft 7y ago
  It is well suited for most tasks though.
- cwbrandsma 7y ago
  I would argue that no language is suited for all tasks. Some can handle more tasks than others tho.
wil421 7y ago
Just did the same thing for my US city. There were a total of 5 job postings from scummy head hunting companies. Try to search Golang.
peterwwillis 7y ago
It's a trendy language. It will increase in popularity (e.g. developers want to learn it because it's new) until it eventually wanes. Remember when everybody wanted to write Ruby? This page shows job postings per programming language from searches on Indeed.com: https://www.codeplatoon.org/the-best-paying-and-most-in-demand-programming-languages-in-2019/ https://www.codeplatoon.org/the-best-paying-and-most-in-dema... Go isn't on the list. The TIOBE index has Go at #17: https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/
- moksly 7y ago
  I remember Ruby, and it’s a similar story really. There is one Ruby shop in my region.
- asdfman123 7y ago
  People really need to break down salary analysis by region. If a skill commands a higher salary, does that mean I can actually get paid for it more in Houston, or does it mean that it's only used in tech hubs which pay way more to keep up with cost of living?
- fjp 7y ago
  > Remember when everybody wanted to write Ruby? There is an absolutely mind-boggling amount of Ruby jobs in the US.
davzie 7y ago
In the UK Go jobs are so prevalent it's insane. But then again we have a lot of fintech, AI and blockchain startups that are knocking about so that could be a contributing factor.
deleted 7y ago
[deleted]
TrackerFF 7y ago
Norwegian here. Searched for "Golang", which yielded a whooping 4 results. "Go" gave me 8, but of those only two were the main language, the rest lumped it in the nice to know category. Python on the other hand yielded almost 200.
robbrit 7y ago
Having Go experience on my resume, I do get a lot of recruiters reaching out to me for Go jobs. It's quite popular for infrastructure work. This is in the US, not sure if it's the same in other countries.
kjeetgill 7y ago
Most companies will hire targeting the language that the vast majority of the company is using: often Java or JavaScript depending on the role. They will also have often have projects in "support languages" for internal tools and services. Often in python atleast, and more and more in Go. Those are often unlisted. Edit: Just noticed you said in your country. I suspect this is true broadly but what do I know?
rb808 7y ago
Yeah I'd love to work in Go, but there are so few jobs it either isn't worth it or makes it risky to specialize in it.
jhoechtl 7y ago
What is your job search engine?

trimbo 7y ago

Java was not considered?

blub 7y ago
One one hand one would be tempted to dismiss Java as uncool, but on the other hand we're talking about Salesforce, so this is a very legitimate question. The truth is probably rather ordinary: even dull corporations want to look cool.

elisharobinson 7y ago

what is this strange obsession with type , it improves performance yeah sure . but to go around claiming that it is a panacea for all things poorly planned is pure unadulterated stupidity. i can concede that some immutable data structures are needed , but types not for me . Python doesnt handle threads well this can be true. also if you are straining python threading system you ought to take a look at your own abstractions and design. chances are you wont go very far before you hit the same bottle neck in a different language if you assumed python gave up at X you could achive 100X with go but you will almost certainly hit the same issue when your data grows 100X.( i am aware of computational complexity is a factor. but the author claims that part is handled in C , from what is described it looks like they replaced flask with go)

wyldfire 7y ago
> what is this strange obsession with type , it improves performance yeah sure Static typing allows for a lot of static decisions which in general brings better performance for a language. CPython pushes so many decisions to runtime that it is a very significant performance impact. But the article cites more significant challenges related to type. Not performance, but design: > "First, Python uses loose typing, which was great for a small team rapidly developing new ideas and putting them into production –but less great for an enterprise-scale application that some customers were paying millions of dollars for," he writes. Cue the reminder that in fact Python is strongly (but dynamically) typed. But LeStum's point stands: dynamic type hurts developers trying to read/write unfamiliar python code. The mypy static typing should help out a lot but I don't think it's very popular yet. For performance, IMO in this order you should consider (1) PyPy, (2) multiprocessing, (3) cython and/or c-extensions. (and I suppose implicitly (0) analyze your algorithm, exploit numpy where possible). If you exhaust those, Go seems like a great alternative.
- elisharobinson 7y ago
  i agree . i might clarify my original post, im half asleep. but i agree with the correction you pointed out in my original post.
- 7y ago

derivagral 7y ago

I found an interesting subtlety to the message here. They had a Python wrapper around a C library, and effectively never updated the library. I wonder if this was due to engineering decisions, hiring/expertise of their staff (or future hires), or something else (risk? etc). I imagine it is easier these days to find Go devs than C devs who want to work at a company like this? Also, for a sense of scale, this rewrite took about 4 years to complete from first concepts.

sho 7y ago
For 9 out of 10 articles like this - "We rewrote our old code in tired old language FOO with hot new language BAR and it's SO MUCH BETTER!" - I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned, and achieved basically the same result, probably faster. But that doesn't scratch the "new hotness" itch, does it? That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Unlike certain other projects I know about where they HAD to write everything in golang/microservices/k8s/etc because it HAD TO SCALE, took 18 months instead of the 3 months it would have taken with rails but credit where credit is due - those 2 or 3 requests a minute (peak) are handled very, very quickly.
- Barrin92 7y ago
  > That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Yes, in this particular case I don't think 'the new hotness syndrome' applies. The performance limits of python given that it's dynamic and the multithreading problems are fundamental to the language.
  - pjmlp 7y ago
    The same old theme. Common Lisp, Dylan, Smalltalk, JavaScript are just as dynamic as Python and yet run circles around its performance. The only thing lacking is more community love for PyPy and similar endeavours.

jbergens 7y ago

With the current upwards trend for python I wonder how many companies will come to the same conclusion the next 5 years? Maybe ending up rewriting large parts of their system. If not to Golang then to some other fast and statically typed language like c# or java.

thanatropism 7y ago

I'm reminded of the trend of starting systems in Lisp (I believe this included reddit) and then rewriting in Python. Maybe the Blub paradox has been hitting its limits. The point of using leftfield expressive languages was that developer time is orders of magnitude more expensive than machine chugging time. But maybe web-scale-big-data-etc. needs enough (latency) juice that the scales tip and the comparative advantage is clearly on the side of adopting "industrial-strength" enterprisey technologies again.

robbrit 7y ago
They touched on this in the article, but I'll go into more detail here since it's been fashionable for years to bash on "Blub" languages like Java or Go. I myself was guilty of this for a long time until I started using these languages in settings where they shine, and developed an appreciation for them. The argument is nothing to do with machine chugging time, and is entirely towards developer time. The problem with expressive languages like Lisp, Ruby, Python, etc. is that the language ends up varying from person to person - the more expressive the language, the more variance there is. This is a feature when you're a small team because the abstractions you build let you move quickly, but it is a bug when you're a large team maintaining a piece of software over years, where developers have come and gone. The ramp-up time to learn and understand the various abstractions that people have built over the years ends up accumulating and cancelling out the gains that those abstractions gave earlier on. Blub languages on the other hand tend to be more uniform, so it's easier for someone who isn't very familiar with the code to dive in and understand what is going on.
- blub 7y ago
  Java is no longer uniform, especially in the latest versions they're adding more and more features. And there's also Kotlin and project Lombok for people that want even more excitement. And yet Java somehow manages to be both a boring language and still have too many things to lean. This achievement is probably not appreciated enough by those that criticize it.

crimsonalucard 7y ago

Analytics needs unstructured data in order to function well as a modular service. Golangs lack of coproducts makes such data much harder to deal with. As much as I hate js. Nodejs is actually better for one of these servers. Then again it's mostly the database that determines the schemas.

erokar 7y ago

Of all new programming languages that have received a certain amount of hype, Go is the only one I never felt inclined to play with after reading about it. It just seems too dumbed-down and backwards. It's like someone wished CS stopped evolving after 1970 and designed a language to pretend it did. The simplest of abstractions are removed from the language, like operations on collections that abstracts away for loops (e.g. filter, map, reduce, etc.). I don't doubt that it has good CPU performance and concurrency, the developer experience just seems so frustrating.

eitland 7y ago
I belive it when other people say they like it, but I agree with you. (That said I really didn't like Java either, and it later became my favorite language for a decade or so.)
- winter_blue 7y ago
  Why did Java later become your favorite language? And why only for a decade? (I started liking Java a lot more after Java 8 came out.)
  - eitland 7y ago
    My comment became so long I moved it to my blog: https://erik.itland.no/questions-about-why-i-like-java https://erik.itland.no/questions-about-why-i-like-java
    - winter_blue 7y ago
      Thank you for the long response! I can relate (and feel the same) about several of the points you made in your blog post.
      - eitland 7y ago
        Thanks a lot for coming back so long after to say that! You actually put a smile on my face :-)

scythe 7y ago

Previous discussions here: https://news.ycombinator.com/item?id=21203651 https://news.ycombinator.com/item?id=21203651 I worked on this project; my comment is here: https://news.ycombinator.com/item?id=21205504 https://news.ycombinator.com/item?id=21205504

weberc2 7y ago

> After these ports, our team has built up some expertise with Go and its compiler quirks. But you can still get burned. For example, you can very easily write data that you want to place on the cheaper stack instead onto the much more expensive heap. You won’t even know this is happening by reading over your code. That’s why, as with any new language that you require high performance from, you need to monitor processes closely and create benchmarks around CPU and memory use. And then share what you learn with the community so that this knowledge becomes less tribal. While Go doesn't have formal semantics that let you specify whether a value is allocated on the heap or stack, it seems like it would be easy enough to create your own. The compiler has flags that cause it to output the sites where it heap-allocates--if you add comments above a function or an allocation site such as `// noalloc`, then you could write a "linter" that compares the allocation sites against those comments and errors if one of those noalloc sites allocates. In lieu of allocation semantics, this seems like a better approach than writing performance tests for each of these sites.

twotwotwo 7y ago
I see why you'd do perf tests as well (you might fail to annotate everything that could make your function run slowly), but a static analyzer that looks at -gcflags -m output does sound like a super cool idea--direct way to check if your intuitions about the code and its compilation are right.
- weberc2 7y ago
  Yeah, I envision fewer performance tests at a higher level of granularity. Not only might you miss annotations here and there, but allocations aren't the only thing (although they are probably the most significant) that would affect your performance and my proposal won't cover you there.
hinkley 7y ago
There are at least two ways to fail. You can get something wrong from the start, or you can get it right and fail to keep it that way. Perf sensitive code can often be that way. Some innocent coworker comes in to add a feature and they don't get why two idioms are not equivalent from the compiler's perspective, and so they change things. Meanwhile, you're busy doing something else. The code is still correct from a testing standpoint. It still satisfies your code standards. But it no longer meets your response time expectations. It's a lot of work to maintain the sort of toolchain that lets you reliably spot this sort of thing at build time or even in pre-prod environments, and it's difficult to identify places where those tools are failing to detect problems until there's been a problem.

SomaticPirate 7y ago

I’m curious to what tribal knowledge they are referring to? What are these incantations to make my Go code faster? Does this mean just stripping debug symbols or are there flags I should know about?

weberc2 7y ago
They were talking about writing code that doesn't allocate in the hot path. Mostly this is pretty easy to eye-ball, but you can make sure it doesn't allocate by passing `-gcflags -m` to the `go build` command. This is almost always the initial bottleneck in naive Go code.

twotwotwo 7y ago

The post dates the rewrite effort as happening in 2016-7. In the past couple years escape analysis has gotten a lot of effort (https://github.com/golang/go/issues/23109 https://github.com/golang/go/issues/23109) and there's also now mid-stack inlining (https://github.com/golang/go/issues/19348 https://github.com/golang/go/issues/19348) which obviously can be useful in itself but also help escape analysis and other optimizations work. Does anyone know if there's any big picture work on GC throughput in progress? The idea of the ROC (fast freeing of memory never shared between threads on goroutine exit) was canned since it regressed on some workloads and I'm having trouble finding signs Austin's idea with hashing memory landed either. A moving GC or a read barrier seem mostly out because they can hurt other things (C interop, performance of code that doesn't allocate much). Though they seem to have addressed pauses quite well, and lots of code out there isn't GC-heavy, it still seems like a key area where it might be possible to give a "free" throughput win to existing code.

SnarkAsh 7y ago

Why is this on the Stack Overflow blog? When I replied to their survey that I wanted "tech articles written by other developers", I was imagining a platform for Stack Overflow authors to contribute longer-form work -- an idea that's been floated by staff for most of the life of the site! I wasn't expecting random cross-promotional content.

thumbsdownnow 7y ago

Simplicity is king.

lugg 7y ago

> First, Python uses loose typing, which was great for a small team rapidly developing new ideas and putting them into production—but less great for an enterprise-scale application that some customers were paying millions of dollars for. Code review is supposed to catch people trying to mix return types, mess with or overload parameters in unexpected ways. Static typing removes this need but adds a million more. If your people aren't catching this stuff in review, they're probably not catching other stuff, that is, static typing isn't going to save you like you think it is.