5 ms·

I would like to hear more about the technical implementations from the author on how he managed to improve performance. He mentions: "Due to the complex nature

by kenkam 13y ago

I would like to hear more about the technical implementations from the author on how he managed to improve performance. He mentions: "Due to the complex nature of the system this script could take up to three minutes to scan nodes and process the results" but how does Go solve this complex system that Python couldn't? 3 minutes to 1 second is a superb improvement. The second paragraph reads like: replacing a relational DB with a (in memory?) key value store resulted in more throughput. He implemented it in Go, but he could have implemented it in any other language. What I want to know is how he convinced management to use Go? Where did he find the programmers? I work in a similar environment and I don't see my firm adopting Go any time soon, although I wished they had as I'm a big fan of Go and believe it has a place especially server side processes (e.g. algo trading, market data feeds) where concurrent connections are prevalent.

deleted 13y ago
[deleted]
elithrar 13y ago
> He implemented it in Go, but he could have implemented it in any other language I assume he used Go's concurrency features, although he could have articulated on that. > Where did he find the programmers I don't think you explicitly need to hire new programmers. If you have capable, existing programmers, the learning curve is pretty minimal for something like Go.
- TheAnimus 13y ago
  >I assume he used Go's concurrency features, although he could have articulated on that. I've yet to find a good explanation of what is so special about them, why I'd choose it over F# or C# Async/Await. In the case of pinging servers, I would have prefered to use a 'cheaper' to develop language, which has a boatload of libraries that are widely used. Anyone got any links or stories about why you would want to use Go for something like that? I don't find this blog story remotely useful in the whole 'why Go' thing.
  - threeseed 13y ago
    This is a good overview of Go concurrency: http://www.slideshare.net/jgrahamc/go-oncurrency http://www.slideshare.net/jgrahamc/go-oncurrency As a Java developer I personally don't find it that impressive compared to something like the LMAX Disruptor or even Vert.x but I can appreciate that it is simple and that always counts for a lot.
    - randomsearch 13y ago
      As an Ada 95 developer, I like that Go seems to have an Adalite concurrency-related syntax.
      - tptacek 13y ago
        Both systems have a CSP design, right? Golang is pretty up front about having lifted its design from Hoare.
        randomsearch 13y ago
        Yes, you're right - the design comes from CSP. The syntax looks nicely Adalike.
  - kenkam 13y ago
    I think async/await is semantically similar to the promises pattern [1] but different to the Go concurrent models using channels. What the compiler does with await/async is quite interesting [2]. Essentially it takes your await/async code and turns the related code into a state machine which keeps track of how control should be switched between the callee and the await keyword (and uses Task for the async bit). [1] http://blog.parse.com/2013/01/29/whats-so-great-about-javascript-promises/ http://blog.parse.com/2013/01/29/whats-so-great-about-javasc... [2] http://www.codeproject.com/Articles/535635/Async-Await-and-the-Generated-StateMachine http://www.codeproject.com/Articles/535635/Async-Await-and-t...
    - TheAnimus 13y ago
- kenkam 13y ago
  I agree with not needing to hire new devs, but convincing management and the business to move to a new, untried system with current devs is no trivial feat. It would be nice if he could expand on how that came about.
  - Ecio78 13y ago
    Probably that would be the most interesting part of the story...
  - Stephen_C 13y ago
    Andy convinced management by spending a lot of personal development time creating and testing the Go components before demonstrating in a peer review process that they were an appropriate solution. There's no special method - the introduction of Go was incremental. If I remember correctly the monitoring collection component (mentioned in the blog post) was the first. It was small, easy to swap in and easy to demonstrate the improvements. You build trust in the technology and work flow before moving on to larger more critical/risky projects. (Andy is very much missed by his former colleagues)
    - j_s 13y ago
      Would be interesting to hear the details on management's side of the story: 'This guy re-wrote all our stuff in the new hotness then left the company'
      - Stephen_C 13y ago
        Well what sort of story are you expecting to hear? Without going into detail it's not really that exciting. There was a requirement for high volume/through put messaging component in our infrastructure. That technological requirement (along with non functional requirements e.g. development time/costs) was more than adequately met by a program(s) written by a team member written in Go. Was it an opportunity to further explore the "new hotness" that is/was Go…absolutely but the solution could have been implemented in one of the other supported technologies C/C++/Java with appropriate advantages/disadvantages associated with each. It will stay in use for as long as it provides the best solution, if that changes a more appropriate solution will be investigated. > 'This guy re-wrote all our stuff in the new hotness then left the company' The Go dataStore is "a" component within a large infrastructure, it is supportable by more than one member of the development team (a core requirement when deploying any new technology to production). I'll also note that before Andy left, being a professional and thoughtful colleague, he put a lot of effort into ensuring appropriate training and knowledge was given to team members that had not been part of the deployment. As Andy is a very well regarded former colleague, it helps that we could easily get in touch for a query, even when we're out for a beer or five* :) (*five might not be a maximum in that scenario)
deleted 13y ago
[deleted]
brown9-2 13y ago
On the Python script, it sounds like the performance problems were more likely than not caused by inadequate use of any concurrency features rather than an inherent problem with Python The Language. As exciting as Go is it sounds like replacing the old script with a sufficiently concurrent and well-written application in any language should have worked well. More details would be really helpful. Also the author seems to be responding to a question of reliability by talking about performance which is not the same thing.
- Stephen_C 13y ago
  I doubt Andy would disagree with you, it's no slight or bad commentary against Python (Andy was the cheerleader for introducing Python into our infrastructure where it is blossoming nicely). With sufficient development time Andy could have introduced more concurrency and optimisation into the Python monitoring component (I'm thinking Gevent might have been a nice fit)...but without introducing additional frameworks or libraries Go had these features as a core component. The offhand performance comment aside, Andy was noting that a component written in Go is depended on to handle VERY high volumes of message throughput in a financial services firm. While no proof of reliability it is merely anecdotal support that Go is being used in production environments.
- coldtea 13y ago
  >On the Python script, it sounds like the performance problems were more likely than not caused by inadequate use of any concurrency features rather than an inherent problem with Python The Language. "Inadequate concurrency features" IS an inherent problem with Python The Language. (Notice I didn't say "inadequate use of concurrency features", I said "inadequate concurrency features").
  - Roboprog 13y ago
    It's been a while, but I remember that Python threads run into a global interpreter lock quite often, yes?
    - bvdbijl 13y ago
codonaut 13y ago
Can anyone explain why exactly a script written in one language would stall, while the same script in another wouldn't? Is there that much inherent instability in Python? Can it be assumed that the scripts in this case aren't comparable?
- _pmf_ 13y ago
  > Can anyone explain why exactly a script written in one language would stall, while the same script in another wouldn't? It stalls in Python because the author is not acquainted with non-blocking system programming (the select()-call, which has been around since at least the eighties and has been a part of core Python since a very long time). As to why a software developer who did not invest some minimal time into learning basic system programming feels qualified to write a blog post about this topic is another question.
  - tptacek 13y ago
    It's funny how people who learn evented programming in scripting languages like Python feel like they've discovered some new hidden concept. No competent systems developer fails to understand what select() does. Suggesting that the author was't "acquainted" with select says more about this comment than about the author of the post it comments on.
  - coldtea 13y ago
    >It stalls in Python because the author is not acquainted with non-blocking system programming Citation needed. How about "he is acquainted but can't be bothered to use it retrofitted to a language not tailor made for it"? >As to why a software developer who did not invest some minimal time into learning basic system programming feels qualified to write a blog post about this topic is another question. And why you think you're any better than him based on a short blog post (and especially one in which he does not delve into why the Python version was slower or says it would be impossible to make it faster, just mentions it's speed in passing), is beyond me. Maybe cut down the snark?
- Cowen 13y ago
  Without actually be able to see either script, I would assume that his Go implementation took advantage of some of Go's more natural concurrency features. The only reason I'd assume the Go version was concurrent and the Python one wasn't is that concurrent processing in Python can be very prickly. That is by design. Guido has talked about how adding too much support for concurrency at the language level would complicate things and probably end up with a language that was very non-Pythonic.

by kenkam 13y ago

deleted 13y ago

[deleted]

elithrar 13y ago

> He implemented it in Go, but he could have implemented it in any other language I assume he used Go's concurrency features, although he could have articulated on that. > Where did he find the programmers I don't think you explicitly need to hire new programmers. If you have capable, existing programmers, the learning curve is pretty minimal for something like Go.

TheAnimus 13y ago
>I assume he used Go's concurrency features, although he could have articulated on that. I've yet to find a good explanation of what is so special about them, why I'd choose it over F# or C# Async/Await. In the case of pinging servers, I would have prefered to use a 'cheaper' to develop language, which has a boatload of libraries that are widely used. Anyone got any links or stories about why you would want to use Go for something like that? I don't find this blog story remotely useful in the whole 'why Go' thing.
- threeseed 13y ago
  This is a good overview of Go concurrency: http://www.slideshare.net/jgrahamc/go-oncurrency http://www.slideshare.net/jgrahamc/go-oncurrency As a Java developer I personally don't find it that impressive compared to something like the LMAX Disruptor or even Vert.x but I can appreciate that it is simple and that always counts for a lot.
  - randomsearch 13y ago
    As an Ada 95 developer, I like that Go seems to have an Adalite concurrency-related syntax.
    - tptacek 13y ago
      Both systems have a CSP design, right? Golang is pretty up front about having lifted its design from Hoare.
      - randomsearch 13y ago
        Yes, you're right - the design comes from CSP. The syntax looks nicely Adalike.
- kenkam 13y ago
  I think async/await is semantically similar to the promises pattern [1] but different to the Go concurrent models using channels. What the compiler does with await/async is quite interesting [2]. Essentially it takes your await/async code and turns the related code into a state machine which keeps track of how control should be switched between the callee and the await keyword (and uses Task for the async bit). [1] http://blog.parse.com/2013/01/29/whats-so-great-about-javascript-promises/ http://blog.parse.com/2013/01/29/whats-so-great-about-javasc... [2] http://www.codeproject.com/Articles/535635/Async-Await-and-the-Generated-StateMachine http://www.codeproject.com/Articles/535635/Async-Await-and-t...
  - TheAnimus 13y ago
kenkam 13y ago
I agree with not needing to hire new devs, but convincing management and the business to move to a new, untried system with current devs is no trivial feat. It would be nice if he could expand on how that came about.
- Ecio78 13y ago
  Probably that would be the most interesting part of the story...
- Stephen_C 13y ago
  Andy convinced management by spending a lot of personal development time creating and testing the Go components before demonstrating in a peer review process that they were an appropriate solution. There's no special method - the introduction of Go was incremental. If I remember correctly the monitoring collection component (mentioned in the blog post) was the first. It was small, easy to swap in and easy to demonstrate the improvements. You build trust in the technology and work flow before moving on to larger more critical/risky projects. (Andy is very much missed by his former colleagues)
  - j_s 13y ago
    Would be interesting to hear the details on management's side of the story: 'This guy re-wrote all our stuff in the new hotness then left the company'
    - Stephen_C 13y ago
      Well what sort of story are you expecting to hear? Without going into detail it's not really that exciting. There was a requirement for high volume/through put messaging component in our infrastructure. That technological requirement (along with non functional requirements e.g. development time/costs) was more than adequately met by a program(s) written by a team member written in Go. Was it an opportunity to further explore the "new hotness" that is/was Go…absolutely but the solution could have been implemented in one of the other supported technologies C/C++/Java with appropriate advantages/disadvantages associated with each. It will stay in use for as long as it provides the best solution, if that changes a more appropriate solution will be investigated. > 'This guy re-wrote all our stuff in the new hotness then left the company' The Go dataStore is "a" component within a large infrastructure, it is supportable by more than one member of the development team (a core requirement when deploying any new technology to production). I'll also note that before Andy left, being a professional and thoughtful colleague, he put a lot of effort into ensuring appropriate training and knowledge was given to team members that had not been part of the deployment. As Andy is a very well regarded former colleague, it helps that we could easily get in touch for a query, even when we're out for a beer or five* :) (*five might not be a maximum in that scenario)

deleted 13y ago

[deleted]

brown9-2 13y ago

On the Python script, it sounds like the performance problems were more likely than not caused by inadequate use of any concurrency features rather than an inherent problem with Python The Language. As exciting as Go is it sounds like replacing the old script with a sufficiently concurrent and well-written application in any language should have worked well. More details would be really helpful. Also the author seems to be responding to a question of reliability by talking about performance which is not the same thing.

Stephen_C 13y ago
I doubt Andy would disagree with you, it's no slight or bad commentary against Python (Andy was the cheerleader for introducing Python into our infrastructure where it is blossoming nicely). With sufficient development time Andy could have introduced more concurrency and optimisation into the Python monitoring component (I'm thinking Gevent might have been a nice fit)...but without introducing additional frameworks or libraries Go had these features as a core component. The offhand performance comment aside, Andy was noting that a component written in Go is depended on to handle VERY high volumes of message throughput in a financial services firm. While no proof of reliability it is merely anecdotal support that Go is being used in production environments.
coldtea 13y ago
>On the Python script, it sounds like the performance problems were more likely than not caused by inadequate use of any concurrency features rather than an inherent problem with Python The Language. "Inadequate concurrency features" IS an inherent problem with Python The Language. (Notice I didn't say "inadequate use of concurrency features", I said "inadequate concurrency features").
- Roboprog 13y ago
  It's been a while, but I remember that Python threads run into a global interpreter lock quite often, yes?
  - bvdbijl 13y ago

codonaut 13y ago

Can anyone explain why exactly a script written in one language would stall, while the same script in another wouldn't? Is there that much inherent instability in Python? Can it be assumed that the scripts in this case aren't comparable?

_pmf_ 13y ago
> Can anyone explain why exactly a script written in one language would stall, while the same script in another wouldn't? It stalls in Python because the author is not acquainted with non-blocking system programming (the select()-call, which has been around since at least the eighties and has been a part of core Python since a very long time). As to why a software developer who did not invest some minimal time into learning basic system programming feels qualified to write a blog post about this topic is another question.
- tptacek 13y ago
  It's funny how people who learn evented programming in scripting languages like Python feel like they've discovered some new hidden concept. No competent systems developer fails to understand what select() does. Suggesting that the author was't "acquainted" with select says more about this comment than about the author of the post it comments on.
- coldtea 13y ago
  >It stalls in Python because the author is not acquainted with non-blocking system programming Citation needed. How about "he is acquainted but can't be bothered to use it retrofitted to a language not tailor made for it"? >As to why a software developer who did not invest some minimal time into learning basic system programming feels qualified to write a blog post about this topic is another question. And why you think you're any better than him based on a short blog post (and especially one in which he does not delve into why the Python version was slower or says it would be impossible to make it faster, just mentions it's speed in passing), is beyond me. Maybe cut down the snark?
Cowen 13y ago
Without actually be able to see either script, I would assume that his Go implementation took advantage of some of Go's more natural concurrency features. The only reason I'd assume the Go version was concurrent and the Python one wasn't is that concurrent processing in Python can be very prickly. That is by design. Guido has talked about how adding too much support for concurrency at the language level would complicate things and probably end up with a language that was very non-Pythonic.