7 ms·

I occasionally use Python for large-scale processing of structured data (XML, JSON) and the data's really messy as you'd expect (malformed responses, missing fi

I occasionally use Python for large-scale processing of structured data (XML, JSON) and the data's really messy as you'd expect (malformed responses, missing fields, normalisation, etc..). One of the things putting me off of Go for these kinds of projects is it's much more verbose/harder to handle edge cases - it feels like I'm less productive writing the program, but end up building something much more stable than the Python equivalent. I'm definitely going to see if Go is acceptable for my next project since I really need the performance for my workload, and I've wanted to put this great language to work in something that isn't a toy project.

axaxs 7y ago
> is it's much more verbose/harder to handle edge cases - it feels like I'm less productive writing the program, but end up building something much more stable than the Python equivalent This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone. > I really need the performance for my workload Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually.
- cameronbrown 7y ago
  > This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone. Yep. Maybe I just suck at programming but I couldn't write enough tests to avoid not being fed up debugging stack traces in production. I started using Python's type hints, since this was the source of most of the bugs which helped but I'd rather just use a strongly typed language next time. > Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually. I've got a few million records pulled from an API so I'm extremely I/O bound and goroutines are easier to work with than Python concurrency. I've also got reasonably complex data processing beyond just XML/JSON parsing. I think on the whole Go is still going to be faster, but thanks for the heads up.
  - timothycrosley 7y ago
    Have you tried async? I find it much easier personally to reason about and debug than channels, and is perfect for IO-bound problems. If you really like channels, note that Python 3.9 will have sub-interpreters and channels as well: https://hackernoon.com/has-the-python-gil-been-slain-9440d28fa93d https://hackernoon.com/has-the-python-gil-been-slain-9440d28...
    - cameronbrown 7y ago
      Unfortunately I deploy on App Engine which is still at Python 3.7.
    - weberc2 7y ago
      Goroutines are quite a lot nicer than async for a few reasons. Firstly, there are no “forgot to await” errors, secondly there is no need to worry about some library making a sync call way down the call stack and therefore blocking your event loop, and thirdly goroutines can use all cores on a CPU to parallelize CPU intensive tasks, which means you don’t block the event loop unless you really are out of CPU. Blocking the event loop is a really big deal since it can cause health checks to fail which can cascale onto other instances and bring your app down. We’ve experienced this with both CPU- and IO-bound event loop blockages, and they’re really hard to debug. These just aren’t issues that happen in Go.
      - timothycrosley 7y ago
        A lot of what you describe is for CPU bound problems, most of the problems I face personally and the the problem, in particular, the parent mentioned are IO-bound. Async in Python is a near-perfect fit for this kind of problem space and is really easy to reason about and debug as you can step through and debug one line of execution at a time.
        weberc2 7y ago
        Only one of those problems is CPU. The others are type system and IO issues, and they happen in IO bound services too.
- tgv 7y ago
  > on the 100th or so Python stacktrace Go's not perfect in this respect, either. I had your typical map[string][]interface{}, because the data in the map was an array of records, but different records in different cases, and made an error in an append(), so ended up with nested arrays. That's correct according to the type definitions, but unwanted. Generic could have helped here. So I think the language still needs to grow a bit.
  - dx034 7y ago
    I tend to write custom parsers for jsoniter in this case so that I catch any unexpected data types as early as possible.
- weberc2 7y ago
  > In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually. Up until you need to do anything interesting with the deserialized data. :)
- timothycrosley 7y ago
  > This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone. Luckily, you don't have to choose between those two outcomes: https://github.com/python/mypy https://github.com/python/mypy
  - kissgyorgy 7y ago
    Python's optional type hints even with mypy are nowhere near strict/expressive than a statically typed language like Go. Type hints in Python are an afterthought and it shows in a lot of cases.
    - timothycrosley 7y ago
      Do you have any concrete examples?
    - emptysea 7y ago
      Type hints in Python are IMO great and quite expressive! You can create sum types (a bit verbose but they are there). You can introspect them at run time and create things like pydantic[0] and fastapi[1]. Strict null checks with Optional[T] are great! Now I'm not too familiar with Go's type system, but with protocols, Unions, Optional[T] and other generics offered by MyPy[2] I'd wager the type system is __more__ expressive than Go. [0]: https://pydantic-docs.helpmanual.io https://pydantic-docs.helpmanual.io [1]: https://fastapi.tiangolo.com https://fastapi.tiangolo.com [2]: https://mypy.readthedocs.io/en/stable/protocols.html https://mypy.readthedocs.io/en/stable/protocols.html
      - weberc2 7y ago
        The type system is more expressive, but much clunkier and also it needs to be expressive to accommodate all of the weird things people do in Python (and you still can’t model SQLAlchemy types).
- Tempest1981 7y ago
  Regexps is another area where GoLang used to lag behind Python (for certain types of expressions?). Not sure if things have changed, but this issue is still open 4 years later: https://github.com/golang/go/issues/11646 https://github.com/golang/go/issues/11646 Not sure I fully understand the NFA vs DFA/re2 choices -- seem like the issue above conflicts with the comment below. Can someone clarify? From [1]: "Go has chosen to follow the re2 path (not surprising, since Russ Cox is a major author of both Go and re2). re2 has much better performance characteristics than some other regexp engines, in that it never has an exponential slowdown, but that comes at a cost for other regexps (https://swtch.com/~rsc/regexp/ https://swtch.com/~rsc/regexp/)." [1] https://www.reddit.com/r/golang/comments/84o986/why_is_gos_regex_so_slow/ https://www.reddit.com/r/golang/comments/84o986/why_is_gos_r...
- dx034 7y ago
  > Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually. Using encoding/json maybe but also using 3rd party libraries? jsoniter is pretty fast, I just ran some tests and for my use cases it outperformed Python.
leetrout 7y ago
I agree with your sentiment. I still reach for Python (and Pandas) for a quick and dirty, get-it-done program and then when feasible I move over to Go. I am slowly ending up with a library of utils from doing this sort of work. In the end it does end up more stable (and more verbose, which is OK with me) and easier to maintain. I've said for a while Python makes me productive and Go makes my whole team productive.
Exuma 7y ago
I kind of wondered that myself. All the completely automatic marshalling using tags seemed like it assumes things are very nice (which they aren't when working with some of the terrible data sources I've seen). Everything else feels very cool to use Go though, I just used it on my first real project.
- cameronbrown 7y ago
  > I just used it on my first real project. Have you got any relevant advice I can take? Interested to hear how people feel about the language before I dive in.
  - Exuma 7y ago
    Yes I do actually -- my best advice is to watch this course. It's hands down the best course I've ever seen (across any language, actually), and he jumps RIGHT into it, almost from a computer science aspect. He doesn't talk about learning the language at all (he assumes you know all that), instead he fills in all the gaps in a very thorough way that are left out from all the tutorials and such you'll read. I've read dozens and dozens of Go articles while learning and I would say 85% of those 15 hours was NEW information to me, that's how good it is. https://learning.oreilly.com/videos/ultimate-go-programming/9780135261651 https://learning.oreilly.com/videos/ultimate-go-programming/... There are 2 versions, I watched the older version because he's more lively in it, however I have heard the newer version is better because he does cover a lot more up to date tooling. I went and rewatched the section on debugging and structs on the new version. For example, its 15 hour course, and in the literal first 5 minutes of the first coding video, he talks about struct field memory alignment on different CPU architectures. That's when I refilled my coffee and said 'dis gon be good'. Aside from that, I would say pick a good project structure that works well, invest time time into researching the different ways (also covered in that course) I also use go modules which are new, and make things way easier as far as packages. Definitely make sure your editor has gofmt and goimports set to auto-run on save, those are lifesavers. Goimports is smart enough to look across packages and in your own package, which is cool All the other suggestions about actually writing code is basically covered entirely in the ultimate go course (when to choose pointers vs structs, how to structure code, good error handling practices, etc)

axaxs 7y ago

> is it's much more verbose/harder to handle edge cases - it feels like I'm less productive writing the program, but end up building something much more stable than the Python equivalent This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone. > I really need the performance for my workload Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually.

cameronbrown 7y ago
> This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone. Yep. Maybe I just suck at programming but I couldn't write enough tests to avoid not being fed up debugging stack traces in production. I started using Python's type hints, since this was the source of most of the bugs which helped but I'd rather just use a strongly typed language next time. > Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually. I've got a few million records pulled from an API so I'm extremely I/O bound and goroutines are easier to work with than Python concurrency. I've also got reasonably complex data processing beyond just XML/JSON parsing. I think on the whole Go is still going to be faster, but thanks for the heads up.
- timothycrosley 7y ago
  Have you tried async? I find it much easier personally to reason about and debug than channels, and is perfect for IO-bound problems. If you really like channels, note that Python 3.9 will have sub-interpreters and channels as well: https://hackernoon.com/has-the-python-gil-been-slain-9440d28fa93d https://hackernoon.com/has-the-python-gil-been-slain-9440d28...
  - cameronbrown 7y ago
    Unfortunately I deploy on App Engine which is still at Python 3.7.
  - weberc2 7y ago
    Goroutines are quite a lot nicer than async for a few reasons. Firstly, there are no “forgot to await” errors, secondly there is no need to worry about some library making a sync call way down the call stack and therefore blocking your event loop, and thirdly goroutines can use all cores on a CPU to parallelize CPU intensive tasks, which means you don’t block the event loop unless you really are out of CPU. Blocking the event loop is a really big deal since it can cause health checks to fail which can cascale onto other instances and bring your app down. We’ve experienced this with both CPU- and IO-bound event loop blockages, and they’re really hard to debug. These just aren’t issues that happen in Go.
    - timothycrosley 7y ago
      A lot of what you describe is for CPU bound problems, most of the problems I face personally and the the problem, in particular, the parent mentioned are IO-bound. Async in Python is a near-perfect fit for this kind of problem space and is really easy to reason about and debug as you can step through and debug one line of execution at a time.
      - weberc2 7y ago
        Only one of those problems is CPU. The others are type system and IO issues, and they happen in IO bound services too.
tgv 7y ago
> on the 100th or so Python stacktrace Go's not perfect in this respect, either. I had your typical map[string][]interface{}, because the data in the map was an array of records, but different records in different cases, and made an error in an append(), so ended up with nested arrays. That's correct according to the type definitions, but unwanted. Generic could have helped here. So I think the language still needs to grow a bit.
- dx034 7y ago
  I tend to write custom parsers for jsoniter in this case so that I catch any unexpected data types as early as possible.
weberc2 7y ago
> In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually. Up until you need to do anything interesting with the deserialized data. :)
timothycrosley 7y ago
> This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone. Luckily, you don't have to choose between those two outcomes: https://github.com/python/mypy https://github.com/python/mypy
- kissgyorgy 7y ago
  Python's optional type hints even with mypy are nowhere near strict/expressive than a statically typed language like Go. Type hints in Python are an afterthought and it shows in a lot of cases.
  - timothycrosley 7y ago
    Do you have any concrete examples?
  - emptysea 7y ago
    Type hints in Python are IMO great and quite expressive! You can create sum types (a bit verbose but they are there). You can introspect them at run time and create things like pydantic[0] and fastapi[1]. Strict null checks with Optional[T] are great! Now I'm not too familiar with Go's type system, but with protocols, Unions, Optional[T] and other generics offered by MyPy[2] I'd wager the type system is __more__ expressive than Go. [0]: https://pydantic-docs.helpmanual.io https://pydantic-docs.helpmanual.io [1]: https://fastapi.tiangolo.com https://fastapi.tiangolo.com [2]: https://mypy.readthedocs.io/en/stable/protocols.html https://mypy.readthedocs.io/en/stable/protocols.html
    - weberc2 7y ago
      The type system is more expressive, but much clunkier and also it needs to be expressive to accommodate all of the weird things people do in Python (and you still can’t model SQLAlchemy types).
Tempest1981 7y ago
Regexps is another area where GoLang used to lag behind Python (for certain types of expressions?). Not sure if things have changed, but this issue is still open 4 years later: https://github.com/golang/go/issues/11646 https://github.com/golang/go/issues/11646 Not sure I fully understand the NFA vs DFA/re2 choices -- seem like the issue above conflicts with the comment below. Can someone clarify? From [1]: "Go has chosen to follow the re2 path (not surprising, since Russ Cox is a major author of both Go and re2). re2 has much better performance characteristics than some other regexp engines, in that it never has an exponential slowdown, but that comes at a cost for other regexps (https://swtch.com/~rsc/regexp/ https://swtch.com/~rsc/regexp/)." [1] https://www.reddit.com/r/golang/comments/84o986/why_is_gos_regex_so_slow/ https://www.reddit.com/r/golang/comments/84o986/why_is_gos_r...
dx034 7y ago
> Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually. Using encoding/json maybe but also using 3rd party libraries? jsoniter is pretty fast, I just ran some tests and for my use cases it outperformed Python.

leetrout 7y ago

I agree with your sentiment. I still reach for Python (and Pandas) for a quick and dirty, get-it-done program and then when feasible I move over to Go. I am slowly ending up with a library of utils from doing this sort of work. In the end it does end up more stable (and more verbose, which is OK with me) and easier to maintain. I've said for a while Python makes me productive and Go makes my whole team productive.

Exuma 7y ago

I kind of wondered that myself. All the completely automatic marshalling using tags seemed like it assumes things are very nice (which they aren't when working with some of the terrible data sources I've seen). Everything else feels very cool to use Go though, I just used it on my first real project.

cameronbrown 7y ago
> I just used it on my first real project. Have you got any relevant advice I can take? Interested to hear how people feel about the language before I dive in.
- Exuma 7y ago
  Yes I do actually -- my best advice is to watch this course. It's hands down the best course I've ever seen (across any language, actually), and he jumps RIGHT into it, almost from a computer science aspect. He doesn't talk about learning the language at all (he assumes you know all that), instead he fills in all the gaps in a very thorough way that are left out from all the tutorials and such you'll read. I've read dozens and dozens of Go articles while learning and I would say 85% of those 15 hours was NEW information to me, that's how good it is. https://learning.oreilly.com/videos/ultimate-go-programming/9780135261651 https://learning.oreilly.com/videos/ultimate-go-programming/... There are 2 versions, I watched the older version because he's more lively in it, however I have heard the newer version is better because he does cover a lot more up to date tooling. I went and rewatched the section on debugging and structs on the new version. For example, its 15 hour course, and in the literal first 5 minutes of the first coding video, he talks about struct field memory alignment on different CPU architectures. That's when I refilled my coffee and said 'dis gon be good'. Aside from that, I would say pick a good project structure that works well, invest time time into researching the different ways (also covered in that course) I also use go modules which are new, and make things way easier as far as packages. Definitely make sure your editor has gofmt and goimports set to auto-run on save, those are lifesavers. Goimports is smart enough to look across packages and in your own package, which is cool All the other suggestions about actually writing code is basically covered entirely in the ultimate go course (when to choose pointers vs structs, how to structure code, good error handling practices, etc)