14 ms·

Notes on the Go translation of Reposurgeon (2020)

127 points by spekcular 5y ago

whoisjohnkid 5y ago
Raphael Poss did a really insightful talk on how to gracefully panic[0]; this gives try catch like functionality, though should be used sparingly IMO. - [0] https://m.youtube.com/watch?v=c78U0MZ4b_c https://m.youtube.com/watch?v=c78U0MZ4b_c
ynth7 5y ago
What’s the point in reposting the same old Go complaints? It’s esr this time? Respect for his early contributions, but he’s basically what PG would be if he had not become obscenely rich; rehashing old points, relying on history to have a reputation, contributing little novel output. He sure can complain about things he has no intention to put effort into improving though.
whoisjohnkid 5y ago
Regarding your point on iterators, I typically use a channel with context; this lets you exit that range loop early and will automatically stop the related go routine. (assuming it’s context aware) e.g. func myFunc(cx context.Context) { ctx, cancel = context.WithCancel(cx) defer cancel() for item := range iterator(ctx) { } }
- jerf 5y ago
  The major problem I have with that is performance is atrocious [1] if the iterator isn't doing something very nontrivial. Since the vast majority of iterators amount to incrementing at most a handful of things and then returning something indexed by those things, you're paying the cost of channel communication per item but not getting that channel cost amortized over any significant costs in the iteration itself. If the thing using the iterator is also not doing anything significant (i.e., adding integers together), this has very bad performance implications. As a rule of thumb, the amount of work transferred by any concurrency primitive should significantly exceed the cost of the concurrency primitive itself. I do have a couple of uses of this pattern where what is on the other side of the channel is something reading off a network and parsing lines of JSON into internal structs, in which case the overhead of the channel isn't necessarily too bad. (In one case, it even chunks the lines of JSON into a slice of several parsed structs, reducing channel overhead even more.) But it's a terrible solution in general; iterating over an array and doing any sort of very fast "thing" to each element that only costs a handful of assembler instructions, a very common use case, has terrible overhead. It's a real pity, because the semantics of that solution are pretty close to the right answer. But it's a huge performance trap. Something as basic as iteration needs to not be a huge performance trap. [1]: Relative to Go, anyhow. I haven't timed it directly but I wouldn't be surprised that a channel-based iterator like that would be comparable to Python's general iteration speed, or at least not off by a very large factor. It's just that "Python's normal level of performance" is "atrocious Go performance".
  - whoisjohnkid 5y ago
    Hmm, can you provide some examples of how a context based iterator is slow? If you need raw performance and you are just doing some minimal operations on a slice then yeah you’d want to use a simple for loop instead. I typically use this pattern in situations where you have a ton of data coming back where you want to avoid storing all of that data in memory.
    - throwaway894345 5y ago
      The problem with using channels is that these require multiple goroutines and locking for a problem that's inherently singlethreaded. Instead, you can define an iterator as a function that returns a function: func intSliceIter(ints []int) func() (int, bool) { i := 0 return func() (int, bool) { if i < len(ints) { ret := ints[i] i++ return ret, true } return 0, false } } iter := intSliceIter([]int{0, 1, 2, 3, 4}) for x, ok := iter(); ok; x, ok = iter() { fmt.Println(x) } Of course, there's not much benefit to a SliceIter; this is a contrived example, but you can apply this pattern in more complicated cases as well. Similarly, you can define an iterator as an interface (which is similar to bufio.Scanner and a few others in the standard library—a closure is an object is a closure): type IntIter interface { Next() (int, bool) } type IntSliceIter struct { Cursor int Ints []int } func (isi *IntSliceIter) Next() (int, bool) { if isi.Cursor < len(isi.Ints) { ret := isi.Ints[isi.Cursor] isi.Cursor++ return ret, true } return 0, false }
badhombres 5y ago
Great writeup. Some of the points are moot now that generics are being released, but some very valid concerns. I would also add that Enums + Exhaustive Switch is a very very weak area in Go that would really benefit the language a ton. I've used those features in other languages and that's one of the things I miss the most, especially when dealing with a ton of web API's that have a defined set of values for properties. Able to have the confidence that we're checking for every situation that could occur on an enum type across the codebase is one less thing I need to worry about.
- thegeekpirate 5y ago
  > Able to have the confidence that we're checking for every situation that could occur on an enum type across the codebase is one less thing I need to worry about. golangci-lint (https://github.com/golangci/golangci-lint https://github.com/golangci/golangci-lint) is an absolute must, and includes https://github.com/nishanths/exhaustive https://github.com/nishanths/exhaustive which will check this for you.
  - badhombres 5y ago
    thank you, I'm going to take a look at these!
  - henvic 5y ago
    my 50 cents: I'd rather just use go vet + staticcheck. Way simpler, no complex configuration, and no license concerns :) and then you can create a file like the following and add whatever analyzer provides value for your specific case (including this exhaustive), easily. Example: https://github.com/FiloSottile/mkcert/blob/master/analysis.go https://github.com/FiloSottile/mkcert/blob/master/analysis.g...
    - jatone 5y ago
      don't let perfect be the enemy of good. =)
  - preseinger 5y ago
LysPJ 5y ago
Really interesting read. Especially the suggestion for how to support iterators. It should be noted that the writeup is a year old, which (understandably) means that the comments on generics are a little out of date.
wpietri 5y ago
What a delight to read, even as somebody who has barely used Go. I really appreciate the author's self-awareness as demonstrated in things like "Expected problems that weren’t". The bits about exceptions and typing remind me of an open question I have about Go: to me it mainly looks like a language for well-understood problems. Static typing and the lack of ability to do broad, high-level exception catching seem to me to be well-matched for problems where you already understand the solution pretty well. E.g., a port like this. But how is it for poorly understood problems? E.g., You're doing a startup where the technical risks are low and the major unknowns are about user needs and the correct experiences to deliver. Or you're doing exploratory work for an art piece. In those contexts, I've felt much more effective starting with vague/implicit typing and some broad, high-level exception-catching blocks. That lets me avoid a bunch of questions about the right types and structures until a more solid domain vocabulary emerges. At that point you can refactor/rewrite toward clarity. But I don't see a good way to be willfully vague/casual in Go.
- whoisjohnkid 5y ago
  All depends on what’s important to you in that startup context. If you need a good mix of high productivity and low costs on the server side I think go is a good fit. If you don’t care so much about costs and scale on the server side then something like Ruby on Rails might give you that extra productivity boost you need to verify the concept.
  - wpietri 5y ago
    Yeah, given how computers keep getting cheaper, I only start caring about low server costs once I have a demonstrated cost problem that's going to be material to the business model or to the budget. Until then I'm very happy to burn CPU time to accelerate developers. Who keep getting more expensive! And yes, the point of verifying the concept is key for me in this. Before product-market fit, I just don't have much confidence in any domain model. Once we have demonstrated that particular people are excited to pay for a particular thing, that changes. Then we can understand how those people think, and how we think about their behavior, such that we can do real domain-driven design. At that point I'm much more willing to lock down types.
nicoburns 5y ago
The article mentions that Go, OCaml, or a compiled lisp were considered for this project. I wonder why Rust wasn't on that list. It seems to cover every concern that is raised here except for keyword arguments (which are high on my list of desired Rust features too). I guess maybe they were worried about the lack of GC, but my experience has been that's it's generally quite easy to port code from dynamic languages like Python/JS/PHP to Rust. So I suspect that may have gone into their "Expected problems that weren’t" section if they'd tried it.
- mwcampbell 5y ago
  Here is what esr wrote about Rust in early 2017: http://esr.ibiblio.org/?p=7294 http://esr.ibiblio.org/?p=7294 http://esr.ibiblio.org/?p=7303 http://esr.ibiblio.org/?p=7303
  - kubb 5y ago
    It sounds that it was hard to learn and use for him, that's why he preferred Go and Python. It's a frequent complaint and the step from Python to Go is indeed easier than from Python to Rust. He also had some philosophical concerns about the crate system.
  - brandmeyer 5y ago
    > A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying. This is exactly the impression I have gotten from the crates system every time I've looked. It happens for things as foundational as mmap. Also terrifying: The plethora of highly-recommended crates that have not yet committed to a stable API (ie, are still semantically on version 0.X).
    - throwaway894345 5y ago
      I agree, but at the same time things tend to stabilize over time as the community settles around the best packages. The real thing to fear is the relative instability of Rust at the present moment (for certain things, like async), but the overall trajectory seems promising.
deleted 5y ago
[deleted]
mwcampbell 5y ago
Personally, I would have gone with Kotlin for a translation like this. It's a closer match to Python than Go, and I think his rule swarm [1] approach to semi-automated translation would have been effective. But then, he might not have liked the fact that, like Python 3 and unlike Go, the Java platform treats strings (particularly filenames) as a sequence of Unicode code points rather than bytes. [1]: http://esr.ibiblio.org/?p=8153 http://esr.ibiblio.org/?p=8153
- colin_mccabe 5y ago
  Go treats strings as a sequence of unicode code points.
  - fnoof 5y ago
    I believe they’re treated as byte sequences: https://go.dev/ref/spec#String_types https://go.dev/ref/spec#String_types
    - ustolemyname 5y ago
      Not treated as, only backed by byte sequences. If you range over them you get a series of rune values that can be multi-byte code points. There's a few other differences, strings cannot be directly mutated or have the address of their elements taken.
- dbt00 5y ago
  > It's a closer match to Python than Go I like Kotlin, but I really think of Go as the most probable outcome of someone saying "I'd like a staticly typed, concurrent/parallel Python", and then mumbled "but I hate exceptions". Go is very close to python in many respects.
kubb 5y ago
> What I really missed was generic map-function-over-slice Me too, and mapping a map, a channel, etc. With generics, things will be easier, we can have iterators, even lazy iterators, but there will still be no type inference in lambdas for arguments and return values and no easy currying, so things will still be awkward, but better. > Keyword arguments You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer. > Annoying limitations on const I think the Go constants are good, the fact that you can perform calculations on them in compile time also. But I wish there was some way to express immutability (and non-nilability) in the language. Programs are simpler with less moving parts. > 14KLOC -> 21KLOC That's to be expected, Go could be nicer if it was more concise, obviously that's a hard thing to balance, because you can end up with too obscure code, I just wish that there were some small improvements from time to time. > Absence of sum/discriminated-union types Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks. > Catchable exceptions require silly contortions I don't like exceptions, they disrupt the flow of the program - any function call could "return early", so you need to be always careful and account for that possibility. In C++ this was solved with RAAI, and it was a source of so many bugs that some companies just disallow exceptions internally. > Aesthetic doubt Yeah, it's there, some things in Go just come out ugly. > Absence of iterators Yes. Hopefully generics will enable implementing them.
- autarch 5y ago
  > You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer. I think using a builder is a better option in Go if you have more than 2-3 arguments. > Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks. Except you can't really do this with interfaces. If you need to get back to the concrete implementation of the interface, there's no way to know what all possible implementations might be. This is especially true since any type can implement an interface, not just the types you create initially. So someone else could implement `Tree` and you would not know to account for that in your function that accepts the `Tree` interface. I don't think there's any substitute for proper sum types in Go.
JulianMorrison 5y ago
FWIW, Go _does_ have keyword struct members, so it's possible to do type A struct { B int C int } myfunc(A{B: 1, C: 2})
- throwaway894345 5y ago
  I do this a lot. You can even do `A{B: 1, C: 2}.myfunc()` in many cases.
azth 5y ago
It's not surprising that the code base grew when reimplemented in golang. It would have probably been even shorter had it been rewritten in Python itself. Just the other day, I was able to condense over 15 lines of golang code into 3 lines (could also have been 2 lines) in a Python-like syntax, both reducing code length, and substantially increasing readability as it would make the underlying logic clearly stand out instead of having several loop and map constructs.
- throwaway894345 5y ago
  I think you have to be very careful before claiming readability gains from fewer lines of code. I've done a lot of Python programming and for simple things like mapping a single function over a list, a list comprehension or generator expression can indeed be a tiny bit more readable than a for loop, but combining that with some filtering or flattening and colleagues (and my future self) get frustrated. I've seen the same thing with overly fancy iterator chaining in Rust. There's some temptation to think that terse==readable, but in practice this rarely extends beyond the simplest cases.
  - formerly_proven 5y ago
    I didn't yet get around to writing Go in any significant capacity (it's not in an interesting spot for me) but I've had to read some and to me it seems like about half of all Go code does approximately nothing. It seems really weird to me that Go doesn't have a Result/Optional type (or sum types in general) and instead prefers multiple return values instead, and consequentially does not have error-handling operators but requires if err != nil repeated approximately 9000 times per file.
    - throwaway894345 5y ago
      This seems like a real sticking point for a lot of people. Personally, I don't seem to think about code volume in units of "lines" but rather in units of "complexity" and "locality". I would like to see Result and Optional types, but in practice everyone understands that (nil, err) is the same as Result::Err and that (*T, nil) is the same as Result::Ok. Of course, you miss out on monadic properties, but after extensively using the language, that's a feature 99% of the time. I would say give it a shot; I was skeptical at first, but this is surprisingly not an issue IMO. The real issues (IMO) with Go error handling are the clunky `errors.Is()` and `errors.As()` functions for catching specific error values and types respectively as well as annotating errors to get the requisite context (this may already have a solution which just hasn't yet become idiomatic across the ecosystem). In the meanwhile, I just wrap errors with extra context which isn't super satisfying but largely does the trick, e.g., `return fmt.Errorf("writing to database: %w", err)`. Rust seems to have similar problems with respect to different patterns and libraries for error handling, despite having a standard Result type.
nauticacom 5y ago
One thing that the people saying "just use a struct for keyword arguments" are missing is that structs should signal intent, i.e. "this is a concrete concept in the system." A Repository, Commit, Message, Person, etc. struct are all concepts in the domain, whereas "the arguments for this particular function" is not. I think Go people are allergic to writing code that does anything other than functionally work.
- jasonhansel 5y ago
  IMHO, if the arguments to a function are just a bunch of heterogeneous values that cannot be sensibly grouped together, then that's usually a sign that the function needs to be refactored (e.g. because it's trying to do a bunch of unrelated things, or because it couples tasks together in an unnatural way).
- scndrycntct 5y ago
  > I think Go people are allergic to writing code that does anything other than functionally work. I think that's what Go was designed for, as a language. To be readable, usable. It's uncaring for your personal programming philosophies. I think that's why it's been successful.
  - danudey 5y ago
    Specifically, Go was designed to be effectively and safely writeable and readable by mediocre programmers.
    - jhawk28 5y ago
      Mediocre programmers can still write bad Go code. Their code is neither safe or readable.
      - jrockway 5y ago
        Don't take the obvious flamebait. HN's favorite comment on any Go article is "it's designed for bad programmers", implying that if you like it, you're bad at programming. I'm good at programming and like Go, so that implication is clearly false.
danudey 5y ago
Reposurgeon was a pretty enticing option for us, as we're currently in the migration of SVN to Git, and the string of tools we have is kind of complicated to explain to newcomers. Currently the workflow is svn-all-fast-export (SVN to Git) -> git filter-repo (trim content from the repository) -> Git LFS (store large files out-of-band), where each stage has to be manually tested and written, and then hopefully put into a shell script or Bash history or something. Unfortunately, I have yet to get Reposurgeon working for us at all. The documentation was inaccurate, in that it referred to things which only existed in the Go version, and the Go version went OOM (on a server with 384 GB of RAM) on every repository I've tried it on. This includes just reading from an existing, pruned svndump file, so it's not resource contention either. Basically, the tool is great conceptually, but it absolutely does not work for our use case. We had to stick with our existing, messy, multi-component solution, which has worked surprisingly well over the last two years, and since I'm the one doing most of the processing, having kind of a messy system is relatively acceptable for the time being.
- db48x 5y ago
  We’d be interested in hearing more about your situation, so that we can further improve Reposurgeon. If you can profile it as it reads your svn dump (using the `profile start` and `profile save` commands), it could really help us out. How many commits does your repository have? The largest one we’ve successfully converted from SVN to Git ourselves was 287k commits.
  - danudey 5y ago
    Sorry for the late reply, but: 1. I'll try doing some profiling and see what happens 2. The latest repository I tried to import has 671,190 revisions
    - db48x 5y ago
      Wow. That is a huge repository. You can also force it to write all file content (blobs) out to temporary files rather than keeping them in memory by reading the stream from standard input: reposurgeon "read -" … <dump.svn That will be slower and use more disk space, but maybe it will fit.
nemo1618 5y ago
What I really missed was generic map-function-over-slice, which could be handled by adding a much narrower feature. If one graded possible Go point extensions by a figure of merit in which the numerator is "how much Python expressiveness this keeps" and the denominator is "how simple and self-contained the Go feature would be", I think this one would be top of list. So: map as a functional builtin takes two arguments, one x = []T and a second f = func(T)T. The expression map(x, f) yields a new slice in which for each element of x, f(x) is appended. I think this is an excellent way of evaluating new features, and it represents a real missed opportunity for Go to explore new PL territory. Instead, we're getting full-blown user-defined generics, which increases the "denominator" far, far more than it increases the "numerator."
- throwaway894345 5y ago
  I also came from Python (15 years of experience) to Go and I think newcomers to Go index too hard on terseness. The for-loop equivalent for a map over a list is more characters, but it's really straightforward and easily recognizable in the code. In the general case, I'm glad that Go doesn't try to explore new PL territory, optimizing instead for things that are known to improve developer productivity. None of this is to say that Go has no room for improvement--only that "terseness" is not high on the list of improvements I'd like to see (for example, I'd rather have sum types with exhaustive pattern matching).
  - nemo1618 5y ago
    For me, it's not a question of terseness -- it's more about communicating intent, and not polluting the scope with incidental variables. Everyone knows what map/filter/reduce do. When reading new code, seeing "map" is better than seeing a for loop: you don't have to think about the underlying iteration at all, you can skip directly to the essence of the transformation. As a side effect of this, when you do see a for loop, you can safely assume that something about the iteration is non-trivial: maybe the loop exits early, for example. I wrote a bit more about this here: https://twitter.com/lukechampine/status/1463715093733122054 https://twitter.com/lukechampine/status/1463715093733122054 btw, by "new PL territory" I mean "reifying a small set of container operations, without supporting user-defined generics," which to my knowledge is not a position taken by any mainstream language.

127 points by spekcular 5y ago

Raphael Poss did a really insightful talk on how to gracefully panic[0]; this gives try catch like functionality, though should be used sparingly IMO. - [0] https://m.youtube.com/watch?v=c78U0MZ4b_c https://m.youtube.com/watch?v=c78U0MZ4b_c

ynth7 5y ago

What’s the point in reposting the same old Go complaints? It’s esr this time? Respect for his early contributions, but he’s basically what PG would be if he had not become obscenely rich; rehashing old points, relying on history to have a reputation, contributing little novel output. He sure can complain about things he has no intention to put effort into improving though.

whoisjohnkid 5y ago

Regarding your point on iterators, I typically use a channel with context; this lets you exit that range loop early and will automatically stop the related go routine. (assuming it’s context aware) e.g. func myFunc(cx context.Context) { ctx, cancel = context.WithCancel(cx) defer cancel() for item := range iterator(ctx) { } }

jerf 5y ago
The major problem I have with that is performance is atrocious [1] if the iterator isn't doing something very nontrivial. Since the vast majority of iterators amount to incrementing at most a handful of things and then returning something indexed by those things, you're paying the cost of channel communication per item but not getting that channel cost amortized over any significant costs in the iteration itself. If the thing using the iterator is also not doing anything significant (i.e., adding integers together), this has very bad performance implications. As a rule of thumb, the amount of work transferred by any concurrency primitive should significantly exceed the cost of the concurrency primitive itself. I do have a couple of uses of this pattern where what is on the other side of the channel is something reading off a network and parsing lines of JSON into internal structs, in which case the overhead of the channel isn't necessarily too bad. (In one case, it even chunks the lines of JSON into a slice of several parsed structs, reducing channel overhead even more.) But it's a terrible solution in general; iterating over an array and doing any sort of very fast "thing" to each element that only costs a handful of assembler instructions, a very common use case, has terrible overhead. It's a real pity, because the semantics of that solution are pretty close to the right answer. But it's a huge performance trap. Something as basic as iteration needs to not be a huge performance trap. [1]: Relative to Go, anyhow. I haven't timed it directly but I wouldn't be surprised that a channel-based iterator like that would be comparable to Python's general iteration speed, or at least not off by a very large factor. It's just that "Python's normal level of performance" is "atrocious Go performance".
- whoisjohnkid 5y ago
  Hmm, can you provide some examples of how a context based iterator is slow? If you need raw performance and you are just doing some minimal operations on a slice then yeah you’d want to use a simple for loop instead. I typically use this pattern in situations where you have a ton of data coming back where you want to avoid storing all of that data in memory.
  - throwaway894345 5y ago
    The problem with using channels is that these require multiple goroutines and locking for a problem that's inherently singlethreaded. Instead, you can define an iterator as a function that returns a function: func intSliceIter(ints []int) func() (int, bool) { i := 0 return func() (int, bool) { if i < len(ints) { ret := ints[i] i++ return ret, true } return 0, false } } iter := intSliceIter([]int{0, 1, 2, 3, 4}) for x, ok := iter(); ok; x, ok = iter() { fmt.Println(x) } Of course, there's not much benefit to a SliceIter; this is a contrived example, but you can apply this pattern in more complicated cases as well. Similarly, you can define an iterator as an interface (which is similar to bufio.Scanner and a few others in the standard library—a closure is an object is a closure): type IntIter interface { Next() (int, bool) } type IntSliceIter struct { Cursor int Ints []int } func (isi *IntSliceIter) Next() (int, bool) { if isi.Cursor < len(isi.Ints) { ret := isi.Ints[isi.Cursor] isi.Cursor++ return ret, true } return 0, false }

badhombres 5y ago

Great writeup. Some of the points are moot now that generics are being released, but some very valid concerns. I would also add that Enums + Exhaustive Switch is a very very weak area in Go that would really benefit the language a ton. I've used those features in other languages and that's one of the things I miss the most, especially when dealing with a ton of web API's that have a defined set of values for properties. Able to have the confidence that we're checking for every situation that could occur on an enum type across the codebase is one less thing I need to worry about.

thegeekpirate 5y ago
> Able to have the confidence that we're checking for every situation that could occur on an enum type across the codebase is one less thing I need to worry about. golangci-lint (https://github.com/golangci/golangci-lint https://github.com/golangci/golangci-lint) is an absolute must, and includes https://github.com/nishanths/exhaustive https://github.com/nishanths/exhaustive which will check this for you.
- badhombres 5y ago
  thank you, I'm going to take a look at these!
- henvic 5y ago
  my 50 cents: I'd rather just use go vet + staticcheck. Way simpler, no complex configuration, and no license concerns :) and then you can create a file like the following and add whatever analyzer provides value for your specific case (including this exhaustive), easily. Example: https://github.com/FiloSottile/mkcert/blob/master/analysis.go https://github.com/FiloSottile/mkcert/blob/master/analysis.g...
  - jatone 5y ago
    don't let perfect be the enemy of good. =)
- preseinger 5y ago

LysPJ 5y ago

Really interesting read. Especially the suggestion for how to support iterators. It should be noted that the writeup is a year old, which (understandably) means that the comments on generics are a little out of date.

wpietri 5y ago

What a delight to read, even as somebody who has barely used Go. I really appreciate the author's self-awareness as demonstrated in things like "Expected problems that weren’t". The bits about exceptions and typing remind me of an open question I have about Go: to me it mainly looks like a language for well-understood problems. Static typing and the lack of ability to do broad, high-level exception catching seem to me to be well-matched for problems where you already understand the solution pretty well. E.g., a port like this. But how is it for poorly understood problems? E.g., You're doing a startup where the technical risks are low and the major unknowns are about user needs and the correct experiences to deliver. Or you're doing exploratory work for an art piece. In those contexts, I've felt much more effective starting with vague/implicit typing and some broad, high-level exception-catching blocks. That lets me avoid a bunch of questions about the right types and structures until a more solid domain vocabulary emerges. At that point you can refactor/rewrite toward clarity. But I don't see a good way to be willfully vague/casual in Go.

whoisjohnkid 5y ago
All depends on what’s important to you in that startup context. If you need a good mix of high productivity and low costs on the server side I think go is a good fit. If you don’t care so much about costs and scale on the server side then something like Ruby on Rails might give you that extra productivity boost you need to verify the concept.
- wpietri 5y ago
  Yeah, given how computers keep getting cheaper, I only start caring about low server costs once I have a demonstrated cost problem that's going to be material to the business model or to the budget. Until then I'm very happy to burn CPU time to accelerate developers. Who keep getting more expensive! And yes, the point of verifying the concept is key for me in this. Before product-market fit, I just don't have much confidence in any domain model. Once we have demonstrated that particular people are excited to pay for a particular thing, that changes. Then we can understand how those people think, and how we think about their behavior, such that we can do real domain-driven design. At that point I'm much more willing to lock down types.

nicoburns 5y ago

The article mentions that Go, OCaml, or a compiled lisp were considered for this project. I wonder why Rust wasn't on that list. It seems to cover every concern that is raised here except for keyword arguments (which are high on my list of desired Rust features too). I guess maybe they were worried about the lack of GC, but my experience has been that's it's generally quite easy to port code from dynamic languages like Python/JS/PHP to Rust. So I suspect that may have gone into their "Expected problems that weren’t" section if they'd tried it.

mwcampbell 5y ago
Here is what esr wrote about Rust in early 2017: http://esr.ibiblio.org/?p=7294 http://esr.ibiblio.org/?p=7294 http://esr.ibiblio.org/?p=7303 http://esr.ibiblio.org/?p=7303
- kubb 5y ago
  It sounds that it was hard to learn and use for him, that's why he preferred Go and Python. It's a frequent complaint and the step from Python to Go is indeed easier than from Python to Rust. He also had some philosophical concerns about the crate system.
- brandmeyer 5y ago
  > A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying. This is exactly the impression I have gotten from the crates system every time I've looked. It happens for things as foundational as mmap. Also terrifying: The plethora of highly-recommended crates that have not yet committed to a stable API (ie, are still semantically on version 0.X).
  - throwaway894345 5y ago
    I agree, but at the same time things tend to stabilize over time as the community settles around the best packages. The real thing to fear is the relative instability of Rust at the present moment (for certain things, like async), but the overall trajectory seems promising.

deleted 5y ago

[deleted]

mwcampbell 5y ago

Personally, I would have gone with Kotlin for a translation like this. It's a closer match to Python than Go, and I think his rule swarm [1] approach to semi-automated translation would have been effective. But then, he might not have liked the fact that, like Python 3 and unlike Go, the Java platform treats strings (particularly filenames) as a sequence of Unicode code points rather than bytes. [1]: http://esr.ibiblio.org/?p=8153 http://esr.ibiblio.org/?p=8153

colin_mccabe 5y ago
Go treats strings as a sequence of unicode code points.
- fnoof 5y ago
  I believe they’re treated as byte sequences: https://go.dev/ref/spec#String_types https://go.dev/ref/spec#String_types
  - ustolemyname 5y ago
    Not treated as, only backed by byte sequences. If you range over them you get a series of rune values that can be multi-byte code points. There's a few other differences, strings cannot be directly mutated or have the address of their elements taken.
dbt00 5y ago
> It's a closer match to Python than Go I like Kotlin, but I really think of Go as the most probable outcome of someone saying "I'd like a staticly typed, concurrent/parallel Python", and then mumbled "but I hate exceptions". Go is very close to python in many respects.

kubb 5y ago

> What I really missed was generic map-function-over-slice Me too, and mapping a map, a channel, etc. With generics, things will be easier, we can have iterators, even lazy iterators, but there will still be no type inference in lambdas for arguments and return values and no easy currying, so things will still be awkward, but better. > Keyword arguments You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer. > Annoying limitations on const I think the Go constants are good, the fact that you can perform calculations on them in compile time also. But I wish there was some way to express immutability (and non-nilability) in the language. Programs are simpler with less moving parts. > 14KLOC -> 21KLOC That's to be expected, Go could be nicer if it was more concise, obviously that's a hard thing to balance, because you can end up with too obscure code, I just wish that there were some small improvements from time to time. > Absence of sum/discriminated-union types Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks. > Catchable exceptions require silly contortions I don't like exceptions, they disrupt the flow of the program - any function call could "return early", so you need to be always careful and account for that possibility. In C++ this was solved with RAAI, and it was a source of so many bugs that some companies just disallow exceptions internally. > Aesthetic doubt Yeah, it's there, some things in Go just come out ugly. > Absence of iterators Yes. Hopefully generics will enable implementing them.

autarch 5y ago
> You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer. I think using a builder is a better option in Go if you have more than 2-3 arguments. > Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks. Except you can't really do this with interfaces. If you need to get back to the concrete implementation of the interface, there's no way to know what all possible implementations might be. This is especially true since any type can implement an interface, not just the types you create initially. So someone else could implement `Tree` and you would not know to account for that in your function that accepts the `Tree` interface. I don't think there's any substitute for proper sum types in Go.

JulianMorrison 5y ago

FWIW, Go _does_ have keyword struct members, so it's possible to do type A struct { B int C int } myfunc(A{B: 1, C: 2})

throwaway894345 5y ago
I do this a lot. You can even do `A{B: 1, C: 2}.myfunc()` in many cases.

azth 5y ago

It's not surprising that the code base grew when reimplemented in golang. It would have probably been even shorter had it been rewritten in Python itself. Just the other day, I was able to condense over 15 lines of golang code into 3 lines (could also have been 2 lines) in a Python-like syntax, both reducing code length, and substantially increasing readability as it would make the underlying logic clearly stand out instead of having several loop and map constructs.

throwaway894345 5y ago
I think you have to be very careful before claiming readability gains from fewer lines of code. I've done a lot of Python programming and for simple things like mapping a single function over a list, a list comprehension or generator expression can indeed be a tiny bit more readable than a for loop, but combining that with some filtering or flattening and colleagues (and my future self) get frustrated. I've seen the same thing with overly fancy iterator chaining in Rust. There's some temptation to think that terse==readable, but in practice this rarely extends beyond the simplest cases.
- formerly_proven 5y ago
  I didn't yet get around to writing Go in any significant capacity (it's not in an interesting spot for me) but I've had to read some and to me it seems like about half of all Go code does approximately nothing. It seems really weird to me that Go doesn't have a Result/Optional type (or sum types in general) and instead prefers multiple return values instead, and consequentially does not have error-handling operators but requires if err != nil repeated approximately 9000 times per file.
  - throwaway894345 5y ago
    This seems like a real sticking point for a lot of people. Personally, I don't seem to think about code volume in units of "lines" but rather in units of "complexity" and "locality". I would like to see Result and Optional types, but in practice everyone understands that (nil, err) is the same as Result::Err and that (*T, nil) is the same as Result::Ok. Of course, you miss out on monadic properties, but after extensively using the language, that's a feature 99% of the time. I would say give it a shot; I was skeptical at first, but this is surprisingly not an issue IMO. The real issues (IMO) with Go error handling are the clunky `errors.Is()` and `errors.As()` functions for catching specific error values and types respectively as well as annotating errors to get the requisite context (this may already have a solution which just hasn't yet become idiomatic across the ecosystem). In the meanwhile, I just wrap errors with extra context which isn't super satisfying but largely does the trick, e.g., `return fmt.Errorf("writing to database: %w", err)`. Rust seems to have similar problems with respect to different patterns and libraries for error handling, despite having a standard Result type.

nauticacom 5y ago

One thing that the people saying "just use a struct for keyword arguments" are missing is that structs should signal intent, i.e. "this is a concrete concept in the system." A Repository, Commit, Message, Person, etc. struct are all concepts in the domain, whereas "the arguments for this particular function" is not. I think Go people are allergic to writing code that does anything other than functionally work.

jasonhansel 5y ago
IMHO, if the arguments to a function are just a bunch of heterogeneous values that cannot be sensibly grouped together, then that's usually a sign that the function needs to be refactored (e.g. because it's trying to do a bunch of unrelated things, or because it couples tasks together in an unnatural way).
scndrycntct 5y ago
> I think Go people are allergic to writing code that does anything other than functionally work. I think that's what Go was designed for, as a language. To be readable, usable. It's uncaring for your personal programming philosophies. I think that's why it's been successful.
- danudey 5y ago
  Specifically, Go was designed to be effectively and safely writeable and readable by mediocre programmers.
  - jhawk28 5y ago
    Mediocre programmers can still write bad Go code. Their code is neither safe or readable.
    - jrockway 5y ago
      Don't take the obvious flamebait. HN's favorite comment on any Go article is "it's designed for bad programmers", implying that if you like it, you're bad at programming. I'm good at programming and like Go, so that implication is clearly false.

danudey 5y ago

Reposurgeon was a pretty enticing option for us, as we're currently in the migration of SVN to Git, and the string of tools we have is kind of complicated to explain to newcomers. Currently the workflow is svn-all-fast-export (SVN to Git) -> git filter-repo (trim content from the repository) -> Git LFS (store large files out-of-band), where each stage has to be manually tested and written, and then hopefully put into a shell script or Bash history or something. Unfortunately, I have yet to get Reposurgeon working for us at all. The documentation was inaccurate, in that it referred to things which only existed in the Go version, and the Go version went OOM (on a server with 384 GB of RAM) on every repository I've tried it on. This includes just reading from an existing, pruned svndump file, so it's not resource contention either. Basically, the tool is great conceptually, but it absolutely does not work for our use case. We had to stick with our existing, messy, multi-component solution, which has worked surprisingly well over the last two years, and since I'm the one doing most of the processing, having kind of a messy system is relatively acceptable for the time being.

db48x 5y ago
We’d be interested in hearing more about your situation, so that we can further improve Reposurgeon. If you can profile it as it reads your svn dump (using the `profile start` and `profile save` commands), it could really help us out. How many commits does your repository have? The largest one we’ve successfully converted from SVN to Git ourselves was 287k commits.
- danudey 5y ago
  Sorry for the late reply, but: 1. I'll try doing some profiling and see what happens 2. The latest repository I tried to import has 671,190 revisions
  - db48x 5y ago
    Wow. That is a huge repository. You can also force it to write all file content (blobs) out to temporary files rather than keeping them in memory by reading the stream from standard input: reposurgeon "read -" … <dump.svn That will be slower and use more disk space, but maybe it will fit.

nemo1618 5y ago

What I really missed was generic map-function-over-slice, which could be handled by adding a much narrower feature. If one graded possible Go point extensions by a figure of merit in which the numerator is "how much Python expressiveness this keeps" and the denominator is "how simple and self-contained the Go feature would be", I think this one would be top of list. So: map as a functional builtin takes two arguments, one x = []T and a second f = func(T)T. The expression map(x, f) yields a new slice in which for each element of x, f(x) is appended. I think this is an excellent way of evaluating new features, and it represents a real missed opportunity for Go to explore new PL territory. Instead, we're getting full-blown user-defined generics, which increases the "denominator" far, far more than it increases the "numerator."

throwaway894345 5y ago
I also came from Python (15 years of experience) to Go and I think newcomers to Go index too hard on terseness. The for-loop equivalent for a map over a list is more characters, but it's really straightforward and easily recognizable in the code. In the general case, I'm glad that Go doesn't try to explore new PL territory, optimizing instead for things that are known to improve developer productivity. None of this is to say that Go has no room for improvement--only that "terseness" is not high on the list of improvements I'd like to see (for example, I'd rather have sum types with exhaustive pattern matching).
- nemo1618 5y ago
  For me, it's not a question of terseness -- it's more about communicating intent, and not polluting the scope with incidental variables. Everyone knows what map/filter/reduce do. When reading new code, seeing "map" is better than seeing a for loop: you don't have to think about the underlying iteration at all, you can skip directly to the essence of the transformation. As a side effect of this, when you do see a for loop, you can safely assume that something about the iteration is non-trivial: maybe the loop exits early, for example. I wrote a bit more about this here: https://twitter.com/lukechampine/status/1463715093733122054 https://twitter.com/lukechampine/status/1463715093733122054 btw, by "new PL territory" I mean "reifying a small set of container operations, without supporting user-defined generics," which to my knowledge is not a position taken by any mainstream language.