8 ms·

Go nulls and SQL

• tedunangst 4y ago
  > Pointers are easy first, but then you realize that you have to put nilness checks everywhere. Well, yeah, if something may or may not be there, you have to check before you use it. How else do people want it to work?
  • xdfgh1112 4y ago
    Sane languages have pointers which either can't be null or require you to check explicitly. Rather than leaving it to explode at runtime.
    • tedunangst 4y ago
      Tell me more about these unexplodable languages.
      • giraffe_lady 4y ago
        Maybe Some other time.
      • dtech 4y ago
        If this question is in good faith, for example Kotlin differentiates between the nullable T? and the non-nullable T. An if x != null check automatically casts T? to T. You can't call a method on T?, preventing null pointer exceptions. C# and Typescript have similar language support. Other languages push users away from the problem by favoring monadic Optional/Maybe solutions.
        • adastra22 4y ago
          T? is just syntactic sugar for Optional/Maybe, no?
          • MrJohz 4y ago
            Roughly, yes. But that's the point: an Optional type along with static typing preventing misuse can prevent these sorts of errors entirely.
          • shirogane86x 4y ago
            I don't think that's quite true. It's sort of the same in how it's used, but the main difference is that T? (or T | null) is a union, and Optional/Maybe/Option is a sum type. Out of all the implementations I've seen, T? can't be nested - you can't have T??, whereas it's quite possible to have a Option<Option<T>> (and it's sometimes useful).
          • dtech 4y ago
            It's semantically similar but not syntactic sugar, since it is not translated to Optional or wrapper types, so it also doesn't incur the overhead of Optional. I actually like the language support more, as it flows a lot better than the (flat)map chain you get in monadic style. Similar to async/await v.s. Future.
        • tedunangst 4y ago
          There are some differences here, but you can't for instance use a string pointer with string functions in go either. IMO The distinction between pointer and optional is not as vast as people make it out to be. (But thanks, I overlooked T? conversions inside if.)
          • morelisp 4y ago
            You can call a value method on a nil pointer and it panics, though, which I think was a mistake.
          • akavi 4y ago
            The distinction between pointer and optional is not vast. The distinction between pointer and not optional is. The whole idea of optional is to make not optional possible.
          • ricardobeat 4y ago
            No you can't, but Go will happily de-reference the pointer without a null check, and your "string" will blow up in production: https://go.dev/play/p/7EVa7q6VgIM https://go.dev/play/p/7EVa7q6VgIM
          • nemothekid 4y ago
            >The distinction between pointer and optional is not as vast as people make it out to be. The billion dollar mistake is overblown?
      • adastra22 4y ago
        Rust being the popular example. But even C++ has this with std::optional, for example.
        • tedunangst 4y ago
          So rust doesn't explode if I unwrap None?
          • avgcorrection 4y ago
            The comment that you respondend to: > > Sane languages have pointers which either can't be null or require you to check explicitly. You cannot try to dereference a nullable pointer in safe Rust. It has got nothing to do with Optional (Some/None).
            • tedunangst 4y ago
              That would make a rust pointer rather less useful for representing sql null, no?
              • imron 4y ago
                Correct, which is why you'd use Option<T> instead.
              • avgcorrection 4y ago
                What? The comment that you -- with aloof disbelief -- replied to only talked about pointers. I don't know what point you think you are proving with these pithy replies. It sure does go above my head. :)
                • tedunangst 4y ago
                  Well, this is a thread about storing sql null. I'm trying to suss out how sane languages represent sql null in a way that can't go wrong. So far the answers mostly seem to be use a type that can't represent sql null or use a type that explodes when you use it wrong. With a side of real programmers don't let bugs pass code review.
      • worik 4y ago
        You cannot use a null pointer in Rust. I am a Rust novice but I have not found a way to get to uninitialised memory in safe Rust. Yet.
        • tedunangst 4y ago
          How do you create a pointer to uninitialized memory in go?
          • Gwypaas 4y ago
            You can't. But a runtime panic also very easily causes an outage. At least it's not as subtle. Still. Rust is infinitely better in this regard. Go feels like a crude hammer, especially regarding the default interaction between deserialization and missing struct members.
          • TheDong 4y ago
            All it takes is a specially formatted comment /* int g() { volatile int x; return x; } */ import "C" import "fmt" func main() { fmt.Printf("%d\n", C.g()) } But I know you'll say that 'import "C"' or 'import "unsafe"' is the same thing as using an unsafe block in rust or such, and really shouldn't count against go. Which is fair and true, but you're chasing down a pointless detail. The point isn't that go is memory unsafe. It's not. The point is that Go's type-system is not powerful enough to express various types of type-safety, and as such it's an error-prone language where you can expect null pointer exceptions frequently.
          • stouset 4y ago
            You can't. But you can absolutely have a `nil` that will unavoiad runtime panics if you don't check it. And further, you only can't do this because go defines by fiat that the zero-value of a type must be legal. Of course, this is wildly inconvenient and annoying for many types, including those that come bundled in the standard library. There are other, very convincingly better options to this that many in this thread have been trying to teach you about in the expectation that your responses have been in good faith.
        • 4y ago
      • stouset 4y ago
        Type systems can enforce that types always contain a value of that type and cannot be null. There is another type that is either null or a value of the enclosed type. At any boundary where something might be null, you do the null check. If it's null, you do whatever logic is necessary right there and only right there. That might be to skip the computation, to use a default value, to get the value from somewhere else, to return an error, or whatever. If it's not null, you use the internal value and from then on every user can operate on a guarantee that it's not null.
      • saghm 4y ago
        >> Sane languages have pointers which either can't be null or require you to check explicitly. Rather than leaving it to explode at runtime. > Tell me more about these unexplodable languages. I don't think anyone else mentioned an entire language being unexplodable, just a pointer type. Later on the thread you talk about `Option` in Rust, which is not what I think most people would consider a pointer type, but even if it is, it's certainly not the only one. I think GP was talking about the basic reference type (i.e. `&T`), which will not ever be null in safe code, so dereferencing it will not explode.
      • anonymoushn 4y ago
        Zig encodes optionality separately from pointer-ness, like Rust. You don't have to ask about unwrapping None in cases where None is not a member of &T.
  • deleted 4y ago
    [deleted]
  • solar-ice 4y ago
    In some languages, the concept of "may or may not be there" is cleanly separated from "exists in a separate block of memory". There's plenty of pointers which will never be nil - checking that they're nil at the entry and exit points of every function is line noise.
    • throwaway894345 4y ago
      I don't think the parent was advocating for checking for nil before/after each function, but rather noting that this option pattern doesn't buy you any more type safety in Go because there's nothing enforcing you to check appropriately any more than there is for a pointer. The salient rebuttal is that this pattern is a strong hint to check, whereas a pointer is ambiguous (it's unclear whether or not a bare pointer may ever be nil, but you would only use this pattern to be explicit that a check is needed). This pattern can also support value types so you don't have to worry as much about unnecessary allocations.
    • IshKebab 4y ago
      Is "some languages" just Rust?
      • solar-ice 4y ago
        Also Swift, and various functional languages too, although "exists in a separate block of memory" is basically the default there.
  • anoctopus 4y ago
    If something may or may not be there, that should be a clear part of its type that the compiler checks you handled (even if it's just you deciding to panic, because sometimes that's the right thing to do). Then there can be a type for just the thing, no hidden possibility of it not being there that you have to check for, and almost all code can be written for the type that is guaranteed to be there, with only a little code dealing with only possibly-there values.
• brushyamoeba 4y ago
  I'm surprised this doesn't discuss JSON marshalling
• eknkc 4y ago
  So, to avoid checking for null, you'll check for `NullString.Valid` now? The string pointer is a part of the language. You can pass it around, expose as part of a library etc. And it conveys the intent perfectly. I have no idea what is the issue here?
  • psanford 4y ago
    The nice thing about the Null* type is they can reduce the number of allocations done on the heap and thus also reduce total GC your program needs to do.
    • legorobot 4y ago
      I haven't thought about it that way! I've not hit the GC as a performance wall when it comes to accessing nullable DB values yet. Thanks for the insight.
      • throwaway894345 4y ago
        It's generally useful, not just in SQL. Any time you have a type that could be multiple things, you have to choose between a reference-based implementation (e.g., interfaces) or a tagged struct (a struct with a flag field that tells you what its runtime type). The tagged struct version is guaranteed not to allocate, while the interface version very likely will. Most of the time it will only matter if you're in a tight loop.
  • legorobot 4y ago
    I think the idea is `sql.NullString` can be used to have a SQL NULL still be an empty string in Go (just avoid checking the `Valid` field, or cast to string -- no check necessary). It seems like the intent of a string pointer vs. `sql.NullString` is their goal with this type anyways[1]? In practice I've used a string pointer when I need a nullable value, or enforce `NOT NULL` in the DB if we can't/don't want to handle null values. Use what works for you, and keep the DB and code's expectations in-sync. [1]: https://groups.google.com/g/golang-nuts/c/vOTFu2SMNeA/m/GB5v3JPSsicJ https://groups.google.com/g/golang-nuts/c/vOTFu2SMNeA/m/GB5v...
• Mawr 4y ago
  Great, except: 1. The type is merely a weak hint that you should check .Valid before you use the value, there's no enforcement: https://go.dev/play/p/nS8RxGujMBk https://go.dev/play/p/nS8RxGujMBk It's much better when the struct members are private and the only way to access the value is through a method that returns (value, bool): if value, ok := optional.Get(); ok { // value is valid } else { // value is invalid } This is a strong hint that you should check the bool before using the value. It's also a common go pattern - checking for existence of a key in a map is done this way. 2. We have generics now. Why use type-and-sql-specific wrappers when you could use a generic Option? Example implementation: https://gist.github.com/MawrBF2/0a60da26f66b82ee87b98b03336e1f84 https://gist.github.com/MawrBF2/0a60da26f66b82ee87b98b03336e....
  • psadauskas 4y ago
    Your comment is helpful, so this sarcasm isn't directed at you. However, this looks like an extremely cumbersome way to wish your language had the Result monad.
    • eyelidlessness 4y ago
      I upvoted both your comment and GP. Here, hopefully this helps your comment feel more productive: this is a Result type, albeit yes not a Monad. It’s (AFAIK) the idiomatic way the type is expressed in Go, so maybe cumbersome to implement but trivial to adopt. It’s also something I realized was lost in the JS progression towards Promise APIs and eventually async/await. Promises have ergonomic benefits over callback hell, but damn if the Node convention of cb: (error, result) => { /* … */ } wasn’t a Result type sitting there begging to be embraced. Again, not a Monad, but it’s a shame the good API design was thrown out with the inconvenient API design.
    • svnpenn 4y ago
      > extremely cumbersome I'd say the opposite. I'd say it beats the hell out of Rust syntax, as Rust would force you to have staircase code here. No thank you.
      • 4y ago
• AtNightWeCode 4y ago
  Maybe I passed out among the nonsense but most langs provide a dbnull constant to check against, no?
  • jrockway 4y ago
    You could do that in Go, but it's not what database/sql's API supports. You read columns out of each row with "row.Scan(&col1, &col2, ...)". The types of col1 and col2 are declared at compile time, and they don't have to be able to represent the concept of null. So there would be no way to store the state that represents that something was null. You could of course have an API that just returns a slice of "any", and conditionally check whether a value is of type "string" or "mylibrary.NullValue" after the fact. This isn't clearly better to me than the Scan API. You are going to have to eventually cast "any" to a real type in order to use it; with Scan the library does that for you. Your own types can implement sql.Scanner to control exactly how you want to handle something. (Indeed, your "Scan" method receives something with type "any", and you need to check what the type is and convert it to your internal representation.) Also wanted to throw this out here; you don't have to be satisfied with lossy versions of your database's built in types. Libraries like https://pkg.go.dev/github.com/jackc/pgtype@v1.11.0 https://pkg.go.dev/github.com/jackc/pgtype@v1.11.0 will give you a 1:1 mapping to Postgres's types. (I'm sure other database systems have similar libraries.)
• jackielii 4y ago
  I have an alternative solution to work with an existing struct that you can't change, e.g. protobuf generated code: https://jackieli.dev/posts/pointers-in-go-used-in-sql-scanner/#my-solution https://jackieli.dev/posts/pointers-in-go-used-in-sql-scanne... type nullString struct { s *string } func (ts nullString) Scan(value interface{}) error { if value == nil { *ts.s = "" // nil to empty return nil } switch t := value.(type) { case string: *ts.s = t default: return fmt.Errorf("expect string in sql scan, got: %T", value) } return nil } func (n *nullString) Value() (driver.Value, error) { if n.s == nil { return "", nil } return *n.s, nil } Then use it: var node struct { Name string } db.QueryRow("select name from node where id=?", id).Scan(nullString(&node.Name))
• davidkuennen 4y ago
  Using go and postgres for my App's backend. After using NULLs this way at first, I noticed it's generally much much easier for me to just avoid nullable SQL columns wherever possible (it was always possible so far). Most of the time there is a much easier was to say a value is empty. For strings '' for example. This seriously made everything so much easier. Not necessarily anything to do with go tho.
  • turboponyy 4y ago
    You should disallow nulls as much as possible, but "" is just as valid of a string value as "John Smith"; if the field is actually semantically nullable, reflect that in the type - don't use arbitrary "blank" values to denote nulls.
    • layer8 4y ago
      A string being empty oftentimes isn’t semantically different from the value being absent. I’d argue that, this being the case, the type should also be able to reflect whether the string can be empty or not (or all-whitespace or not, etc.).
      • turboponyy 4y ago
        I completely agree, but conventional database systems don't support dependent typing unfortunately. Given the constraints, the best you can do in this situation whilst remaining semantically consistent in the type signatures is to leave the field nullable and enforce value checks (e.g. can't be blank).
        • layer8 4y ago
          Database systems support constraint checks. Given that SQL isn’t statically typed, that’s almost the best you can do. The only lack is that you usually can’t bind constraints to a user-defined type name, so you have to repeat the constraint definition for each relevant column/table. But on the programming language side, dependent types (or something equivalent) would be appropriate, and I expect will become common practice at some point in the future.
• cratermoon 4y ago
  Yea I ran into this the first time I wrote much Go code to deal with SQL databases. After being initially annoyed I realized how much more correct it is for the SQL interface code to check .Valid and write proper, meaningful special case/null object handling for the domain types.
• hans_castorp 4y ago
  > by declaring our SQL columns as NOT NULL when possible. I hear this advice quite often, but that doesn't relief you of handling NULL values. E.g. a NOT NULL column can be NULL in a result of an outer join.