3 ms·

This release finally brings a way to raise an error when an unknown key is encountered in a JSON object: https://go-review.googlesource.com/c/go/+/74830 https:/

• jitl 9y ago
  This is the improvement that I’m most excited about for this release. I waded through some code recently that implemented its own validation in a hacks way by text-digging the re-serialization of structs... We were unable to put default (zero) values in our config files...
  • majewsky 9y ago
    > We were unable to put default (zero) values in our config files... Huh? I don't know the details of what you're (de)serializing, but the standard strategy for default values that are not zero values is to have unpack into a pointer. For example, if you have a message like { "foo": 23, "bar": 42 } and the "bar" field has a default value of 5, you unpack like: var msg struct { Foo int Bar *int } err := json.Unmarshal(data, &msg) //handle error if msg.Bar == nil { defaultValue := 5 msg.Bar = &defaultValue } If you cannot change the type of the message to have a pointer field, you can implement json.Unmarshaler for the receiving type such that UnmarshalJSON deserializes into a temporary type, then copies the values into the recipient: type Message struct { Foo int Bar int } func (m *Message) UnmarshalJSON(in []byte) error { var msg struct { Foo int Bar *int } err := json.Unmarshal(data, &msg) if err != nil { return err } m.Foo = msg.Foo m.Bar = 5 if msg.Bar != nil { m.Bar = *msg.Bar } return nil } Now you can deserialize into Message (or into a struct containing a Message or *Message or []Message or whatever) and it will fill in the correct default value.
  • jerf 9y ago
    One thing you may want to consider in the future, or, at least, whoever wrote that code, is that it's not that difficult to fork the standard library and make it do something different in your program. I forked the encoding/xml library once so I could add tag stack depth limits and size limits for tags and text nodes, so I could use it to write an XML validator (in which you must be careful that the user can not simply blow out your RAM by sending you too much text), and I forked encoding/json so that you could declare a map[string]interface{} field on an struct you are decoding into that will "catch" any unmatched keys in the JSON object and treat it the same as if you passed the map[string]interface{} in as the thing to decode into, instead of discarding them. (Encoding works too.) The first was on the order of two days and the second was on the order of four hours (untested in production, but the unit tests are fairly easy and comprehensive), though in both of those particular cases if you've not fiddled with the "reflect" package enough to understand it, you'll also need to pay that price. Other packages may require other knowledge, of course. But the time required is such that it may be less than the time spent on dirty, half-working hacks, which tends to grow from the "just five minutes" of the initial estimate to "five days and we're still paying for the hack five years later...".
  • shabbyrobe 9y ago
    You can have whatever default value you like by initialising the struct before you pass it to json.Unmarshal. If the key isn't present, it won't be set: type Test struct { Foo string Bar string } func NewTest() *Test { return &Test{Foo: "a"} } func main() { t := NewTest() if err := json.Unmarshal([]byte(`{"Bar": "b"}`), t); err != nil { log.Fatal(err) } fmt.Printf("%#v\n", t) } Output: &main.Test{Foo:"a", Bar:"b"}
• bpicolo 9y ago
  I wouldn’t recommend that against external APIs in general, though there are use cases. Adding keys is usually the best way to make backwards compatible changes in an API, but you’d make those changes backwards incompatible in your own infra.
  • shabbyrobe 9y ago
    That's a good point. It's more useful for config files than APIs I reckon. That's why I've been hanging for it.
  • lobster_johnson 9y ago
    My experience is that strictness is preferable here; silently accepting bad keys in a JSON object will lead the caller to think it's working, when it's not. If you then break the API's backwards compatibility by, say, removing a field, then your API will visibly break, which is correct, something you'd otherwise not detect. The solution to maintaining backwards compatibility is a combination of explicit versioning, erroring like this, good documentation, and trying to support old features for as long as possible. Definitely never liked Go's silent ignoring of JSON keys. The workarounds (e.g. overriding the unmarshal code to apply a check) have always been very invasive. Edit: Maybe you meant in the client, since you wrote "against". If so, I would still argue that strictness is good, as you do want your client to break (or at least warn) if something changes.
    • bpicolo 9y ago
      > If you then break the API's backwards compatibility by, say, removing a field, then your API will visibly break That's a different issue. Fields going missing is definitely something I want to cause big problems. I do mean consumers (e.g. unmarshalling JSON responses from APIs), as the typical mechanism to support backwards compatibility is by adding keys. Otherwise you can't upgrade ever without entirely new API versions, and you force lockstep for deployments. Needing new data is a common thing - maybe you have user endpoint that's not returning <some_flag> that you added that you now want in some new service. Bumping APIs for every added field would get unwieldy fast and you'll have services scattered among many API 'versions'. It's just like in protobuf. In protobuf (or most other service-to-service serialization libs), adding a new field is not backwards incompatible.