3 ms·

We did a great many of them over the decades; I find the mongoldb ones we encountered the most horrible (even compared to vague 80s + 90s plain text formats as

We did a great many of them over the decades; I find the mongoldb ones we encountered the most horrible (even compared to vague 80s + 90s plain text formats as they usually at least had some documentation as to what the developer was storing and how), but in general, we found the most reliable and, when you are used to it, the fastest (in the end) is creating types for everything and convertors from one to the other type. We did try other (more sloppy) ways before it and they always kind of result in bad quality. We have noticed that making (very exact) types for everything when doing complex conversions works well. Most notably, we try to avoid stringy types and add a lot of validations in the deserialisers. You can run conversions 10 years later on new batches and still get understandable errors.

cratermoon 5y ago
> try to avoid stringy types and add a lot of validations Right there you've stated a key practice that may, by itself, be 80% of the solution. Of course there's no "right" set of of types, but just putting the work into thinking about the types and the essential constraints will drive towards exposing and eliminating assumptions that will come back to bite later. Another huge chunk of problems can come from over-reliance on language primitive types. Is that numeric field really an integer, meaning that any signed value representable in 32, 64, or however many bits, including zero, is valid? Probably not. To give an example: A person's age is not an signed integer, it's a duration, we don't expect it to be negative. Maybe we should flag values above 90 or 100. If you're doing a one-off, that's the sort of thing you can code once and forget. If you dataset migration is your business, your Age type needs to be more robust, and perhaps there's even a need for the behavior of Age to vary with the domain.
justsomeuser 5y ago
Which language did you use to encode these types?
- tluyben2 5y ago
  We work in C# mainly; unfortunately; I have been in favour of F# for a long time, but it's just not popular enough so it would make our (already large, with decades of work) codebase less maintainable. However it does not really matter as long as you can make the types (classes in the case of C#) precise enough. You can indeed (see the other commenter) make (an) Age classe(s) which will give a nice error message if you do something illegal, for instance; int i = -2; Console.WriteLine((Age)i); Console.WriteLine((Age)200); Console.WriteLine((AdultAge<UnitedStates>)17); will throw three Exceptions. We define every single piece of data like that.

cratermoon 5y ago

> try to avoid stringy types and add a lot of validations Right there you've stated a key practice that may, by itself, be 80% of the solution. Of course there's no "right" set of of types, but just putting the work into thinking about the types and the essential constraints will drive towards exposing and eliminating assumptions that will come back to bite later. Another huge chunk of problems can come from over-reliance on language primitive types. Is that numeric field really an integer, meaning that any signed value representable in 32, 64, or however many bits, including zero, is valid? Probably not. To give an example: A person's age is not an signed integer, it's a duration, we don't expect it to be negative. Maybe we should flag values above 90 or 100. If you're doing a one-off, that's the sort of thing you can code once and forget. If you dataset migration is your business, your Age type needs to be more robust, and perhaps there's even a need for the behavior of Age to vary with the domain.

justsomeuser 5y ago

Which language did you use to encode these types?

tluyben2 5y ago
We work in C# mainly; unfortunately; I have been in favour of F# for a long time, but it's just not popular enough so it would make our (already large, with decades of work) codebase less maintainable. However it does not really matter as long as you can make the types (classes in the case of C#) precise enough. You can indeed (see the other commenter) make (an) Age classe(s) which will give a nice error message if you do something illegal, for instance; int i = -2; Console.WriteLine((Age)i); Console.WriteLine((Age)200); Console.WriteLine((AdultAge<UnitedStates>)17); will throw three Exceptions. We define every single piece of data like that.