4 ms·

(author) I appreciate Haskell's power in this domain, and apparently sum types are life-changing (they get mentioned in so many language conversations). That

by tmcw 6y ago

(author) I appreciate Haskell's power in this domain, and apparently sum types are life-changing (they get mentioned in so many language conversations). That said, I'm mostly concerned with types like columns, rows, tables, colors, urls, documents, or images. These are implementable in lots of languages, but many languages just have a lot of ad-hoc implementations of them, of the same concept but with slightly different tilts on it. I'm saying that it would be interesting if a language focused on that part of the ecosystem, so that things could connect together. Haskell might have a very nice type system, but it does not seem like it has many universally-accepted standard types for everything to use. For example, as soon as it gets up to the complexity of a "text" type, there are lots of slightly-different flavors of text.

sillysaurusx 6y ago
Hey, you're the author! I just wanted to thank you for writing this. I saw it yesterday, actually, and it set me off on an adventure: you mentioned seaborn, so I went and learned that (it's beautiful), which mentioned xarray (http://xarray.pydata.org/en/stable/ http://xarray.pydata.org/en/stable/) which is the generalized solution to the problem you mention! I'm a bit skeptical, but, xarray's demos are very impressive. They do all kinds of geospatial modeling on ocean salinity, ozone, etc, and the data model seems general enough to create beautiful plots automatically (for various reasons, but mostly because of what you were saying in your post: simplicity enforced by constraints). I'd love your thoughts on xarray, since it was such a serendipitous discovery in relation to your post. (If you're not impressed with xarray, then that's probably a useful signal that perhaps it's a step in the wrong direction on the ladder of complexity.) But either way, thanks for reminding me that "the solution is elegance, not a battalion of special cases."
- legobmw99 6y ago
  (Not the OP) I’ve been using xarray for the better part of a year now on some serious scientific projects. It certainly has some pain points and places to grow (it’s still a 0.x release after all), but when it works it’s truly excellent to use. So many things “just work”
  - sillysaurusx 6y ago
    Thanks for the review! I'll give it a serious try then. Does your code happen to be available anywhere? I'd love to learn by example from someone who's used it for so long on real work. (Feel free to DM me on twitter: https://twitter.com/theshawwn https://twitter.com/theshawwn if you'd like to share it privately.) Have a great night!
    - legobmw99 6y ago
      Sadly it’s internal. I’ve found the xarray docs to be fairly comprehensive, and GitHub issues/stackoverflow fill the gaps.
jerf 6y ago
"of the same concept but with slightly different tilts on it" The answer is, they aren't the same concept. In English, we may look at multiple implementations and say "hey, those are all an 'image'", but in reality, each of these is a distinct concept: A representation of the image meant for manipulation by pixel, such as for a 2d image editor. A representation of the internal structure of the image format, such as giving access to the PNG or JPG frames. The raw bytes of the specific serialization, such as for shipping around a network. Representations of the image in a display context, such as the display driver might use. Representation of the image from the point of view of metadata, such as a file explorer may need. Representation of an image from the point of view of editing, where it may be disassembled into frames or layers for later reconstruction after applying edits, such as Photoshop may use. You can't create an "image" class/package/library that will fit all those needs, to say nothing of the ways those needs may interact with other things (i.e. I may need metadata for that file explorer that includes non-images as well, even there are image-specific elements to it like a thumbnail). Moreover, the lack of connectivity between this sort of object isn't even my top 25 programming problems... I'm almost always several layers composed on top of these sorts of things anyhow with my own local data concerns. Even if I've got an "image" I've also got a ton of other things with it that nothing else can be expected to understand (like which user this is a avatar for or where the image came from or any number of other things). It's sometimes annoying to deal with too many color types or something, but that's a local concern for an ecosystem, not a universal problem a general-purpose language should be spending valuable design budget trying to dictate. (DSLs can, if appropriate, precisely because they are domain-specific languages.)
lmm 6y ago
I think this is the same kind of misguided as "every app has its own ad-hoc implementation of login, it's the same concept with just a slightly different tilt on it, so the language should offer a login function". The essence of programming is properly representing similarities between similar things but also properly representing the sometimes quite subtle differences. So most mature languages, rather than offering us a bunch of premade functions, try to offer us some low-level building blocks and make it easy to compose them together to suit our purposes (and I think this is why giant class libraries have gone out of fashion - even though classes are sort of extensible, customising someone else's class turns out to be a lot harder than using someone else's function). And I think that's the part that's really missing in data-land (and I'd say even Haskell is barely any way along this road) - not a bunch of predefined datatypes but really easy ways of composing basic datatypes and building them up into more complex datatypes that meet our needs. Having lots of similar but subtly distinct datatypes isn't actually a problem if your language is good at dealing with it, just as having lots of similar but subtly distinct functions isn't actually a problem.
gampleman 6y ago
I'd just like to point out some examples of things that are implemented incompatibly commonly, not because they are different: Colors: (r{0.0-255.0}, g{0.0-255.0}, b{0.0-255.0}, a{0.0-255.0}) vs { red = r{0.0-1.0}, green = g{0.0-1.0}, blue={0.0-1.0}, opacity = o{0.0-1.0} } That is tuple vs record, different scaling factor, opacity vs transparency, but these representations are equivalent. Or geopoint: (lat, lng) vs (lng, lat) I could go on...
- jacobolus 6y ago
  https://macwright.com/lonlat/ https://macwright.com/lonlat/
  - gampleman 6y ago
    If transfer size isn't an (important) consideration, then the one true answer is: just use a record, which doesn't enforce an ordering and is explicit about which coordinate is which.
- Quekid5 6y ago
  The color ones are not (unambiguously) equivalent, you're ignoring bit depth and assuming 8-bit color values in one of them.
  - gampleman 6y ago
    I am not, both are floats. They are represented only in that range, because that's customary in web dev circles.
    - Quekid5 6y ago
      Floats? What does R=255.0, etc. mean when the other example has R=1.0? Precision has nothing to do with it, the scale isn't even defined... which was my point.
mannykannot 6y ago
I think you have a point, but the situation you describe is largely because things are in flux - that is certainly the case for web pages. When consensus is reached, things settle down - so most new languages give you associative arrays out of the box. On the other hand, there was a time when exceptions seemed to be the consensus on runtime error handling, but there has been some rethinking on that... Then there is APL, which seems to exemplify what you are asking for. It is very well-regarded by those who get it, but it seems to have been just too difficult (or alien) for most of the engineers whose purpose it addresses. Haskell may not be the best example to make a case from, as it is still, to a large extent, an experimental language. Spreadsheets are also something of an outlier, as they are a system of data manipulation already developed before it was practical to computerize them.

by tmcw 6y ago

sillysaurusx 6y ago

Hey, you're the author! I just wanted to thank you for writing this. I saw it yesterday, actually, and it set me off on an adventure: you mentioned seaborn, so I went and learned that (it's beautiful), which mentioned xarray (http://xarray.pydata.org/en/stable/ http://xarray.pydata.org/en/stable/) which is the generalized solution to the problem you mention! I'm a bit skeptical, but, xarray's demos are very impressive. They do all kinds of geospatial modeling on ocean salinity, ozone, etc, and the data model seems general enough to create beautiful plots automatically (for various reasons, but mostly because of what you were saying in your post: simplicity enforced by constraints). I'd love your thoughts on xarray, since it was such a serendipitous discovery in relation to your post. (If you're not impressed with xarray, then that's probably a useful signal that perhaps it's a step in the wrong direction on the ladder of complexity.) But either way, thanks for reminding me that "the solution is elegance, not a battalion of special cases."

legobmw99 6y ago
(Not the OP) I’ve been using xarray for the better part of a year now on some serious scientific projects. It certainly has some pain points and places to grow (it’s still a 0.x release after all), but when it works it’s truly excellent to use. So many things “just work”
- sillysaurusx 6y ago
  Thanks for the review! I'll give it a serious try then. Does your code happen to be available anywhere? I'd love to learn by example from someone who's used it for so long on real work. (Feel free to DM me on twitter: https://twitter.com/theshawwn https://twitter.com/theshawwn if you'd like to share it privately.) Have a great night!
  - legobmw99 6y ago
    Sadly it’s internal. I’ve found the xarray docs to be fairly comprehensive, and GitHub issues/stackoverflow fill the gaps.

jerf 6y ago

"of the same concept but with slightly different tilts on it" The answer is, they aren't the same concept. In English, we may look at multiple implementations and say "hey, those are all an 'image'", but in reality, each of these is a distinct concept: A representation of the image meant for manipulation by pixel, such as for a 2d image editor. A representation of the internal structure of the image format, such as giving access to the PNG or JPG frames. The raw bytes of the specific serialization, such as for shipping around a network. Representations of the image in a display context, such as the display driver might use. Representation of the image from the point of view of metadata, such as a file explorer may need. Representation of an image from the point of view of editing, where it may be disassembled into frames or layers for later reconstruction after applying edits, such as Photoshop may use. You can't create an "image" class/package/library that will fit all those needs, to say nothing of the ways those needs may interact with other things (i.e. I may need metadata for that file explorer that includes non-images as well, even there are image-specific elements to it like a thumbnail). Moreover, the lack of connectivity between this sort of object isn't even my top 25 programming problems... I'm almost always several layers composed on top of these sorts of things anyhow with my own local data concerns. Even if I've got an "image" I've also got a ton of other things with it that nothing else can be expected to understand (like which user this is a avatar for or where the image came from or any number of other things). It's sometimes annoying to deal with too many color types or something, but that's a local concern for an ecosystem, not a universal problem a general-purpose language should be spending valuable design budget trying to dictate. (DSLs can, if appropriate, precisely because they are domain-specific languages.)

lmm 6y ago

I think this is the same kind of misguided as "every app has its own ad-hoc implementation of login, it's the same concept with just a slightly different tilt on it, so the language should offer a login function". The essence of programming is properly representing similarities between similar things but also properly representing the sometimes quite subtle differences. So most mature languages, rather than offering us a bunch of premade functions, try to offer us some low-level building blocks and make it easy to compose them together to suit our purposes (and I think this is why giant class libraries have gone out of fashion - even though classes are sort of extensible, customising someone else's class turns out to be a lot harder than using someone else's function). And I think that's the part that's really missing in data-land (and I'd say even Haskell is barely any way along this road) - not a bunch of predefined datatypes but really easy ways of composing basic datatypes and building them up into more complex datatypes that meet our needs. Having lots of similar but subtly distinct datatypes isn't actually a problem if your language is good at dealing with it, just as having lots of similar but subtly distinct functions isn't actually a problem.

gampleman 6y ago

I'd just like to point out some examples of things that are implemented incompatibly commonly, not because they are different: Colors: (r{0.0-255.0}, g{0.0-255.0}, b{0.0-255.0}, a{0.0-255.0}) vs { red = r{0.0-1.0}, green = g{0.0-1.0}, blue={0.0-1.0}, opacity = o{0.0-1.0} } That is tuple vs record, different scaling factor, opacity vs transparency, but these representations are equivalent. Or geopoint: (lat, lng) vs (lng, lat) I could go on...

jacobolus 6y ago
https://macwright.com/lonlat/ https://macwright.com/lonlat/
- gampleman 6y ago
  If transfer size isn't an (important) consideration, then the one true answer is: just use a record, which doesn't enforce an ordering and is explicit about which coordinate is which.
Quekid5 6y ago
The color ones are not (unambiguously) equivalent, you're ignoring bit depth and assuming 8-bit color values in one of them.
- gampleman 6y ago
  I am not, both are floats. They are represented only in that range, because that's customary in web dev circles.
  - Quekid5 6y ago
    Floats? What does R=255.0, etc. mean when the other example has R=1.0? Precision has nothing to do with it, the scale isn't even defined... which was my point.

mannykannot 6y ago

I think you have a point, but the situation you describe is largely because things are in flux - that is certainly the case for web pages. When consensus is reached, things settle down - so most new languages give you associative arrays out of the box. On the other hand, there was a time when exceptions seemed to be the consensus on runtime error handling, but there has been some rethinking on that... Then there is APL, which seems to exemplify what you are asking for. It is very well-regarded by those who get it, but it seems to have been just too difficult (or alien) for most of the engineers whose purpose it addresses. Haskell may not be the best example to make a case from, as it is still, to a large extent, an experimental language. Spreadsheets are also something of an outlier, as they are a system of data manipulation already developed before it was practical to computerize them.