5 ms·

> Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language

by zeveb 11y ago

> Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though. The Common Lisp standard includes pathname objects, with an implementation-defined (since different platforms have different conventions) mapping from strings to path objects. It doesn't specifically address the type-safety issue, though, since just about every standard function which accepts a pathname will also accept a namestring. There exist libraries such as CLSQL which enable representation of SQL queries as objects. Common Lisp: doing it right for so long that most people haven't even heard of it! Also: > When measuring a string to ensure that it fits in a 140-character tweet, you want to go by unicode code points. Really? Doesn't one actually want 140 graphemes? Regardless of what one wants, does Twitter enforce 140 code points, or 140 bytes? It seems to me that graphemes = characters, and that the fact that some characters are made up of multiple codepoints is as important as the fact that some code points are made up of multiple bytes.

zzalpha 11y ago
I wonder if part of the reason for this is that Common Lisp dates back to a time when filesystem path structures were a lot more variable. These days, you either have Unix-style or Windows-style paths and that's basically it But go back 30 years and there was quite a bit more variety that would make an abstraction layer all but a requirement if you wanted to define a portable language spec.
- rprospero 11y ago
  I remember reading a LISP book from the 80's that gently bragged about how all the legacy C code would have to be rewritten after upcoming operating systems moved beyond identifying files by a tree of folders, but that your LISP code would be future proof.
  - jdmichal 11y ago
    This strikes me as one of those instances were the cost of the abstraction exceeds the savings. There are some things which are simply core to your code. Attempting to abstract those things is going to create a lot of pain when, realistically, those things changing would require component rewrites anyway. If you're working with files, you've probably made many implicit assumptions about how the file system works and how it is represented. So you are either going to have a leaky abstraction that works, or a non-leaky abstraction that has those assumptions built in and will break when porting to a system without those assumptions anyway.
- gongador 11y ago
  Indeed. "The dominant file systems at the time the design was done were TOPS-10, TENEX, TOPS-20, VAX VMS, AT&T Unix, MIT Multics, MIT ITS, not to mention a bunch of mainframe OS's. Some were uppercase only, some mixed, some were case-sensitive but case-translating (like CL). Some had dirs as files, some not. Some had quote chars for funny file chars, some not. Some had wildcards, some didn't. Some had :up in relative pathnames, some didn't. Some had namable root dirs, some didn't. There were file systems with no directories, file systems with non-hierarchical directories, file systems with no file types, file systems with no versions, file systems with no devices, and so on. People often critique the design now because Unix and Mac and DOS are so comparatively similar (compared to some of those others)..." -- Kent Pitman in comp.lang.lisp, https://groups.google.com/forum/#!original/comp.lang.lisp/Pl0j7k9pNHA/tnnuPztZmHkJ https://groups.google.com/forum/#!original/comp.lang.lisp/Pl...
grey-area 11y ago
Isn't this quite common? Most languages allow you to represent those strings as different types and parse them/to from a plain string representation. From Ruby: String, Pathname, Url, ActiveRecord::Relation etc or from Go: strings, database/sql stmt + orm types built on top, net/url etc though file paths are just strings.
- MrBuddyCasino 11y ago
  It is, even Java does that. Where it gets interesting is advanced trickery, as performed by Haskell. You can have a type String that cares of escaping/unescaping stuff that comes from tainted sources in a webapp, eliminating a whole lot of security issues. Thats just one example I know, but I'm not a Haskell guy - there are surely more.
- mozumder 11y ago
  Yep. He basically just described "classes" in the object-oriented world.
  - tokenrove 11y ago
    I don't think it has much to do with classes. Python, for example, has no pathname/string distinction that I know of. Common Lisp is in the object-oriented world, and indeed some of the things mentioned are implemented as classes. I think the point was more about the standard library's attempt to do The Right Thing and how library implementors tend to follow that. (Also, for better or for worse, CL's pathnames contain information like host, device, and version, that aren't so relevant on the major OSes in use these days; but try dealing with VMS filenames with Ruby's Pathname...)
    - lispm 11y ago
      Device and even host are quite common in the Windows world: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... Volume = Device Or Microsoft's UNC path: https://en.wikipedia.org/wiki/Path_(computing)#Uniform_Naming_Convention https://en.wikipedia.org/wiki/Path_(computing)#Uniform_Namin... \\ComputerName\SharedFolder\Resource -> \\host\device\path
kazinator 11y ago
> implementation-defined (since different platforms have different conventions) mapping from strings to path objects. Unfortunately, this translates to a problem. We don't have that many path representations any more, but Common Lisp implementations for the same platform (or platform family, like POSIX) do the translation differently. If you want to work with CL pathnames, but expose paths as strings to the user, and use multiple CL implementations, you have to write your own path translation which behaves the same way. For instance, what is a/b/c/ ? Is that a pathname with directory a b c, and an empty name? Or directory a b and name c? Do you map suffixes to types? How about compounded suffixes like .tar.gz? Is that a "tar.gz" type or a "gz" type?
- Retra 11y ago
  Ideally, you'd provide interfaces for both interpretations.
  - kazinator 11y ago
    Two interpretations across N features: (expt 2 N) interface combinations. :)
    - Retra 11y ago
      That's really only a problem if you're building interfaces that can't be composed.
- wnoise 11y ago
  Are there any filesystems that support empty names? If not, clearly the second is correct.
zuck9 11y ago
Twitter has a limit of 140 characters to fit in a regular SMS (the rest 20 are used for the username). From https://en.wikipedia.org/wiki/Short_Message_Service https://en.wikipedia.org/wiki/Short_Message_Service > Short messages can be encoded using a variety of alphabets: the default GSM 7-bit alphabet, the 8-bit data alphabet, and the 16-bit UCS-2 alphabet. Depending on which alphabet the subscriber has configured in the handset, this leads to the maximum individual short message sizes of 160 7-bit characters, 140 8-bit characters, or 70 16-bit characters.
- richardwhiuk 11y ago
  That's not true. Firstly, a single tweet doesn't necessarily fit into a single SMS (as the Twitter limit is per character, whereas an SMS is limited per byte). Secondly, an SMS is limited to 140 bytes, which is 160 7 bit 'characters', or 140 8 bit 'characters' or 70 16 bit 'characters' (in UCS 2, normally).
  - mikeash 11y ago
    It was true. Twitter heavily emphasized SMS back when it was new. The 140-thingy limit hasn't been about SMS for a while now, and has just transitioned into a weird artificial limitation that's become part of what Twitter is. But that is how it started out.
  - deleted 11y ago
    [deleted]
- moron4hire 11y ago
  That might be the origin, but that is not the current implementation. For example, URLs only ever count as 23 letters, because they always get shortened using Twitter's own t.co shortener. BUT they still display in their original form, so unless they are de-referencing the shortened URL on the fly, they have to be storing what the user typed and a pre-processed form of the text are being stored. That's a lot more than 140 ASCII characters.
  - mikeash 11y ago
mikeash 11y ago
I'm actually not sure what Twitter is doing, now. Putting Emoji into the box decreases the remaining character count by one for each Emoji. But putting Zalgo into the box decreases the remaining character count by several for each grapheme cluster. My head hurts.
- TazeTSchnitzel 11y ago
  Twitter actually counts codepoints, but after normalisation, so it doesn't matter which of the two ways you represent é: https://dev.twitter.com/overview/api/counting-characters https://dev.twitter.com/overview/api/counting-characters
  - mikeash 11y ago
    Right, stupid me, emoji are still a single code point. Put a flag in the box, and the count decreases by two.
    - TazeTSchnitzel 11y ago
      Well, depends on the emoji. Some use combing characters, e.g. a red heart is the black heart character plus some invisible joiner IIRC, and some of the family emoji are composed of their individual components (mom/dad/girl/boy) with joiners.
      - mikeash 11y ago
        Yes, you're right. In a fit of silliness I was testing only with the ones that are a single code point, though.
TazeTSchnitzel 11y ago
> Regardless of what one wants, does Twitter enforce 140 code points, or 140 bytes? Number of code points after normalisation: https://dev.twitter.com/overview/api/counting-characters https://dev.twitter.com/overview/api/counting-characters

by zeveb 11y ago

zzalpha 11y ago

I wonder if part of the reason for this is that Common Lisp dates back to a time when filesystem path structures were a lot more variable. These days, you either have Unix-style or Windows-style paths and that's basically it But go back 30 years and there was quite a bit more variety that would make an abstraction layer all but a requirement if you wanted to define a portable language spec.

rprospero 11y ago
I remember reading a LISP book from the 80's that gently bragged about how all the legacy C code would have to be rewritten after upcoming operating systems moved beyond identifying files by a tree of folders, but that your LISP code would be future proof.
- jdmichal 11y ago
  This strikes me as one of those instances were the cost of the abstraction exceeds the savings. There are some things which are simply core to your code. Attempting to abstract those things is going to create a lot of pain when, realistically, those things changing would require component rewrites anyway. If you're working with files, you've probably made many implicit assumptions about how the file system works and how it is represented. So you are either going to have a leaky abstraction that works, or a non-leaky abstraction that has those assumptions built in and will break when porting to a system without those assumptions anyway.
gongador 11y ago
Indeed. "The dominant file systems at the time the design was done were TOPS-10, TENEX, TOPS-20, VAX VMS, AT&T Unix, MIT Multics, MIT ITS, not to mention a bunch of mainframe OS's. Some were uppercase only, some mixed, some were case-sensitive but case-translating (like CL). Some had dirs as files, some not. Some had quote chars for funny file chars, some not. Some had wildcards, some didn't. Some had :up in relative pathnames, some didn't. Some had namable root dirs, some didn't. There were file systems with no directories, file systems with non-hierarchical directories, file systems with no file types, file systems with no versions, file systems with no devices, and so on. People often critique the design now because Unix and Mac and DOS are so comparatively similar (compared to some of those others)..." -- Kent Pitman in comp.lang.lisp, https://groups.google.com/forum/#!original/comp.lang.lisp/Pl0j7k9pNHA/tnnuPztZmHkJ https://groups.google.com/forum/#!original/comp.lang.lisp/Pl...

grey-area 11y ago

Isn't this quite common? Most languages allow you to represent those strings as different types and parse them/to from a plain string representation. From Ruby: String, Pathname, Url, ActiveRecord::Relation etc or from Go: strings, database/sql stmt + orm types built on top, net/url etc though file paths are just strings.

MrBuddyCasino 11y ago
It is, even Java does that. Where it gets interesting is advanced trickery, as performed by Haskell. You can have a type String that cares of escaping/unescaping stuff that comes from tainted sources in a webapp, eliminating a whole lot of security issues. Thats just one example I know, but I'm not a Haskell guy - there are surely more.
mozumder 11y ago
Yep. He basically just described "classes" in the object-oriented world.
- tokenrove 11y ago
  I don't think it has much to do with classes. Python, for example, has no pathname/string distinction that I know of. Common Lisp is in the object-oriented world, and indeed some of the things mentioned are implemented as classes. I think the point was more about the standard library's attempt to do The Right Thing and how library implementors tend to follow that. (Also, for better or for worse, CL's pathnames contain information like host, device, and version, that aren't so relevant on the major OSes in use these days; but try dealing with VMS filenames with Ruby's Pathname...)
  - lispm 11y ago
    Device and even host are quite common in the Windows world: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... Volume = Device Or Microsoft's UNC path: https://en.wikipedia.org/wiki/Path_(computing)#Uniform_Naming_Convention https://en.wikipedia.org/wiki/Path_(computing)#Uniform_Namin... \\ComputerName\SharedFolder\Resource -> \\host\device\path

kazinator 11y ago

> implementation-defined (since different platforms have different conventions) mapping from strings to path objects. Unfortunately, this translates to a problem. We don't have that many path representations any more, but Common Lisp implementations for the same platform (or platform family, like POSIX) do the translation differently. If you want to work with CL pathnames, but expose paths as strings to the user, and use multiple CL implementations, you have to write your own path translation which behaves the same way. For instance, what is a/b/c/ ? Is that a pathname with directory a b c, and an empty name? Or directory a b and name c? Do you map suffixes to types? How about compounded suffixes like .tar.gz? Is that a "tar.gz" type or a "gz" type?

Retra 11y ago
Ideally, you'd provide interfaces for both interpretations.
- kazinator 11y ago
  Two interpretations across N features: (expt 2 N) interface combinations. :)
  - Retra 11y ago
    That's really only a problem if you're building interfaces that can't be composed.
wnoise 11y ago
Are there any filesystems that support empty names? If not, clearly the second is correct.

zuck9 11y ago

Twitter has a limit of 140 characters to fit in a regular SMS (the rest 20 are used for the username). From https://en.wikipedia.org/wiki/Short_Message_Service https://en.wikipedia.org/wiki/Short_Message_Service > Short messages can be encoded using a variety of alphabets: the default GSM 7-bit alphabet, the 8-bit data alphabet, and the 16-bit UCS-2 alphabet. Depending on which alphabet the subscriber has configured in the handset, this leads to the maximum individual short message sizes of 160 7-bit characters, 140 8-bit characters, or 70 16-bit characters.

richardwhiuk 11y ago
That's not true. Firstly, a single tweet doesn't necessarily fit into a single SMS (as the Twitter limit is per character, whereas an SMS is limited per byte). Secondly, an SMS is limited to 140 bytes, which is 160 7 bit 'characters', or 140 8 bit 'characters' or 70 16 bit 'characters' (in UCS 2, normally).
- mikeash 11y ago
  It was true. Twitter heavily emphasized SMS back when it was new. The 140-thingy limit hasn't been about SMS for a while now, and has just transitioned into a weird artificial limitation that's become part of what Twitter is. But that is how it started out.
- deleted 11y ago
  [deleted]
moron4hire 11y ago
That might be the origin, but that is not the current implementation. For example, URLs only ever count as 23 letters, because they always get shortened using Twitter's own t.co shortener. BUT they still display in their original form, so unless they are de-referencing the shortened URL on the fly, they have to be storing what the user typed and a pre-processed form of the text are being stored. That's a lot more than 140 ASCII characters.
- mikeash 11y ago

mikeash 11y ago

I'm actually not sure what Twitter is doing, now. Putting Emoji into the box decreases the remaining character count by one for each Emoji. But putting Zalgo into the box decreases the remaining character count by several for each grapheme cluster. My head hurts.

TazeTSchnitzel 11y ago
Twitter actually counts codepoints, but after normalisation, so it doesn't matter which of the two ways you represent é: https://dev.twitter.com/overview/api/counting-characters https://dev.twitter.com/overview/api/counting-characters
- mikeash 11y ago
  Right, stupid me, emoji are still a single code point. Put a flag in the box, and the count decreases by two.
  - TazeTSchnitzel 11y ago
    Well, depends on the emoji. Some use combing characters, e.g. a red heart is the black heart character plus some invisible joiner IIRC, and some of the family emoji are composed of their individual components (mom/dad/girl/boy) with joiners.
    - mikeash 11y ago
      Yes, you're right. In a fit of silliness I was testing only with the ones that are a single code point, though.

TazeTSchnitzel 11y ago

> Regardless of what one wants, does Twitter enforce 140 code points, or 140 bytes? Number of code points after normalisation: https://dev.twitter.com/overview/api/counting-characters https://dev.twitter.com/overview/api/counting-characters