4 ms·
I like seeing more things come out helping people with data analysis in various languages, but this put me off: > isWhitespace x = elem x " \t\r\n" This is th
by IanCal 10y ago
I like seeing more things come out helping people with data analysis in various languages, but this put me off:
> isWhitespace x = elem x " \t\r\n"
This is the kind of thing that makes me concerned about using this resource for real-world data. In real-world data you're going to get all kinds of crazy things coming in, and if you're assuming nobody will ever have something like a zero width non-breaking space, or a form feed, you're going to have a problem.
It's the kind of thing I see with people starting out when dealing with data, similarly the punctuation detection here: https://github.com/BinRoot/Haskell-Data-Analysis-Cookbook/blob/master/Ch02/Code02_punctuation/Main.hs https://github.com/BinRoot/Haskell-Data-Analysis-Cookbook/bl...
If you rely on these things, you will have problems. Text is hard and weird and terribly more complicated than people usually expect.
Does haskell have good libraries for dealing with the more awkward parts? Can I easily remove all characters marked as whitespace in unicode, for example? Detecting and managing mangled encodings?
- ocharles 10y agoFor more realistic stuff, I generally use https://hackage.haskell.org/package/text-icu https://hackage.haskell.org/package/text-icu. For example, in an old project we had a type called `Texty`, which was roughly non-empty normalized text: mkTexty :: Text -> Maybe Texty mkTexty = \a -> let a' = T.dropAround Char.isSpace (ICU.normalize ICU.NFC a) in if T.null a' then Nothing else Just (Texty a') {-# INLINE mkTexty #-}
- IanCal 10y agoGreat, thanks for the link.
- unhammer 10y agoYeah, that doesn't give high hopes for this book teaching best practices :/ If you don't want to depend on text-icu, there is support for more Unicode space characters in https://hackage.haskell.org/package/base-4.8.2.0/docs/Data-Char.html#v:isSpace https://hackage.haskell.org/package/base-4.8.2.0/docs/Data-C... import Data.Char (isAlpha, isLower, isSpace) import qualified Data.Text as T giving you e.g. textIsAllSpace t = T.all isSpace t
- gh02t 10y agoImportant note for non-Haskellers: the library OP is talking about is part of the standard base libraries usually distributed with GHC. `text-icu`, while excellent, is not.
- zallarak 10y agoThat line of code was demonstrative. The actual book uses 'Data.Char.IsSpace' which properly handled it. "Returns True for any Unicode space character, and the control characters \t, \n, \r, \f, \v." Before you chastise them for not handling something, verify it. I'm not affiliated with the book but you probably deterred people from buying it. https://github.com/BinRoot/Haskell-Data-Analysis-Cookbook/blob/master/Ch02/Code01_whitespace/Main.hs https://github.com/BinRoot/Haskell-Data-Analysis-Cookbook/bl...
- kornish 10y agoI agree that line of code was a turn-off because of its oversimplification. > That line of code was demonstrative. Why demonstrate with something completely unrepresentative of the actual contents, then?
- IanCal 10y agoWell for one the line shouldn't really be one of the first bits of advertising for the book if the author also knows it's wrong. The second example is taken from the GitHub repo for the book though, and is exactly the same type of error. > you probably deterred people from buying it. Quite possibly, but I think with good reason. I don't know what's in the book but I'm concerned it won't contain things like a discussion of what whitespace is and is not, how to decide what you should do for your data and when isSpace might not do what you really need. I can't review it properly but at least one bit of code in the repo looks dangerous and one bit on the website looks dangerous.