4 ms·

Tangent, but another interesting problem with date parsing besides lack of standardization is that it's ridiculously context-sensitive. For example, if I sched

Tangent, but another interesting problem with date parsing besides lack of standardization is that it's ridiculously context-sensitive. For example, if I schedule an event for Oct 1-Jan 1 right now, the most likely (least surprising) parse would be Oct 1, 2021 through January 1, 2022. Which is both surprising because parsing Oct 1 depends on the current date, and because the parsing of Jan 1 depends on the parse of Oct 1, and because both of these depend on the fact that you're scheduling (in all likelihood) a future event, rather than describing a past event. So whatever date parser you use (no matter how good it is at handling different notations) will return garbage if it doesn't take all these into account.

komali2 6y ago
Heh, interesting, I actually assumed it would be oct 2020 to Jan 2021, and that you were describing an event already in progress.
- mehrdadn 6y ago
  Sure :-) even then, you have even more fun... imagine if the next item was Sep 1-Dec 1 and the list was known to be sorted! The parser would need to be stateful to be able to disambiguate!! Even worse (well, maybe similar, but more surprising): imagine if on Feb 28, you parse "Feb 29-Mar 1"... both of those dates could land in an entirely different year than "Feb 28-Mar 1" would, depending on whether the current year is a leap year... I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that. And all of these problems are before we even consider time zones, leap seconds, daylight savings, syntactic ambiguities, etc... not just how they affect individual dates, but also ordered dates (/intervals) like above...
  - enriquepablo 6y ago
    I would say that you are mixing up "parsing" and "calendaring (or something of the sort)". As far as I understand parsing is syntactic analysis, i.e. going from a linear structure to a more complex structure (usually a tree); it should not add to the tree anything that was not in the linear structure. It shouldn't consider a semantic context (such as the current date) to produce an ast.
    - mehrdadn 6y ago
      By parsing I mean it in the usual sense for a date... strptime, Date.Parse, etc... i.e. turning a string to a date (or multiple dates). You can call it something else if you'd like.
  - monsieurbanana 6y ago
    > I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that. You can google "nlp date extraction" maybe? I've used libraries in the past that do this. Note: they're far from being perfect. I ended up not using any, as each had their weird corner cases. Here's an old one: http://natty.joestelmach.com/try.jsp# http://natty.joestelmach.com/try.jsp# I tried with: "first week of december to end of january" And it gave me: Tue Dec 01 15:31:30 UTC 2020 Fri Jan 31 15:31:30 UTC 2020 Edit: this one seems more polished: http://nlp.stanford.edu:8080/sutime/process http://nlp.stanford.edu:8080/sutime/process
    - mehrdadn 6y ago
      Cool, thanks for the tip!
jandrese 6y ago
Parsing anything that was originally developed by humans writing it on paper (or clay tablet, or whatever) is a nightmare. Natural means chaotic. If your CSV file contains any field entered by humans AWK isn't going to be powerful enough to parse it at scale. Someone somewhere is going to have the name 'Mbat"a, Sho,dlo' in some bizarre ass romanization (and this assume you're not accepting Unicode, which is a whole other can of worms that AWK is not prepared to deal with) that breaks your parser.
- mulmen 6y ago
  I'm saving that as a test case name but making some small adjustments. 'Mbaät\"a, Sho,dló' "a" followed by "ä" because some suggest encoding umlauts as double characters. When you decode that does it go first or second? Answer: use Unicode. Throw an escape character in there, before the quote character just to make it interesting. This is a good time for everyone to review: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

komali2 6y ago

Heh, interesting, I actually assumed it would be oct 2020 to Jan 2021, and that you were describing an event already in progress.

mehrdadn 6y ago
Sure :-) even then, you have even more fun... imagine if the next item was Sep 1-Dec 1 and the list was known to be sorted! The parser would need to be stateful to be able to disambiguate!! Even worse (well, maybe similar, but more surprising): imagine if on Feb 28, you parse "Feb 29-Mar 1"... both of those dates could land in an entirely different year than "Feb 28-Mar 1" would, depending on whether the current year is a leap year... I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that. And all of these problems are before we even consider time zones, leap seconds, daylight savings, syntactic ambiguities, etc... not just how they affect individual dates, but also ordered dates (/intervals) like above...
- enriquepablo 6y ago
  I would say that you are mixing up "parsing" and "calendaring (or something of the sort)". As far as I understand parsing is syntactic analysis, i.e. going from a linear structure to a more complex structure (usually a tree); it should not add to the tree anything that was not in the linear structure. It shouldn't consider a semantic context (such as the current date) to produce an ast.
  - mehrdadn 6y ago
    By parsing I mean it in the usual sense for a date... strptime, Date.Parse, etc... i.e. turning a string to a date (or multiple dates). You can call it something else if you'd like.
- monsieurbanana 6y ago
  > I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that. You can google "nlp date extraction" maybe? I've used libraries in the past that do this. Note: they're far from being perfect. I ended up not using any, as each had their weird corner cases. Here's an old one: http://natty.joestelmach.com/try.jsp# http://natty.joestelmach.com/try.jsp# I tried with: "first week of december to end of january" And it gave me: Tue Dec 01 15:31:30 UTC 2020 Fri Jan 31 15:31:30 UTC 2020 Edit: this one seems more polished: http://nlp.stanford.edu:8080/sutime/process http://nlp.stanford.edu:8080/sutime/process
  - mehrdadn 6y ago
    Cool, thanks for the tip!

jandrese 6y ago

Parsing anything that was originally developed by humans writing it on paper (or clay tablet, or whatever) is a nightmare. Natural means chaotic. If your CSV file contains any field entered by humans AWK isn't going to be powerful enough to parse it at scale. Someone somewhere is going to have the name 'Mbat"a, Sho,dlo' in some bizarre ass romanization (and this assume you're not accepting Unicode, which is a whole other can of worms that AWK is not prepared to deal with) that breaks your parser.

mulmen 6y ago
I'm saving that as a test case name but making some small adjustments. 'Mbaät\"a, Sho,dló' "a" followed by "ä" because some suggest encoding umlauts as double characters. When you decode that does it go first or second? Answer: use Unicode. Throw an escape character in there, before the quote character just to make it interesting. This is a good time for everyone to review: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...