3 ms·

> Are regular expressions hard to read? Sometimes they can, especially if you haven't gone out of your way to take deep dives. Are they not perfect for a lot of

> Are regular expressions hard to read? Sometimes they can, especially if you haven't gone out of your way to take deep dives. Are they not perfect for a lot of parsing tasks? Sure, there are definitely things people shouldn't use regexs for that they do, like validating emails. At their core though, regular expressions are one of the most power dense tools we have in pretty much every language. The insane stuff you can get done (sometimes in a way that would be frowned upon) in so little time if just incredible. The real crowning use case for regular expressions (at least in terms of parsing-like tasks) is when you've got formats with varied delimiters. So one format I was parsing this weekend was basically header:field1,field2,field3"data"hash (with a fixed number of fields). Or another format I work with is suite~split/test1,test2@opt1:opt2^hw1^hw2#flags1#flags2 (most of the elements of which are optional). Regular expressions shine here; using primitives like split don't quite cut it in these formats. And I think this also is a major cause of why regular expressions tend to become unreadable quickly. If you look at parsing via regular expressions as a whole, there's basically three things that are being jammed into one expression: what are the delimiters between fields, what is valid in each field, and which fields are optional. These are basically three separate concerns, but most regex APIs are absolutely terrible at letting you separate these concerns into separate steps (all you can do is provide the string that combines all of them).

tialaramex 3y ago
> header:field1,field2,field3"data"hash (with a fixed number of fields). In a language where regex is just right there, like Perl, I agree that the natural way to parse this is always a regex, although on the other hand in Perl the natural way to do almost everything is a regex so maybe Perl was a bad example. In a language with split_once (and a proper string reference type) I actually rather like split_once here, splitting away header, and then field1, and then field2, and then field3, and then data, leaving just hash. I guess this gets to your concern, by writing it with split_once we're clear about a lot of your answers, each delimiter is specified with what it's splitting, none of the fields are optional (unless we write that) and (if we write code to check) what is valid in each field.
- jcranmer 3y ago
  Yeah, split_once is pretty handy, although chaining together can get a little verbose. It would be nice to write this: let (y,m,d,h,m,s) = break_str!(s, year-month-day hour:minute:second)?; instead of this: let (y, split) = s.split_once('-')?; let (m, split) = split.split_once('-')?; let (d, split) = split.split_once(' ')?; let (h, split) = split.split_once(':')?; let (m, s) = split.split_once(':')?;
  - burntsushi 3y ago
    I'll use this as an opportunity to plug the regex crate's new 'extract' API. :-) use regex::Regex; fn main() { println!("{:?}", extract("1973-01-05 09:30:00")); } fn extract(haystack: &str) -> Option<(&str, &str, &str, &str, &str, &str)> { let re = Regex::new( r"([0-9]{4})-([0-9]{2})-([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2})", ).unwrap(); let (_, [y, m, d, h, min, s]) = re.captures(haystack)?.extract(); Some((y, m, d, h, min, s)) } Output: Some(("1973", "01", "05", "09", "30", "00")) That gets you pretty close to what you want here. (The regex matches more than what is a valid date/time of course.)
    - Izkata 3y ago
      Python (even in 2): >>> import re >>> pattern = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2})") >>> results = pattern.match('1973-01-05 09:30:00') >>> results.groups() ('1973', '01', '05', '09', '30', '00') >>> (y, m, d, h, min, s) = results.groups() (`results` will be None if the regex didn't match) Regexes are one of those things where, once you understand it (and capture groups in particular) and it's available in the language you're working in, string-splitting usually doesn't feel right anymore.
      - burntsushi 3y ago
        I believe all of the people in this thread understand regexes extremely well. :-) There is a lot of reasonable room to disagree about when and where regexes should be used.
kbenson 3y ago
The thing that I think sometimes gets lost when people are talking about the strengths and weaknesses of regular expression use, is that it can matter quite a bit what the language in question is like. If you're using a language that compiles down to a low level or is expected to have a fairly close match of expressions to specific code run (C, C++, Java, etc) then the choice is between using a regular expression library which often does an good enough match of what you want to accomplish in a performance manner that rolling your own optimized parser is hard to justify. When using an interpreted language like Perl or Python, it's a slightly different story, since often there's no way to get as good of performance out of a custom parser written in that language as you can get from the optimized regular expression library included (if you use it competently). The overhead in the interpreted language can make some tasks impossible to do as quickly as teh regular expression engine can. In a way, a regular expression library is to parsing what Numpy is to computation. I remember a task I had to parse a bunch of very simplistic XML in Perl a decade ago, where it consisted of the equivalent of a bunch of records of dictionaries/hashes (solr), and needed them as Perl hashes to work on them. I surveyed every XML parsing module available to me in Perl, and couldn't quite get the performance I needed, given the file was a few GB and it needed to be done as quick as possible multiple times a day. I implemented a simple outer loop to grab a working chunk of the file, a regex to split that chunk into individual records text chunks, and another regex to return each key/value as an alternating list I could assign directly to a hash. It was 5-6x faster than the faster XML streaming solution I found that I could use as a Perl module (which were all just wrapping X libs). Could I have coded a custom solution in C that parsed this specialized format faster? Undoubtedly, but I'd still be marshaling data through the C/Perl interface, so would have taken a hit to get the data easily accessible to the rest of the codebase, which is less of an issue for regular expressions in Perl since they're a native capability of the language and deeply integrated. Using regular expressions as a parsing toolkit yielded very good results at a fraction of the time and effort, and honestly in my opinion that's what regular expressions are all about.

tialaramex 3y ago

> header:field1,field2,field3"data"hash (with a fixed number of fields). In a language where regex is just right there, like Perl, I agree that the natural way to parse this is always a regex, although on the other hand in Perl the natural way to do almost everything is a regex so maybe Perl was a bad example. In a language with split_once (and a proper string reference type) I actually rather like split_once here, splitting away header, and then field1, and then field2, and then field3, and then data, leaving just hash. I guess this gets to your concern, by writing it with split_once we're clear about a lot of your answers, each delimiter is specified with what it's splitting, none of the fields are optional (unless we write that) and (if we write code to check) what is valid in each field.

jcranmer 3y ago
Yeah, split_once is pretty handy, although chaining together can get a little verbose. It would be nice to write this: let (y,m,d,h,m,s) = break_str!(s, year-month-day hour:minute:second)?; instead of this: let (y, split) = s.split_once('-')?; let (m, split) = split.split_once('-')?; let (d, split) = split.split_once(' ')?; let (h, split) = split.split_once(':')?; let (m, s) = split.split_once(':')?;
- burntsushi 3y ago
  I'll use this as an opportunity to plug the regex crate's new 'extract' API. :-) use regex::Regex; fn main() { println!("{:?}", extract("1973-01-05 09:30:00")); } fn extract(haystack: &str) -> Option<(&str, &str, &str, &str, &str, &str)> { let re = Regex::new( r"([0-9]{4})-([0-9]{2})-([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2})", ).unwrap(); let (_, [y, m, d, h, min, s]) = re.captures(haystack)?.extract(); Some((y, m, d, h, min, s)) } Output: Some(("1973", "01", "05", "09", "30", "00")) That gets you pretty close to what you want here. (The regex matches more than what is a valid date/time of course.)
  - Izkata 3y ago
    Python (even in 2): >>> import re >>> pattern = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2})") >>> results = pattern.match('1973-01-05 09:30:00') >>> results.groups() ('1973', '01', '05', '09', '30', '00') >>> (y, m, d, h, min, s) = results.groups() (`results` will be None if the regex didn't match) Regexes are one of those things where, once you understand it (and capture groups in particular) and it's available in the language you're working in, string-splitting usually doesn't feel right anymore.
    - burntsushi 3y ago
      I believe all of the people in this thread understand regexes extremely well. :-) There is a lot of reasonable room to disagree about when and where regexes should be used.

kbenson 3y ago

The thing that I think sometimes gets lost when people are talking about the strengths and weaknesses of regular expression use, is that it can matter quite a bit what the language in question is like. If you're using a language that compiles down to a low level or is expected to have a fairly close match of expressions to specific code run (C, C++, Java, etc) then the choice is between using a regular expression library which often does an good enough match of what you want to accomplish in a performance manner that rolling your own optimized parser is hard to justify. When using an interpreted language like Perl or Python, it's a slightly different story, since often there's no way to get as good of performance out of a custom parser written in that language as you can get from the optimized regular expression library included (if you use it competently). The overhead in the interpreted language can make some tasks impossible to do as quickly as teh regular expression engine can. In a way, a regular expression library is to parsing what Numpy is to computation. I remember a task I had to parse a bunch of very simplistic XML in Perl a decade ago, where it consisted of the equivalent of a bunch of records of dictionaries/hashes (solr), and needed them as Perl hashes to work on them. I surveyed every XML parsing module available to me in Perl, and couldn't quite get the performance I needed, given the file was a few GB and it needed to be done as quick as possible multiple times a day. I implemented a simple outer loop to grab a working chunk of the file, a regex to split that chunk into individual records text chunks, and another regex to return each key/value as an alternating list I could assign directly to a hash. It was 5-6x faster than the faster XML streaming solution I found that I could use as a Perl module (which were all just wrapping X libs). Could I have coded a custom solution in C that parsed this specialized format faster? Undoubtedly, but I'd still be marshaling data through the C/Perl interface, so would have taken a hit to get the data easily accessible to the rest of the codebase, which is less of an issue for regular expressions in Perl since they're a native capability of the language and deeply integrated. Using regular expressions as a parsing toolkit yielded very good results at a fraction of the time and effort, and honestly in my opinion that's what regular expressions are all about.