5 ms·

I started reading this thinking it would talk about LC_ALL/locale related, but not too surprised to see another env variable throwing a wrench in the works.

nisa 7y ago
for context you refer to the fact that LC_ALL=C grep string hugefile is often magnitudes faster (also works for other shell utilities) - this is due to unicode handling that is either complex due to the nature of unicode or just performances badly on glibc - probably it's the complexity.
- jschwartzi 7y ago
  Seems analogous to xml parsing in Perl, where I realized a 10x speedup in parsing by switching from a general-purpose XML parser to a set of regular expressions.
  - sk5t 7y ago
    Using regex on serialized XML has the potential to go off the rails thanks to the wiiiide variety of ways to express character entities, import docs, use namespaces, escape or not, etc.; consider an event-y/SAX parser for performance if you might run afoul of such issues.
    - jschwartzi 7y ago
      Unless you know exactly the format of the data you're getting back. So in my use case it's not arbitrary XML but rather it's constrained to a particular message with specific fields that are named in a specific way. So it's really a text response that looks like XML rather than being some arbitrary XML message that differs after each response. We're literally getting the exact same XML message back, but with different text in each field. The framework we were using insisted on parsing each response separately even though the responses were basically the same, but with different values in the exact same XML tags.
      - sk5t 7y ago
        If you control precisely the inputs, then it could certainly be fine... until it's not because other thing changed. Please consider it a brittle hack.
        jschwartzi 7y ago
        Calling it a "brittle hack" implies that I don't have any control over the other endpoint. In reality my employer controls the endpoint I'm hitting. So it's not a brittle hack in our organization even though it would be a brittle hack if we were, say, hitting a Twitter API. In my opinion it's the same kind of optimization as massaging your data inputs to span cache lines in your MCU to improve performance. By requiring the data to be well-specified in its format you can ignore all of the clues that XML provides about the format of the data, because you can assume the format in advance. So all the stuff like schemas and validation become completely unnecessary because you already know what valid data looks like.
  - jasonjayr 7y ago
    https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/ https://stackoverflow.com/questions/1732348/regex-match-open... If you read nothing else, make sure you read the first answer there.
    - adrianN 7y ago
      If you want to extract just the values of a single tag or similarly trivial things or the XML is a very restricted set, regexes are just fine for parsing.
      - brokensegue 7y ago
        it's really not
        sk5t 7y ago
        Agreed. It's almost as odorous as xmldoc.toString.getBytes--which is definitely a thing some folks do--okay, welcome to encoding problems and busted documents.
        hopscotch 7y ago
        If you want to extract strings from XML that arrives in a well-known format that meets certain expectations that may not be rigorous, then regexes can be fine.
      - etaoins 7y ago
        You’re not parsing XML at that point. You’re parsing a text based data format that looks superficially like XML. This can work if you strictly control the encoder but it’s not generally interoperable.
        jschwartzi 7y ago
        Why does general interoperability matter in the context of two devices that you control both interoperating? I mean, if I had written the software on the other end I would've used a different interchange format altogether. But when you constrain the problem down from your software connecting to any and all systems on the internet to your software connecting to a specific system for the purpose of exchanging well-defined messages, you don't need a general-purpose parser with validation. And you end up spending a lot of time validating and analyzing the same message over and over again even though you can guarantee that it's the same message.
- imglorp 7y ago
  Yup, been bitten by this a number of times. You can't keep it set to C for daily use these days however. Modern things like Python have a bird.
  - minusf 7y ago
    in my experience `LC_CTYPE=en_US.UTF-8` is sufficient for python. not sure if that slows down grep or not.
bironran 7y ago
I used export LC_COLLATE=C export LANG=C to gain several orders of magnitude sorting through 6TB of data by 2 GUID-sized fields. Default collation is terribly slow compared to ascii only.

nisa 7y ago

for context you refer to the fact that LC_ALL=C grep string hugefile is often magnitudes faster (also works for other shell utilities) - this is due to unicode handling that is either complex due to the nature of unicode or just performances badly on glibc - probably it's the complexity.

jschwartzi 7y ago
Seems analogous to xml parsing in Perl, where I realized a 10x speedup in parsing by switching from a general-purpose XML parser to a set of regular expressions.
- sk5t 7y ago
  Using regex on serialized XML has the potential to go off the rails thanks to the wiiiide variety of ways to express character entities, import docs, use namespaces, escape or not, etc.; consider an event-y/SAX parser for performance if you might run afoul of such issues.
  - jschwartzi 7y ago
    Unless you know exactly the format of the data you're getting back. So in my use case it's not arbitrary XML but rather it's constrained to a particular message with specific fields that are named in a specific way. So it's really a text response that looks like XML rather than being some arbitrary XML message that differs after each response. We're literally getting the exact same XML message back, but with different text in each field. The framework we were using insisted on parsing each response separately even though the responses were basically the same, but with different values in the exact same XML tags.
    - sk5t 7y ago
      If you control precisely the inputs, then it could certainly be fine... until it's not because other thing changed. Please consider it a brittle hack.
      - jschwartzi 7y ago
        Calling it a "brittle hack" implies that I don't have any control over the other endpoint. In reality my employer controls the endpoint I'm hitting. So it's not a brittle hack in our organization even though it would be a brittle hack if we were, say, hitting a Twitter API. In my opinion it's the same kind of optimization as massaging your data inputs to span cache lines in your MCU to improve performance. By requiring the data to be well-specified in its format you can ignore all of the clues that XML provides about the format of the data, because you can assume the format in advance. So all the stuff like schemas and validation become completely unnecessary because you already know what valid data looks like.
- jasonjayr 7y ago
  https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/ https://stackoverflow.com/questions/1732348/regex-match-open... If you read nothing else, make sure you read the first answer there.
  - adrianN 7y ago
    If you want to extract just the values of a single tag or similarly trivial things or the XML is a very restricted set, regexes are just fine for parsing.
    - brokensegue 7y ago
      it's really not
      - sk5t 7y ago
        Agreed. It's almost as odorous as xmldoc.toString.getBytes--which is definitely a thing some folks do--okay, welcome to encoding problems and busted documents.
      - hopscotch 7y ago
        If you want to extract strings from XML that arrives in a well-known format that meets certain expectations that may not be rigorous, then regexes can be fine.
    - etaoins 7y ago
      You’re not parsing XML at that point. You’re parsing a text based data format that looks superficially like XML. This can work if you strictly control the encoder but it’s not generally interoperable.
      - jschwartzi 7y ago
        Why does general interoperability matter in the context of two devices that you control both interoperating? I mean, if I had written the software on the other end I would've used a different interchange format altogether. But when you constrain the problem down from your software connecting to any and all systems on the internet to your software connecting to a specific system for the purpose of exchanging well-defined messages, you don't need a general-purpose parser with validation. And you end up spending a lot of time validating and analyzing the same message over and over again even though you can guarantee that it's the same message.
imglorp 7y ago
Yup, been bitten by this a number of times. You can't keep it set to C for daily use these days however. Modern things like Python have a bird.
- minusf 7y ago
  in my experience `LC_CTYPE=en_US.UTF-8` is sufficient for python. not sure if that slows down grep or not.

bironran 7y ago

I used export LC_COLLATE=C export LANG=C to gain several orders of magnitude sorting through 6TB of data by 2 GUID-sized fields. Default collation is terribly slow compared to ascii only.