4 ms·

Author here, in case anyone wants to question, abuse, argue, etc.

bratao 8y ago
Thank you! I' m very excited to test in my projects ( I found a working Python extension here https://github.com/shenfe/python-hyperscan https://github.com/shenfe/python-hyperscan) Let me seize the opportunity. I have a problem where I need to match multiple person names (hundreds of thousands) in huge texts. Aho-Corasick works good for exact matches. Could HyperScan works for approximated matches?
- glangdale 8y ago
  We have both Levenstein and Hamming distance parameters. Do note that putting large distance numbers is not expected to perform well (i.e. matching "Smith" at Levenstein distance = 3 won't be a happy experience).
- taleinat 8y ago
  Have you tried some existing Python libraries which support fuzzy searching, such as regex and fuzzysearch?
  - bratao 8y ago
    Yes. Both grow the complexity linearly/exponentially based on the number of patterns to be searched.
    - taleinat 8y ago
      Ah. You should likely use a different algorithmic approach. If you add contact details to your user info I'll be happy to get in touch and help.
acuozzo 8y ago
I ported Hyperscan to ARM several months ago with help from the following project: https://github.com/nemequ/simde https://github.com/nemequ/simde Would you be interested in working together to get an ARM port that's a bit cleaner and author-approved?
- glangdale 8y ago
  I understand this possibility, but it's not the way I would do an ARM port in an ideal world. Being the original designer of Hyperscan isn't much help; I don't work at Intel any more and wouldn't have all that much clout in trying to get an ARM port into the codebase. I'm thinking about building a followup regex matcher - considerably smaller scale, and starting with a Hyperscan code base for the tools/infrastructure/boring bits - ("ure3"). This would definitely have ARM as a first-class architecture. Anyone 64-bit and little-endian can play, I would think (legacy stuff like 32-bit, etc can go somewhere else). I just with the ARM guys would get their shit together and support their own much ballyhooed SVE. The idea of announcing something in 2016 and not supporting it in a server chip that's due for mid-2020 is... novel.
- leon_young 8y ago
  Does the ARM version hyperscan work well? Can you describe the detailed steps? Thanks
femto 8y ago
The reference to "Australian-free" has piqued my curiosity. Do you care to elaborate? In my imagination you had an Australian collaborator that you fell out with and have been busy expunging all "Australian" code from the project!
- glangdale 8y ago
  I am Australian, as is the original technical team for Hyperscan. As of March 2018, no member of that team is still at Intel. Expunging Australian code from Hyperscan would be a non-trivial task.

Thank you! I' m very excited to test in my projects ( I found a working Python extension here https://github.com/shenfe/python-hyperscan https://github.com/shenfe/python-hyperscan) Let me seize the opportunity. I have a problem where I need to match multiple person names (hundreds of thousands) in huge texts. Aho-Corasick works good for exact matches. Could HyperScan works for approximated matches?

glangdale 8y ago
We have both Levenstein and Hamming distance parameters. Do note that putting large distance numbers is not expected to perform well (i.e. matching "Smith" at Levenstein distance = 3 won't be a happy experience).
taleinat 8y ago
Have you tried some existing Python libraries which support fuzzy searching, such as regex and fuzzysearch?
- bratao 8y ago
  Yes. Both grow the complexity linearly/exponentially based on the number of patterns to be searched.
  - taleinat 8y ago
    Ah. You should likely use a different algorithmic approach. If you add contact details to your user info I'll be happy to get in touch and help.

acuozzo 8y ago

I ported Hyperscan to ARM several months ago with help from the following project: https://github.com/nemequ/simde https://github.com/nemequ/simde Would you be interested in working together to get an ARM port that's a bit cleaner and author-approved?

glangdale 8y ago
I understand this possibility, but it's not the way I would do an ARM port in an ideal world. Being the original designer of Hyperscan isn't much help; I don't work at Intel any more and wouldn't have all that much clout in trying to get an ARM port into the codebase. I'm thinking about building a followup regex matcher - considerably smaller scale, and starting with a Hyperscan code base for the tools/infrastructure/boring bits - ("ure3"). This would definitely have ARM as a first-class architecture. Anyone 64-bit and little-endian can play, I would think (legacy stuff like 32-bit, etc can go somewhere else). I just with the ARM guys would get their shit together and support their own much ballyhooed SVE. The idea of announcing something in 2016 and not supporting it in a server chip that's due for mid-2020 is... novel.
leon_young 8y ago
Does the ARM version hyperscan work well? Can you describe the detailed steps? Thanks

femto 8y ago

The reference to "Australian-free" has piqued my curiosity. Do you care to elaborate? In my imagination you had an Australian collaborator that you fell out with and have been busy expunging all "Australian" code from the project!

glangdale 8y ago
I am Australian, as is the original technical team for Hyperscan. As of March 2018, no member of that team is still at Intel. Expunging Australian code from Hyperscan would be a non-trivial task.