3 ms·

I wrote the JavaScript regex engine in Hermes [1], and also regress in Rust. I want to explain what's sucky about this decision, but first some ES regexp backgr

I wrote the JavaScript regex engine in Hermes [1], and also regress in Rust. I want to explain what's sucky about this decision, but first some ES regexp background. The regexp grammar portion of the ES spec is different, in that it's not sufficient. You can implement JavaScript from the ES spec, but NOT regexp. The grammar doesn't really make sense [2]. The Chakra team found the same thing [3]. I have not confirmed this but I have been told the regexp portion is tightly controlled by Google, more than most parts of the spec. When v8's regexp implementation is found to deviate from the spec, the spec gets changed. This makes sense (why risk breaking websites), but it illustrates how little weight the regexp spec actually has. With Firefox using irregexp, it further cements this unfortunate reality. 1: https://hermesengine.dev https://hermesengine.dev 2: Example: the NonemptyClassRangesNoDash production may include a dash 3: https://docs.microsoft.com/en-us/archive/blogs/ie/chakra-interoperability-means-more-than-just-standards#whats-the-real-regular-expression-grammar https://docs.microsoft.com/en-us/archive/blogs/ie/chakra-int...

magicalist 6y ago
> You can implement JavaScript from the ES spec, but NOT regexp. The grammar doesn't really make sense [2]. The Chakra team found the same thing [3]. I haven't tried to implement the RegExp section, so I can't speak to that, but this is largely why we have Test262[1]. Also that Chakra blogpost is from 10 years ago, talking about ES5. Things have improved considerably since then, though of course we have to live with historical baggage. > With Firefox using irregexp, it further cements this unfortunate reality. Is this a specific unfortunate reality? I don't spot any RegExp entry in the last spec Incompatibilities section[2] that seems like a bad change or even having to do with regex grammar at all. [1] https://github.com/tc39/test262/tree/master/test/built-ins/RegExp https://github.com/tc39/test262/tree/master/test/built-ins/R... [2] https://www.ecma-international.org/ecma-262/#sec-additions-and-changes-that-introduce-incompatibilities-with-prior-editions https://www.ecma-international.org/ecma-262/#sec-additions-a...
- ridiculous_fish 6y ago
  Yes, thank goodness for Test262. One implements JS-compliant regexps by doing the natural thing, and then making Test262 pass. Some things have indeed improved. The 'u' flag introduced in ES6 provides a much cleaner grammar for example. However the process is still busted. Here is a concrete example. /{1}/ is permitted by the ES6 spec (including Annex B). But in practice, "web browsers" rejected it. Rather than fixing the browsers, the spec was changed in ES8 to reflect "web-reality". https://github.com/tc39/ecma262/pull/303 https://github.com/tc39/ecma262/pull/303 Browsers could have become ES6 compliant here, with zero compatibility risk. But when there is non-compliance, it is the spec that moves.
  - bawolff 6y ago
    That's how specs should work, when there is a feature nobody wants and nobody has implemented.
IainIreland 6y ago
> I have not confirmed this but I have been told the regexp portion is tightly controlled by Google, more than most parts of the spec. When v8's regexp implementation is found to deviate from the spec, the spec gets changed. This makes sense (why risk breaking websites), but it illustrates how little weight the regexp spec actually has. A few responses: 1. This was not at all my experience working with the V8 team. For example, when I noticed that Irregexp did not properly implement the Canonicalize algorithm for non-unicode case-insensitive comparisons [1], they were happy to accept my patches [2][3]. Nobody ever suggested changing the spec. 2. Nothing changes in the JS spec without consensus from all the major stakeholders. See, for example, the second sentence in the TC39 process document here [4]: "The committee operates by consensus and has discretion to alter the specification as it sees fit." Mozilla can block (and has blocked) proposals that it feels are bad for the web platform. Google has no special power to bend the committee to its will. 3. Another way of saying "when V8's regexp implementation is found to deviate from the spec, the spec gets changed" is "when the spec is found to deviate from the consensus of implementations, we correct the spec". The Chakra team's post notes: "In practice, all browsers accept regular expressions such as `/]/` and web developers write them." The purpose of the spec is to make the web work better by maximizing compatibility between implementations. In cases where all implementations agree, and real websites depend on that behaviour, the spec should change to match reality. Don't get me wrong: Chromium monoculture is a real problem. If we thought that Google was going to add a bunch of new regexp features without going through the standards process, or if we had plans to prototype regexp features that we feared Google wouldn't accept upstream, we might have made a different decision. But regexps are not a plausible battleground, and the compatibility benefits of sharing code outweigh the harms. PS: regress looks really sweet, and I am excited to see how it turns out. [1] https://tc39.es/ecma262/#sec-runtime-semantics-canonicalize-ch https://tc39.es/ecma262/#sec-runtime-semantics-canonicalize-... [2] https://github.com/v8/v8/commit/3fab9d05cf34a7f0bc0e9405729ab8b78c0671ac https://github.com/v8/v8/commit/3fab9d05cf34a7f0bc0e9405729a... [3] https://github.com/v8/v8/commit/b65fcfe92566415c8aae373d5fb5a9b16bed21ce https://github.com/v8/v8/commit/b65fcfe92566415c8aae373d5fb5... [4] https://tc39.es/process-document/ https://tc39.es/process-document/
- IainIreland 6y ago
  tl;dr: The spec exists for the web; the web does not exists for the spec.

magicalist 6y ago

> You can implement JavaScript from the ES spec, but NOT regexp. The grammar doesn't really make sense [2]. The Chakra team found the same thing [3]. I haven't tried to implement the RegExp section, so I can't speak to that, but this is largely why we have Test262[1]. Also that Chakra blogpost is from 10 years ago, talking about ES5. Things have improved considerably since then, though of course we have to live with historical baggage. > With Firefox using irregexp, it further cements this unfortunate reality. Is this a specific unfortunate reality? I don't spot any RegExp entry in the last spec Incompatibilities section[2] that seems like a bad change or even having to do with regex grammar at all. [1] https://github.com/tc39/test262/tree/master/test/built-ins/RegExp https://github.com/tc39/test262/tree/master/test/built-ins/R... [2] https://www.ecma-international.org/ecma-262/#sec-additions-and-changes-that-introduce-incompatibilities-with-prior-editions https://www.ecma-international.org/ecma-262/#sec-additions-a...

ridiculous_fish 6y ago
Yes, thank goodness for Test262. One implements JS-compliant regexps by doing the natural thing, and then making Test262 pass. Some things have indeed improved. The 'u' flag introduced in ES6 provides a much cleaner grammar for example. However the process is still busted. Here is a concrete example. /{1}/ is permitted by the ES6 spec (including Annex B). But in practice, "web browsers" rejected it. Rather than fixing the browsers, the spec was changed in ES8 to reflect "web-reality". https://github.com/tc39/ecma262/pull/303 https://github.com/tc39/ecma262/pull/303 Browsers could have become ES6 compliant here, with zero compatibility risk. But when there is non-compliance, it is the spec that moves.
- bawolff 6y ago
  That's how specs should work, when there is a feature nobody wants and nobody has implemented.

IainIreland 6y ago

> I have not confirmed this but I have been told the regexp portion is tightly controlled by Google, more than most parts of the spec. When v8's regexp implementation is found to deviate from the spec, the spec gets changed. This makes sense (why risk breaking websites), but it illustrates how little weight the regexp spec actually has. A few responses: 1. This was not at all my experience working with the V8 team. For example, when I noticed that Irregexp did not properly implement the Canonicalize algorithm for non-unicode case-insensitive comparisons [1], they were happy to accept my patches [2][3]. Nobody ever suggested changing the spec. 2. Nothing changes in the JS spec without consensus from all the major stakeholders. See, for example, the second sentence in the TC39 process document here [4]: "The committee operates by consensus and has discretion to alter the specification as it sees fit." Mozilla can block (and has blocked) proposals that it feels are bad for the web platform. Google has no special power to bend the committee to its will. 3. Another way of saying "when V8's regexp implementation is found to deviate from the spec, the spec gets changed" is "when the spec is found to deviate from the consensus of implementations, we correct the spec". The Chakra team's post notes: "In practice, all browsers accept regular expressions such as `/]/` and web developers write them." The purpose of the spec is to make the web work better by maximizing compatibility between implementations. In cases where all implementations agree, and real websites depend on that behaviour, the spec should change to match reality. Don't get me wrong: Chromium monoculture is a real problem. If we thought that Google was going to add a bunch of new regexp features without going through the standards process, or if we had plans to prototype regexp features that we feared Google wouldn't accept upstream, we might have made a different decision. But regexps are not a plausible battleground, and the compatibility benefits of sharing code outweigh the harms. PS: regress looks really sweet, and I am excited to see how it turns out. [1] https://tc39.es/ecma262/#sec-runtime-semantics-canonicalize-ch https://tc39.es/ecma262/#sec-runtime-semantics-canonicalize-... [2] https://github.com/v8/v8/commit/3fab9d05cf34a7f0bc0e9405729ab8b78c0671ac https://github.com/v8/v8/commit/3fab9d05cf34a7f0bc0e9405729a... [3] https://github.com/v8/v8/commit/b65fcfe92566415c8aae373d5fb5a9b16bed21ce https://github.com/v8/v8/commit/b65fcfe92566415c8aae373d5fb5... [4] https://tc39.es/process-document/ https://tc39.es/process-document/

IainIreland 6y ago
tl;dr: The spec exists for the web; the web does not exists for the spec.