4 ms·

Show HN: “Did You Mean?” for Python

66 points by jamesdutc 12y ago

jamesdutc 12y ago
This is probably not a good idea. (dutc = Don't Use This Code) It's horrible that it even works. It's surprising how well it works when it does work. It was a great opportunity to solidify some knowledge about low-level (implementation) details.
- maxerickson 12y ago
  Since you mention thinking there must be a better way in hook.c, you can override the traceback printer: https://docs.python.org/3.4/library/sys.html#sys.excepthook https://docs.python.org/3.4/library/sys.html#sys.excepthook I haven't thought through how to access the state needed to make a suggestion though.
  - jamesdutc 12y ago
    Yeah, that was one of the (less-bad) approaches I tried first: https://gist.github.com/dutc/3f2c79048d95287be138 https://gist.github.com/dutc/3f2c79048d95287be138 But you can see it's somewhat limited. And I was curious to actually hook an internal Python call! By the way, that hook.c comment was wondering whether there might be a nicer (more nearly portable?) way to set-up these assembly sections. Some of this stuff was hacked together, but I have a couple other games I want to play with this gimmick, and I want to make the hooking a bit better. https://github.com/dutc/libhook https://github.com/dutc/libhook
acron0 12y ago
It niggles me that the library is called "Did you mean?" but the error message is "Maybe you meant?". Why is the error message not "Did you mean X?"
cbp 12y ago
Did you mean to use a better language?
Chronic29 12y ago
This is a terrible idea.
w-m 12y ago
Didn't I see a question on how to get the object when an AttributeError is thrown on stackoverflow yesterday?
- Noelkd 12y ago
  You did[1] [1] http://stackoverflow.com/questions/26548094/how-to-get-hold-of-the-object-missing-an-attribute http://stackoverflow.com/questions/26548094/how-to-get-hold-...
jamesdutc 12y ago
Someone on GitHub asked "why shouldn't this be used": https://github.com/dutc/didyoumean/issues/1 https://github.com/dutc/didyoumean/issues/1 Here's the answer I gave: It's not a particularly useful feature in practice. Misspellings resulting in AttributeErrors are generally caught pretty quickly, and this addition to standard error reporting would mostly be useful in interactive settings where the mistake would be obvious. (Note that linting tools like PyFlakes already do a fairly good job of picking up on NameErrors, which result from using variables that don't exist.) I have two other approaches to supplement error reporting with spelling suggestions (https://gist.github.com/dutc/3f2c79048d95287be138 https://gist.github.com/dutc/3f2c79048d95287be138) that are a little less ‘janky.’ What makes this approach particularly offensive are: - I implement this by hooking into a C function in the Python interpreter itself, `PyObject_GetAttr`: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/didyoumean.c#L147 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - I find the function, unprotect its memory page, then clobber first few assembly instructions with a jump: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/hook.c#L29 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - I need to jump to an absolute address, since I don't want to (or can't?) calculate the relative addresses. I don't believe I can do this with a `push` and a `ret` or with a regular `call`, so I use a `jmp` instruction. The `jmp` instruction won't take an absolute address as an immediate value, so I have to use the `%rax` register: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/hook.c#L12 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - Since I'm using this register, I have to save & restore its value. In the hooking code, I save its value with a `push %rax`. In order to restore its value, I have to patch the assembly for the hook function to stick a `pop %rax` before any other instructions: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/hook.c#L50 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - In order to figure out the candidates for the spelling correction, I need to call `dir` on the object. But `dir` calls`PyObject_GetAttr` internally, and those calls can themselves trigger exceptions. In order to avoid this unbounded recursion, I have to implement a parallel code-path for `dir` by creating a `safe_PyObject_Dir`: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/safe_PyObject_Dir.c https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - In some builds of Python, the internal CFunction which provides the Python builtin function `getattr()`, `builtin_getattr`, is compiled without explicit calls to `PyObject_GetAttr`. In order to hook into these `getattr` calls, I need to patch the builtin module. But because someone could have already gotten a handle on `getattr`, I need to directly patch the function: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/didyoumean.c#L150 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... This approach is probably not portable, and it's definitely not a good idea. However, figuring out all these small problems was a great way to put some low-level knowledge to use!

66 points by jamesdutc 12y ago

This is probably not a good idea. (dutc = Don't Use This Code) It's horrible that it even works. It's surprising how well it works when it does work. It was a great opportunity to solidify some knowledge about low-level (implementation) details.

maxerickson 12y ago
Since you mention thinking there must be a better way in hook.c, you can override the traceback printer: https://docs.python.org/3.4/library/sys.html#sys.excepthook https://docs.python.org/3.4/library/sys.html#sys.excepthook I haven't thought through how to access the state needed to make a suggestion though.
- jamesdutc 12y ago
  Yeah, that was one of the (less-bad) approaches I tried first: https://gist.github.com/dutc/3f2c79048d95287be138 https://gist.github.com/dutc/3f2c79048d95287be138 But you can see it's somewhat limited. And I was curious to actually hook an internal Python call! By the way, that hook.c comment was wondering whether there might be a nicer (more nearly portable?) way to set-up these assembly sections. Some of this stuff was hacked together, but I have a couple other games I want to play with this gimmick, and I want to make the hooking a bit better. https://github.com/dutc/libhook https://github.com/dutc/libhook

acron0 12y ago

It niggles me that the library is called "Did you mean?" but the error message is "Maybe you meant?". Why is the error message not "Did you mean X?"

cbp 12y ago

Did you mean to use a better language?

Chronic29 12y ago

This is a terrible idea.

w-m 12y ago

Didn't I see a question on how to get the object when an AttributeError is thrown on stackoverflow yesterday?

Noelkd 12y ago
You did[1] [1] http://stackoverflow.com/questions/26548094/how-to-get-hold-of-the-object-missing-an-attribute http://stackoverflow.com/questions/26548094/how-to-get-hold-...

jamesdutc 12y ago

Someone on GitHub asked "why shouldn't this be used": https://github.com/dutc/didyoumean/issues/1 https://github.com/dutc/didyoumean/issues/1 Here's the answer I gave: It's not a particularly useful feature in practice. Misspellings resulting in AttributeErrors are generally caught pretty quickly, and this addition to standard error reporting would mostly be useful in interactive settings where the mistake would be obvious. (Note that linting tools like PyFlakes already do a fairly good job of picking up on NameErrors, which result from using variables that don't exist.) I have two other approaches to supplement error reporting with spelling suggestions (https://gist.github.com/dutc/3f2c79048d95287be138 https://gist.github.com/dutc/3f2c79048d95287be138) that are a little less ‘janky.’ What makes this approach particularly offensive are: - I implement this by hooking into a C function in the Python interpreter itself, `PyObject_GetAttr`: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/didyoumean.c#L147 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - I find the function, unprotect its memory page, then clobber first few assembly instructions with a jump: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/hook.c#L29 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - I need to jump to an absolute address, since I don't want to (or can't?) calculate the relative addresses. I don't believe I can do this with a `push` and a `ret` or with a regular `call`, so I use a `jmp` instruction. The `jmp` instruction won't take an absolute address as an immediate value, so I have to use the `%rax` register: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/hook.c#L12 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - Since I'm using this register, I have to save & restore its value. In the hooking code, I save its value with a `push %rax`. In order to restore its value, I have to patch the assembly for the hook function to stick a `pop %rax` before any other instructions: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/hook.c#L50 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - In order to figure out the candidates for the spelling correction, I need to call `dir` on the object. But `dir` calls`PyObject_GetAttr` internally, and those calls can themselves trigger exceptions. In order to avoid this unbounded recursion, I have to implement a parallel code-path for `dir` by creating a `safe_PyObject_Dir`: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/safe_PyObject_Dir.c https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... - In some builds of Python, the internal CFunction which provides the Python builtin function `getattr()`, `builtin_getattr`, is compiled without explicit calls to `PyObject_GetAttr`. In order to hook into these `getattr` calls, I need to patch the builtin module. But because someone could have already gotten a handle on `getattr`, I need to directly patch the function: https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155df6c6e3b59b914bacf20/src/didyoumean.c#L150 https://github.com/dutc/didyoumean/blob/2c9f01d03c1574f93155... This approach is probably not portable, and it's definitely not a good idea. However, figuring out all these small problems was a great way to put some low-level knowledge to use!