6 ms·

Here is a list of basic errors in Python. Only builtin ones, I'm not talking about ones from other libs. And I'm talking of error categories, as we all know one

Here is a list of basic errors in Python. Only builtin ones, I'm not talking about ones from other libs. And I'm talking of error categories, as we all know one exception can be used in a lot of different contexts: BaseException +-- SystemExit +-- KeyboardInterrupt +-- GeneratorExit +-- Exception +-- StopIteration +-- StopAsyncIteration +-- ArithmeticError | +-- FloatingPointError | +-- OverflowError | +-- ZeroDivisionError +-- AssertionError +-- AttributeError +-- BufferError +-- EOFError +-- ImportError +-- LookupError | +-- IndexError | +-- KeyError +-- MemoryError +-- NameError | +-- UnboundLocalError +-- OSError | +-- BlockingIOError | +-- ChildProcessError | +-- ConnectionError | | +-- BrokenPipeError | | +-- ConnectionAbortedError | | +-- ConnectionRefusedError | | +-- ConnectionResetError | +-- FileExistsError | +-- FileNotFoundError | +-- InterruptedError | +-- IsADirectoryError | +-- NotADirectoryError | +-- PermissionError | +-- ProcessLookupError | +-- TimeoutError +-- ReferenceError +-- RuntimeError | +-- NotImplementedError | +-- RecursionError +-- SyntaxError | +-- IndentationError | +-- TabError +-- SystemError +-- TypeError +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError 52 exceptions. Now, on this list, how many can be linked to language typing ? +-- TypeError +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError +-- AttributeError And ALL those errors can also be triggered with something a type system would no catch : runtime class generation, unexpexted user input, corrupted database, wrong data header, etc) So basically, relying only on typing will cover about 10% of the error types, most of which are caught with linters such as flake8 and a code intelligence tools such as Jedi. You are making a very weak case.

naasking 10y ago
> And ALL those errors can also be triggered with something a type system would no catch : runtime class generation, unexpexted user input, corrupted database, wrong data header, etc) Runtime class generation and unexpected user input can actually be handled in types. Any language with Type:Type, generative modules, first-class modules, and other variations all handle various forms of runtime type generation. Something like lightweight static capabilities [1] can ensure runtime input conforms to expectations encoded in types. Furthermore, you are way underselling types with that restricted list. NameError, SyntaxError, ImportError wouldn't occur in any typed languages at runtime. LookupError, AttributeError, AssertionError, ArithmeticError wouldn't occur in some of them. Finally, your breakdown doesn't cover how frequent any of these errors occur. For example, even if typing were to solve only TypeError, if TypeError consists of 90% of runtime errors, that would be a huge win. [1] http://okmij.org/ftp/papers/lightweight-static-capabilities.pdf http://okmij.org/ftp/papers/lightweight-static-capabilities....
- sametmax 10y ago
  > Runtime class generation and unexpected user input can actually be handled in types. Any language with Type:Type, generative modules, first-class modules, and other variations all handle various forms of runtime type generation. Something like lightweight static capabilities [1] can ensure runtime input conforms to expectations encoded in types. How would you if you can't deternmine type in advance, check for the type it will be ? I don't understand. > Furthermore, you are way underselling types with that restricted list. NameError, SyntaxError, ImportError wouldn't occur in any typed languages at runtime. Absolutly not. Those are not linked to typing in any way, and any decent Python editor catch them. > LookupError, AttributeError, AssertionError, ArithmeticError wouldn't occur in some of them. AssertionError ? Serioulsy ? Do you even know what it does in Python ? And LookupError, unless you got constant sized containers, which has nothing to do with types, you can't check that. ArithmeticError ? Come on ! Are you numbers constants ? You need to check the inputs so that they belong to the domain of your problem, at that's it. Nothing to do with types. > Finally, your breakdown doesn't cover how frequent any of these errors occur. For example, even if typing were to solve only TypeError, if TypeError consists of 90% of runtime errors, that would be a huge win. Yes but it's not. My last week have 90% keyerror, and empty values. The input is usually the source of errors. Types are useful, and they come at a cost. Whether your want to use them or not is a technical choice to make. But selling types the way it's been done on this thread is dishonest. Or you are all here working with algo problems and very little user input. Which are the easy program to code. The hard part would be the algo, not the code. If you are dealing with a user app, a web site, a video game, typing will not save you from the 90% of the bugs, and you DO need unit tests.
  - naasking 10y ago
    > How would you if you can't deternmine type in advance, check for the type it will be ? I don't understand. Compilers do this all the time. Just consider all those programs that dynamically generate code based on objects that they haven't seen in advance, like object-relational mappers. Those can all be statically typed and ensure that they generate correct code [1]. > Absolutly not. Those are not linked to typing in any way, and any decent Python editor catch them. It's all part of compilation, and furthermore, we can type check code generation so that names errors don't even occur in runtime generated code. Sorry, but all of these errors are related to type checking. > AssertionError ? Serioulsy ? Do you even know what it does in Python ? http://research.microsoft.com/apps/pubs/default.aspx?id=138696 http://research.microsoft.com/apps/pubs/default.aspx?id=1386... Just google "static contract checking" to find plenty of work, including compilers already available for Haskell. Heck, code contracts have been widely deployed on .NET for years now. > And LookupError, unless you got constant sized containers, which has nothing to do with types, you can't check that. Please read the lightweight static capabilities paper I already provided. It demonstrates using the Haskell and OCaml type systems to statically assure that all array bounds are in range, even for dynamically sized structures. So yes, you can check that, which doesn't even go into dependent typing where checking such dynamic properties is the whole point. > ArithmeticError ? Come on ! Are you numbers constants ? You need to check the inputs so that they belong to the domain of your problem, at that's it. Nothing to do with types. You don't seem to realize that types are simply logical propositions about a program. They can be ANY proposition about a program, including that indexing is within bounds, that a concurrent program has no race conditions or deadlocks [2], that programs terminate, that HTML forms correctly serialize/deserialize values from/to structures [1], and more. Like most dynamic typing enthusiasts, you don't have a proper appreciation for the true scope of static typing. You are correct that typing has its costs that aren't always warranted, but you are incorrect about where you draw that line because I don't think you fully understand how powerful static typing truly is. [1] http://www.impredicative.com/ur/ http://www.impredicative.com/ur/ [2] http://www.kb.ecei.tohoku.ac.jp/~koba/typical/ http://www.kb.ecei.tohoku.ac.jp/~koba/typical/
vog 10y ago
I'd also exclude: +-- StopIteration +-- StopAsyncIterat which are used for mere communication rather than describing real error cases. Also, you should exclude: +-- SystemExit +-- KeyboardInterrupt +-- GeneratorExit +-- ImportError +-- MemoryError +-- SyntaxError | +-- IndentationError | +-- TabError because unit tests won't help you to catch these, either (because they are either thrown directly on program start, or depend of specific system circumstances rather than bugs in your code). Finally, I would argue that most instances of this exception are caught by a type system: | +-- KeyError Why? Because most of the time the key is not user input, but a hard coded string, and accessing non-existing properties of an object/struct is caught by most type systems on compile time.
- sametmax 10y ago
  > I'd also exclude: > > +-- StopIteration > +-- StopAsyncIterat > > which are used for mere communication rather than describing real error cases. Absolutly not. If you implement very low level logic, you can use those. I have to catch them several times in my life. > Also, you should exclude: > > +-- SystemExit > +-- KeyboardInterrupt > +-- GeneratorExit > Certainly not. You should DEFINITLY unit test for those. I do it all the time: def test_ready_keyboard_interrupt(aioloop, app): beacon = Mock() real_stop = app._finish def beacon_stop(): beacon() real_stop() app._finish = beacon_stop @app.on('running') def stahp(): beacon() raise KeyboardInterrupt() @app.on('stop') def stop(): beacon() app.ready() assert beacon.call_count == 3 You must check if you app behavre well when killed. > +-- ImportError That a type system can't check for, but a good linter can. I never have import errors because I got proper tooling. > +-- MemoryError That you should test for of course. Last week I worked on a split/merge algo on big files. I had to test that. > +-- SyntaxError > > | +-- IndentationError > | +-- TabError Well, doh :) Type system or unit tests won't catch that. Tooling, tooling, tooling. > Finally, I would argue that most instances of this exception are caught by a type system: > > | +-- KeyError No, no no ! KeyError should be super mega tested: - keys can be any mutable in Python, not string; - keys are very often generated on the fly, not constants; - dict are mutable, you can add or remove keys.
discreteevent 10y ago
"relying only on typing will cover about 10% of the error types". It's not about the percentage of error types. Its about the percentage of errors. A very high percentage of unit tests in a dynamic language are written to catch type errors. That work is a waste of time when a type system could check it for you. The reason for this is that type errors are involved in all code. A FileExistsError only applies if I am working with files. (Plus with types you get the added benefit of inline documentation and better tooling.)
- gaius 10y ago
  It's also about the compiler guaranteeing to enumerate every code path in and out of every function. A human writing tests is limited by what they think will happen, but the compiler knows.
- sametmax 10y ago
  > A very high percentage of unit tests in a dynamic language are written to catch type errors. Absolutly not. If you do it you are doing it wrong. Check this out: https://github.com/Tygs/tygs/blob/master/tests/test_components.py https://github.com/Tygs/tygs/blob/master/tests/test_componen... The type checks are "isinstance". It's 10% of the checks, and most of the time I don't even have those, my colleague insisted. The rest are the most important tests: they test behavior, not types. Do not assert a dynamic language means testing types, it's just not true. Bad testers test types.
  - the_af 10y ago
    "isInstance" doesn't cover the full range of type errors. It simply covers whether something is, say, an Int or a String. Types can encode much more. They can encode behavior.
- zephyrfalcon 10y ago
  > A very high percentage of unit tests in a dynamic language are written to catch type errors. I'm not sure where you got that information. You don't actually write tests to catch types. You write tests to check behavior. While it is possible that actual type errors come out of the woodwork while doing that, that is not the reason why the test was written in the first place; and after fixing such a type error, the test itself is still valid (again because it is making sure that the code exhibits the desired behavior). I don't think I have EVER written a test just to check for types (in Python). Neither do I pepper my code with 'assert isinstance(x, some_class)' because it just goes against the grain of a dynamically typed language.
imtringued 10y ago
How do unit tests protect you from a corrupted database? You are making a very weak case.
- sametmax 10y ago
  The ones you write: this month I'm working on medical data, we got tests where we scramble the database to test the reaction of our software of course. Error reporting, cleaning, logging. You mean you don't do that ?
  - deleted 10y ago
    [deleted]
deadfoxygrandpa 10y ago
Most of these would be caught at compile time in a language like Elm or (to a lesser extent) Haskell. A common pattern in some of these languages with actually strong type systems is to encode these failure possibility points in the type of the associated functions. Also, in Elm as an example, there is no runtime class generation, the type of user input that is possible is checked at compile time, and you're forced to explicitly handle many of these potential failures. If you want to crash the program if you get unexpected user input, you literally need to type "Debug.crash" and then you pretty much deserve what you get. If you're actually interested in maintainable code you should concentrate on reducing the state space of your program wherever possible, then using exhaustive (if possible) or property testing to fill in most of the rest of the gaps. Unit tests can help too, but I really think of them as less useful than all of the above.
- sametmax 10y ago
  You can not catch a keyerror at run time. You can not check for user input with a type system. Those have nothing to do with types. Input and output are out of your control, they need check sanitizing, and dynamic language are actually great at that.
  - pka 10y ago
    You are wrong. You can very much catch "key errors" at compile time.
Artemis2 10y ago
The bugs are way more subtle than that. You're lucky if your bad programming is caught somewhere at runtime. In most cases, Python and other weakly-typed languages will do everything they can to fit the data you're providing with what the code is expecting. This means converting strings to numbers and the other way around, plus a lot of other oddities you would not always expect (see PHP's intval for a crash course in bad language design). All that results in bugs that are far from obvious, and that don't always raise neat exceptions. EDIT: I stand corrected, I don't know enough about Python in particular. The argument is still valid for other languages with weak types.
- sametmax 10y ago
  You clearly haven't done much Python. It doesn't do implicit convertion. Byte are separated from strings and you need to decode explicitly. You can't add '1' and 1. Python is dynamically type. It is NOT weakly typed.
- to3m 10y ago
  This is actually something Python doesn't do... generally, if you have a type error in Python, you get an exception at runtime. I certainly don't disagree with what I guess will be the overall direction of your argument.
- zephyrfalcon 10y ago
  Python is not "weakly typed", nor does it ever automatically convert strings to numbers or the other way around.
  - brianwawok 10y ago
    Heck with Python3 it won't even convert bytes to String.
- a3n 10y ago
  > In most cases, Python and other weakly-typed languages will do everything they can to fit the data you're providing with what the code is expecting. This means converting strings to numbers ... Python is strongly typed, as well as dynamically typed. In [11]: 1 + '1' --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-11-861a99da769e> in <module>() ----> 1 1 + '1' TypeError: unsupported operand type(s) for +: 'int' and 'str'
gaius 10y ago
You absolutely can handle runtime errors via the type system just using a Maybe.

naasking 10y ago

> And ALL those errors can also be triggered with something a type system would no catch : runtime class generation, unexpexted user input, corrupted database, wrong data header, etc) Runtime class generation and unexpected user input can actually be handled in types. Any language with Type:Type, generative modules, first-class modules, and other variations all handle various forms of runtime type generation. Something like lightweight static capabilities [1] can ensure runtime input conforms to expectations encoded in types. Furthermore, you are way underselling types with that restricted list. NameError, SyntaxError, ImportError wouldn't occur in any typed languages at runtime. LookupError, AttributeError, AssertionError, ArithmeticError wouldn't occur in some of them. Finally, your breakdown doesn't cover how frequent any of these errors occur. For example, even if typing were to solve only TypeError, if TypeError consists of 90% of runtime errors, that would be a huge win. [1] http://okmij.org/ftp/papers/lightweight-static-capabilities.pdf http://okmij.org/ftp/papers/lightweight-static-capabilities....

sametmax 10y ago
> Runtime class generation and unexpected user input can actually be handled in types. Any language with Type:Type, generative modules, first-class modules, and other variations all handle various forms of runtime type generation. Something like lightweight static capabilities [1] can ensure runtime input conforms to expectations encoded in types. How would you if you can't deternmine type in advance, check for the type it will be ? I don't understand. > Furthermore, you are way underselling types with that restricted list. NameError, SyntaxError, ImportError wouldn't occur in any typed languages at runtime. Absolutly not. Those are not linked to typing in any way, and any decent Python editor catch them. > LookupError, AttributeError, AssertionError, ArithmeticError wouldn't occur in some of them. AssertionError ? Serioulsy ? Do you even know what it does in Python ? And LookupError, unless you got constant sized containers, which has nothing to do with types, you can't check that. ArithmeticError ? Come on ! Are you numbers constants ? You need to check the inputs so that they belong to the domain of your problem, at that's it. Nothing to do with types. > Finally, your breakdown doesn't cover how frequent any of these errors occur. For example, even if typing were to solve only TypeError, if TypeError consists of 90% of runtime errors, that would be a huge win. Yes but it's not. My last week have 90% keyerror, and empty values. The input is usually the source of errors. Types are useful, and they come at a cost. Whether your want to use them or not is a technical choice to make. But selling types the way it's been done on this thread is dishonest. Or you are all here working with algo problems and very little user input. Which are the easy program to code. The hard part would be the algo, not the code. If you are dealing with a user app, a web site, a video game, typing will not save you from the 90% of the bugs, and you DO need unit tests.
- naasking 10y ago
  > How would you if you can't deternmine type in advance, check for the type it will be ? I don't understand. Compilers do this all the time. Just consider all those programs that dynamically generate code based on objects that they haven't seen in advance, like object-relational mappers. Those can all be statically typed and ensure that they generate correct code [1]. > Absolutly not. Those are not linked to typing in any way, and any decent Python editor catch them. It's all part of compilation, and furthermore, we can type check code generation so that names errors don't even occur in runtime generated code. Sorry, but all of these errors are related to type checking. > AssertionError ? Serioulsy ? Do you even know what it does in Python ? http://research.microsoft.com/apps/pubs/default.aspx?id=138696 http://research.microsoft.com/apps/pubs/default.aspx?id=1386... Just google "static contract checking" to find plenty of work, including compilers already available for Haskell. Heck, code contracts have been widely deployed on .NET for years now. > And LookupError, unless you got constant sized containers, which has nothing to do with types, you can't check that. Please read the lightweight static capabilities paper I already provided. It demonstrates using the Haskell and OCaml type systems to statically assure that all array bounds are in range, even for dynamically sized structures. So yes, you can check that, which doesn't even go into dependent typing where checking such dynamic properties is the whole point. > ArithmeticError ? Come on ! Are you numbers constants ? You need to check the inputs so that they belong to the domain of your problem, at that's it. Nothing to do with types. You don't seem to realize that types are simply logical propositions about a program. They can be ANY proposition about a program, including that indexing is within bounds, that a concurrent program has no race conditions or deadlocks [2], that programs terminate, that HTML forms correctly serialize/deserialize values from/to structures [1], and more. Like most dynamic typing enthusiasts, you don't have a proper appreciation for the true scope of static typing. You are correct that typing has its costs that aren't always warranted, but you are incorrect about where you draw that line because I don't think you fully understand how powerful static typing truly is. [1] http://www.impredicative.com/ur/ http://www.impredicative.com/ur/ [2] http://www.kb.ecei.tohoku.ac.jp/~koba/typical/ http://www.kb.ecei.tohoku.ac.jp/~koba/typical/

vog 10y ago

I'd also exclude: +-- StopIteration +-- StopAsyncIterat which are used for mere communication rather than describing real error cases. Also, you should exclude: +-- SystemExit +-- KeyboardInterrupt +-- GeneratorExit +-- ImportError +-- MemoryError +-- SyntaxError | +-- IndentationError | +-- TabError because unit tests won't help you to catch these, either (because they are either thrown directly on program start, or depend of specific system circumstances rather than bugs in your code). Finally, I would argue that most instances of this exception are caught by a type system: | +-- KeyError Why? Because most of the time the key is not user input, but a hard coded string, and accessing non-existing properties of an object/struct is caught by most type systems on compile time.

sametmax 10y ago
> I'd also exclude: > > +-- StopIteration > +-- StopAsyncIterat > > which are used for mere communication rather than describing real error cases. Absolutly not. If you implement very low level logic, you can use those. I have to catch them several times in my life. > Also, you should exclude: > > +-- SystemExit > +-- KeyboardInterrupt > +-- GeneratorExit > Certainly not. You should DEFINITLY unit test for those. I do it all the time: def test_ready_keyboard_interrupt(aioloop, app): beacon = Mock() real_stop = app._finish def beacon_stop(): beacon() real_stop() app._finish = beacon_stop @app.on('running') def stahp(): beacon() raise KeyboardInterrupt() @app.on('stop') def stop(): beacon() app.ready() assert beacon.call_count == 3 You must check if you app behavre well when killed. > +-- ImportError That a type system can't check for, but a good linter can. I never have import errors because I got proper tooling. > +-- MemoryError That you should test for of course. Last week I worked on a split/merge algo on big files. I had to test that. > +-- SyntaxError > > | +-- IndentationError > | +-- TabError Well, doh :) Type system or unit tests won't catch that. Tooling, tooling, tooling. > Finally, I would argue that most instances of this exception are caught by a type system: > > | +-- KeyError No, no no ! KeyError should be super mega tested: - keys can be any mutable in Python, not string; - keys are very often generated on the fly, not constants; - dict are mutable, you can add or remove keys.

discreteevent 10y ago

"relying only on typing will cover about 10% of the error types". It's not about the percentage of error types. Its about the percentage of errors. A very high percentage of unit tests in a dynamic language are written to catch type errors. That work is a waste of time when a type system could check it for you. The reason for this is that type errors are involved in all code. A FileExistsError only applies if I am working with files. (Plus with types you get the added benefit of inline documentation and better tooling.)

gaius 10y ago
It's also about the compiler guaranteeing to enumerate every code path in and out of every function. A human writing tests is limited by what they think will happen, but the compiler knows.
sametmax 10y ago
> A very high percentage of unit tests in a dynamic language are written to catch type errors. Absolutly not. If you do it you are doing it wrong. Check this out: https://github.com/Tygs/tygs/blob/master/tests/test_components.py https://github.com/Tygs/tygs/blob/master/tests/test_componen... The type checks are "isinstance". It's 10% of the checks, and most of the time I don't even have those, my colleague insisted. The rest are the most important tests: they test behavior, not types. Do not assert a dynamic language means testing types, it's just not true. Bad testers test types.
- the_af 10y ago
  "isInstance" doesn't cover the full range of type errors. It simply covers whether something is, say, an Int or a String. Types can encode much more. They can encode behavior.
zephyrfalcon 10y ago
> A very high percentage of unit tests in a dynamic language are written to catch type errors. I'm not sure where you got that information. You don't actually write tests to catch types. You write tests to check behavior. While it is possible that actual type errors come out of the woodwork while doing that, that is not the reason why the test was written in the first place; and after fixing such a type error, the test itself is still valid (again because it is making sure that the code exhibits the desired behavior). I don't think I have EVER written a test just to check for types (in Python). Neither do I pepper my code with 'assert isinstance(x, some_class)' because it just goes against the grain of a dynamically typed language.

imtringued 10y ago

How do unit tests protect you from a corrupted database? You are making a very weak case.

sametmax 10y ago
The ones you write: this month I'm working on medical data, we got tests where we scramble the database to test the reaction of our software of course. Error reporting, cleaning, logging. You mean you don't do that ?
- deleted 10y ago
  [deleted]

deadfoxygrandpa 10y ago

Most of these would be caught at compile time in a language like Elm or (to a lesser extent) Haskell. A common pattern in some of these languages with actually strong type systems is to encode these failure possibility points in the type of the associated functions. Also, in Elm as an example, there is no runtime class generation, the type of user input that is possible is checked at compile time, and you're forced to explicitly handle many of these potential failures. If you want to crash the program if you get unexpected user input, you literally need to type "Debug.crash" and then you pretty much deserve what you get. If you're actually interested in maintainable code you should concentrate on reducing the state space of your program wherever possible, then using exhaustive (if possible) or property testing to fill in most of the rest of the gaps. Unit tests can help too, but I really think of them as less useful than all of the above.

sametmax 10y ago
You can not catch a keyerror at run time. You can not check for user input with a type system. Those have nothing to do with types. Input and output are out of your control, they need check sanitizing, and dynamic language are actually great at that.
- pka 10y ago
  You are wrong. You can very much catch "key errors" at compile time.

Artemis2 10y ago

The bugs are way more subtle than that. You're lucky if your bad programming is caught somewhere at runtime. In most cases, Python and other weakly-typed languages will do everything they can to fit the data you're providing with what the code is expecting. This means converting strings to numbers and the other way around, plus a lot of other oddities you would not always expect (see PHP's intval for a crash course in bad language design). All that results in bugs that are far from obvious, and that don't always raise neat exceptions. EDIT: I stand corrected, I don't know enough about Python in particular. The argument is still valid for other languages with weak types.

sametmax 10y ago
You clearly haven't done much Python. It doesn't do implicit convertion. Byte are separated from strings and you need to decode explicitly. You can't add '1' and 1. Python is dynamically type. It is NOT weakly typed.
to3m 10y ago
This is actually something Python doesn't do... generally, if you have a type error in Python, you get an exception at runtime. I certainly don't disagree with what I guess will be the overall direction of your argument.
zephyrfalcon 10y ago
Python is not "weakly typed", nor does it ever automatically convert strings to numbers or the other way around.
- brianwawok 10y ago
  Heck with Python3 it won't even convert bytes to String.
a3n 10y ago
> In most cases, Python and other weakly-typed languages will do everything they can to fit the data you're providing with what the code is expecting. This means converting strings to numbers ... Python is strongly typed, as well as dynamically typed. In [11]: 1 + '1' --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-11-861a99da769e> in <module>() ----> 1 1 + '1' TypeError: unsupported operand type(s) for +: 'int' and 'str'

gaius 10y ago

You absolutely can handle runtime errors via the type system just using a Maybe.