6 ms·

Opaque Types in Python

131 points by lumpa 5mo ago

corwinxpro 5mo ago
The main problem with such approach is that `class _RealShipOpts:` is very ugly to write unit tests for. You need to import a private entity in tests. I would slightly change the presented approach, and move the "public" `ShippingOptions`, `shipFast`, etc., into a new module that is a public API, for my users to use something like `from my_lib.shipping.api import ShippingOptions`. That way, I can use "normal" naming in `class RealShipOpts:...`, and be explicit that it's not really public for the end users (they should use the `.api` module instead).
- sanderjd 4mo ago
  Why wouldn't you just write the tests against the public api?
jnwatson 5mo ago
You're holding it (Python) wrong. Python OO was a counter reaction to the bondage and discipline that languages like C++ had with private members and protected inheritance. If you have members that users probably shouldn't touch, you prepend them with an underscore. This is just a hint; It doesn't actually change anything. We're all adults here and we know the consequences of reaching into implementation details.
- ddavis 5mo ago
  I agreed with this 100% for a long time. Then I started working on a library at $WORK with dozens of downstream users abusing the hell out of my idiomatic underscore usage, especially in the context of lazy tests with folks writing endless mocks. When I’d “break” their test suite (blocking some time sensitive release) I’d get all kinds of shit. But _they_ were breaking the contract. Unfortunately I had little (if any) control on the path of application code making it to production (yeah yeah not great engineering org, but it’s the world I lived in). Strategies like this post would be helpful for said situations.
  - jghn 5mo ago
    Something similar happened to me. I told those groups to pound sand because they knew they were relying on something which they should not. Manager had my back, they whined a lot but they had to change and improve their processes.
  - senkora 5mo ago
    There’s always the extra idiomatic __SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED for coworkers that can’t take a hint. https://github.com/reactjs/react.dev/issues/3896 https://github.com/reactjs/react.dev/issues/3896
    - nomel 4mo ago
      You can technically "enforce" this at runtime with __getattribute__ or decorators[1]. Maybe be evil and add 1000 "Private access not allowed: {name}" with 1 second delays between each. Aaaand, this flexibility is exactly why python is slow. [1] https://pypi.org/project/accessify/ https://pypi.org/project/accessify/
  - 1-more 4mo ago
    My secret for test-only code is to provide functions whose first argument is a value that only exist in a test scenario. There are also static analysis rules for the linter in my language of choice that disallows test-only functions being called outside of test files. This gets you closer to "we're actually adults here" but still a tiny bit cheatable.
  - taeric 4mo ago
    I'm torn. On the one hand, this is not too uncommon of a problem to run into. On the other, poor practices from coworkers are not going to go away thanks to a language filter. So, the question will come down to which causes more grief, people abusing this convention, or people that overly use the language features that combat it? It is the standard optimization question between poor practices and enforcement that you have in any question of enforcement. I would be delighted if we could get some empirical data on this.
  - 4mo ago
- sdeframond 5mo ago
  > We're all adults here and we know the consequences of reaching into implementation details. I wish you were right but, IMHE, it requires a lot of communication once teams grow and many team member do not fully understand the consequences of what they do. It is nice to have something that helps when reviewing code. > If you have members that users probably shouldn't touch, you prepend them with an underscore Well, this is precisely what TFA does. It prepends the constructor with an underscore.
- masklinn 5mo ago
  I think you missed the issue at hand: > even if you keep all your fields private, the constructor is still, inherently, public. ShippingOptions and the literals / enums are part of the public API, so the user would just be writing ShippingOptions(Carrier.USPS, Conveyance.Air) with no hint that they're doing anything wrong. Dataclasses do have a `kw_only` option, but I'm not sure how well underscore prefixes would be understood as private parameters / a private ctor, whereas wrapping a clearly "private" type should be clear to everybody. Glyph is not entirely correct on the "any class" bit as you can always break the default init path: class ShippingOptions: _ship: Literal["fast", "normal", "slow"] __init__ = None def shipFast() -> ShippingOptions: opts = object.__new__(ShippingOptions) opts._ship = "fast" return opts however that's a pretty ugly pattern, and unlike the one they propose I doubt tooling would understand it.
- 5691827 5mo ago
  "Glyph" knows. He has been in the Python inner circle for decades back to when the circle promoted "spam and eggs" and "consenting adults". Like the rest of that circle, he moves with the times, supports public shaming of Tim Peters and others and now promotes poorly implemented information hiding so Python ticks a few more boxes for the industry. Information hiding in a language that allows changing the values of small integers at runtime via ctypes is doomed anyway. And there are plenty of better languages that do it out of the box and in a straightforward manner.
  - sanderjd 4mo ago
    I don't know who any of these people are. This seems very gossipy.
- prerok 5mo ago
  Are we, though? No offence but if an external module starts calling an internal function they better prepare a PR that changes the internal (yes, I know, by convention) to a public one. This is a signal to the developers of that module that they have to maintain the behavior. That PR might well be rejected. And you have to work with the module owners to get your case supported. Anything else is not responsible and I would not call it "adult".
  - dec0dedab0de 4mo ago
    or they could just test and maintain the hack on their own.
    - prerok 4mo ago
      Yeah, good luck with that. I've had too many libraries change underneath me by clearly violating the previous contract that I know it doesn't work.
- zem 5mo ago
  nothing is stopping adult users from disabling the type checker and using your internal type directly. the newtype is just a private class mechanism that comes with better tooling to validate that you aren't breaking the intended contract.
- deleted 5mo ago
  [deleted]
sdeframond 5mo ago
Funny, I ran into the same pattern just a few months ago! In practice, I found it difficult for coworkers to read and understand so I dropped the idea. Another limitation I found is that it breaks down when you start using inheritance. For example: ``` class _A: pass A = NewType("A", _A) class _B(_A): pass B = NewType("B", _B) def foo(a: A) -> None: pass b = B(_B()) foo(b) # Mypy is not happy: Argument 1 to "foo" has incompatible type "B"; expected "A" foo(A(b)) # Mypy is OK ```
- simonw 5mo ago
  (On Hacker News you can do code blocks by indenting each line with two spaces.)
  - sdeframond 5mo ago
    Aaah nice! Thank you!
- whilenot-dev 5mo ago
  Just use a generic and make it bound to (A, B): from typing import * class _A: pass class _B(_A): pass A = NewType("A", _A) B = NewType("B", _B) def foo[T: (A, B)](val: T) -> T: return val a = A(_A()) b = B(_B()) _a = foo(a) _b = foo(b) reveal_type(_a) reveal_type(_b) Playground here: https://mypy-play.net/?mypy=latest&python=3.12&gist=3657336346fd926ac50cccccfc09696a https://mypy-play.net/?mypy=latest&python=3.12&gist=36573363...
  - sanderjd 4mo ago
    This does seem like an abstraction leak though.
    - whilenot-dev 4mo ago
      The abstraction gets leaky once you expect the distinct NewTypes to adhere to the original inheritance property. I think that's a wrong assumption from the get-go. OP could just do: def foo(val: _A) -> None: pass ...and it'll accept both NewTypes just fine. I guess it depends on whether foo is designed to be public or private.
tcdent 5mo ago
I'm sorry but if you write Python functions/methods in camel case I can't take you seriously.
- jdnier 5mo ago
  Just fyi, the author is https://en.wikipedia.org/wiki/Glyph_Lefkowitz https://en.wikipedia.org/wiki/Glyph_Lefkowitz, creator of Twisted.
  - TACD 5mo ago
    Seems like this is even more reason to set a good example and follow the style guide.
    - simonw 5mo ago
      Twisted has had its own style guide for decades: https://docs.twisted.org/en/stable/development/coding-standard.html#methods https://docs.twisted.org/en/stable/development/coding-standa... (Earliest mention I could find for camelCase methods was 25 years ago: https://github.com/twisted/twisted/blob/d7c19cd40d07c8c37f856c9a364881ce2179988e/doc/CodingStandard.html#L136-L141 https://github.com/twisted/twisted/blob/d7c19cd40d07c8c37f85... )
      - flexagoon 5mo ago
        > get_someAttribute Hell no
    - dxdm 5mo ago
      Maybe, but the position to not listen to him at all because he doesn't do that is still a much worse look in my book.
  - lmm 4mo ago
    Wow, that's a much better reason to not take them seriously.
nayuki 5mo ago
Java made opaque types possible from the very start by private and package-private constructors. It's sad to see that many features regarding object-oriented programming and static typing are implemented worse in Python than Java. Various examples: __str__() vs. toString(); underscore vs. private; @staticmethod/@classmethod vs. static; generic types are so clunky in Python; types are not shown in the official Python standand library documentation; __init__() doesn't force you to call super() whereas it's mandatory in Java; @override (Python 3.12; year 2023) copying Java @Override (JDK 1.5; year 2004) very late; convention changing from duck typing (always available in Python) to structural typing (optional in Python, mandatory in Java).
- causal 5mo ago
  Python is much older than Java, and Java is a big OO-first language. It's a bit like saying Python doesn't do functional as well as Erlang.
  - qrobit 5mo ago
    Much older? Wikipedia says[^1][^2] java appeared in 1995 (started in 1991), while python appeared in 1991 (started in late 1980s). 4 years doesn't seem too far apart, considering both language are >30 years old now. [^1]: https://en.wikipedia.org/wiki/Python_(programming_language) https://en.wikipedia.org/wiki/Python_(programming_language) [^2]: https://en.wikipedia.org/wiki/Java_(programming_language) https://en.wikipedia.org/wiki/Java_(programming_language)
    - nayuki 5mo ago
      Not only that, but Python had the benefit of doing a very painful break in version 3 (2008), when they had the option to cleaned up almost anything they wanted. (Some changes in Python 3 I can recall: bytes/str/unicode being the biggest one; fixing mutable variables in nested functions; changing some obscure behavior in class hierarchies and overload resolution; changing things like range() and map() to lazy evaluation.) For better or for worse, Java has maintained very good (not perfect) compatibility throughout, even with painful changes like generics in 1.5, lambdas in 8, modules in 9, eventual removal of applets and SecurityManager, etc. This also contrasts with C#/.NET, which I think had some breaking changes over the decades.
CreRecombinase 5mo ago
Why not just use a dictionary, or why not just leave the type unannotated? If you really can't (or don't want to) say anything about the type, then don't. Python is dynamically typed!
- techn00 5mo ago
  average python script writer
- sdeframond 5mo ago
  The point is to mark the constructor as "private" so that it is easy to spot unintended use during code reviews (or using linters).
- MeetingsBrowser 5mo ago
  The blog post does want to share some type information with users. They just want to prevent users from relying on a specific implementation of that type. They are basically describing a public API backed by a private type that they can extend, rearrange, or otherwise modify without breaking the public contract.
gorgoiler 5mo ago
An alternative to consider might be to accept a Literal[“fast”, “slow”] or an Enum FAST or SLOW, and then decode that into shipping options inside the shipping code. Only then are you truly putting a solid boundary between your library and the folks using your library. Everything else is just praying that you and only you have an underscore on your keyboard! :) And of course another alternative is to accept that there is no true private in Python other than defdef*, so you allow your ShippingOption to be publicly visible while also documenting that the helper-constructors are what should really be used. *”defdef” as in function definitions inside other function definitions — closures if you will, although I prefer to write mine as taking most if not all their parameters explicitly: def public(foo): def private(foo): … class Private: … # less common …
OutOfHere 4mo ago
Do not ever do this Javaesque nonsense in Python or anywhere really. Use ShippingOptions directly.
stephenlf 4mo ago
The conversation here is surprisingly devoid of comments on name mangling, which _almost_ enforces private properties. https://docs.python.org/3/reference/expressions.html https://docs.python.org/3/reference/expressions.html
deleted 4mo ago
[deleted]
- hun3 4mo ago
  _RealShipOpts has a constructor.

131 points by lumpa 5mo ago

The main problem with such approach is that `class _RealShipOpts:` is very ugly to write unit tests for. You need to import a private entity in tests. I would slightly change the presented approach, and move the "public" `ShippingOptions`, `shipFast`, etc., into a new module that is a public API, for my users to use something like `from my_lib.shipping.api import ShippingOptions`. That way, I can use "normal" naming in `class RealShipOpts:...`, and be explicit that it's not really public for the end users (they should use the `.api` module instead).

sanderjd 4mo ago
Why wouldn't you just write the tests against the public api?

jnwatson 5mo ago

You're holding it (Python) wrong. Python OO was a counter reaction to the bondage and discipline that languages like C++ had with private members and protected inheritance. If you have members that users probably shouldn't touch, you prepend them with an underscore. This is just a hint; It doesn't actually change anything. We're all adults here and we know the consequences of reaching into implementation details.

ddavis 5mo ago
I agreed with this 100% for a long time. Then I started working on a library at $WORK with dozens of downstream users abusing the hell out of my idiomatic underscore usage, especially in the context of lazy tests with folks writing endless mocks. When I’d “break” their test suite (blocking some time sensitive release) I’d get all kinds of shit. But _they_ were breaking the contract. Unfortunately I had little (if any) control on the path of application code making it to production (yeah yeah not great engineering org, but it’s the world I lived in). Strategies like this post would be helpful for said situations.
- jghn 5mo ago
  Something similar happened to me. I told those groups to pound sand because they knew they were relying on something which they should not. Manager had my back, they whined a lot but they had to change and improve their processes.
- senkora 5mo ago
  There’s always the extra idiomatic __SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED for coworkers that can’t take a hint. https://github.com/reactjs/react.dev/issues/3896 https://github.com/reactjs/react.dev/issues/3896
  - nomel 4mo ago
    You can technically "enforce" this at runtime with __getattribute__ or decorators[1]. Maybe be evil and add 1000 "Private access not allowed: {name}" with 1 second delays between each. Aaaand, this flexibility is exactly why python is slow. [1] https://pypi.org/project/accessify/ https://pypi.org/project/accessify/
- 1-more 4mo ago
  My secret for test-only code is to provide functions whose first argument is a value that only exist in a test scenario. There are also static analysis rules for the linter in my language of choice that disallows test-only functions being called outside of test files. This gets you closer to "we're actually adults here" but still a tiny bit cheatable.
- taeric 4mo ago
  I'm torn. On the one hand, this is not too uncommon of a problem to run into. On the other, poor practices from coworkers are not going to go away thanks to a language filter. So, the question will come down to which causes more grief, people abusing this convention, or people that overly use the language features that combat it? It is the standard optimization question between poor practices and enforcement that you have in any question of enforcement. I would be delighted if we could get some empirical data on this.
- 4mo ago
sdeframond 5mo ago
> We're all adults here and we know the consequences of reaching into implementation details. I wish you were right but, IMHE, it requires a lot of communication once teams grow and many team member do not fully understand the consequences of what they do. It is nice to have something that helps when reviewing code. > If you have members that users probably shouldn't touch, you prepend them with an underscore Well, this is precisely what TFA does. It prepends the constructor with an underscore.
masklinn 5mo ago
I think you missed the issue at hand: > even if you keep all your fields private, the constructor is still, inherently, public. ShippingOptions and the literals / enums are part of the public API, so the user would just be writing ShippingOptions(Carrier.USPS, Conveyance.Air) with no hint that they're doing anything wrong. Dataclasses do have a `kw_only` option, but I'm not sure how well underscore prefixes would be understood as private parameters / a private ctor, whereas wrapping a clearly "private" type should be clear to everybody. Glyph is not entirely correct on the "any class" bit as you can always break the default init path: class ShippingOptions: _ship: Literal["fast", "normal", "slow"] __init__ = None def shipFast() -> ShippingOptions: opts = object.__new__(ShippingOptions) opts._ship = "fast" return opts however that's a pretty ugly pattern, and unlike the one they propose I doubt tooling would understand it.
5691827 5mo ago
"Glyph" knows. He has been in the Python inner circle for decades back to when the circle promoted "spam and eggs" and "consenting adults". Like the rest of that circle, he moves with the times, supports public shaming of Tim Peters and others and now promotes poorly implemented information hiding so Python ticks a few more boxes for the industry. Information hiding in a language that allows changing the values of small integers at runtime via ctypes is doomed anyway. And there are plenty of better languages that do it out of the box and in a straightforward manner.
- sanderjd 4mo ago
  I don't know who any of these people are. This seems very gossipy.
prerok 5mo ago
Are we, though? No offence but if an external module starts calling an internal function they better prepare a PR that changes the internal (yes, I know, by convention) to a public one. This is a signal to the developers of that module that they have to maintain the behavior. That PR might well be rejected. And you have to work with the module owners to get your case supported. Anything else is not responsible and I would not call it "adult".
- dec0dedab0de 4mo ago
  or they could just test and maintain the hack on their own.
  - prerok 4mo ago
    Yeah, good luck with that. I've had too many libraries change underneath me by clearly violating the previous contract that I know it doesn't work.
zem 5mo ago
nothing is stopping adult users from disabling the type checker and using your internal type directly. the newtype is just a private class mechanism that comes with better tooling to validate that you aren't breaking the intended contract.
deleted 5mo ago
[deleted]

sdeframond 5mo ago

Funny, I ran into the same pattern just a few months ago! In practice, I found it difficult for coworkers to read and understand so I dropped the idea. Another limitation I found is that it breaks down when you start using inheritance. For example: ``` class _A: pass A = NewType("A", _A) class _B(_A): pass B = NewType("B", _B) def foo(a: A) -> None: pass b = B(_B()) foo(b) # Mypy is not happy: Argument 1 to "foo" has incompatible type "B"; expected "A" foo(A(b)) # Mypy is OK ```

simonw 5mo ago
(On Hacker News you can do code blocks by indenting each line with two spaces.)
- sdeframond 5mo ago
  Aaah nice! Thank you!
whilenot-dev 5mo ago
Just use a generic and make it bound to (A, B): from typing import * class _A: pass class _B(_A): pass A = NewType("A", _A) B = NewType("B", _B) def foo[T: (A, B)](val: T) -> T: return val a = A(_A()) b = B(_B()) _a = foo(a) _b = foo(b) reveal_type(_a) reveal_type(_b) Playground here: https://mypy-play.net/?mypy=latest&python=3.12&gist=3657336346fd926ac50cccccfc09696a https://mypy-play.net/?mypy=latest&python=3.12&gist=36573363...
- sanderjd 4mo ago
  This does seem like an abstraction leak though.
  - whilenot-dev 4mo ago
    The abstraction gets leaky once you expect the distinct NewTypes to adhere to the original inheritance property. I think that's a wrong assumption from the get-go. OP could just do: def foo(val: _A) -> None: pass ...and it'll accept both NewTypes just fine. I guess it depends on whether foo is designed to be public or private.

tcdent 5mo ago

I'm sorry but if you write Python functions/methods in camel case I can't take you seriously.

jdnier 5mo ago
Just fyi, the author is https://en.wikipedia.org/wiki/Glyph_Lefkowitz https://en.wikipedia.org/wiki/Glyph_Lefkowitz, creator of Twisted.
- TACD 5mo ago
  Seems like this is even more reason to set a good example and follow the style guide.
  - simonw 5mo ago
    Twisted has had its own style guide for decades: https://docs.twisted.org/en/stable/development/coding-standard.html#methods https://docs.twisted.org/en/stable/development/coding-standa... (Earliest mention I could find for camelCase methods was 25 years ago: https://github.com/twisted/twisted/blob/d7c19cd40d07c8c37f856c9a364881ce2179988e/doc/CodingStandard.html#L136-L141 https://github.com/twisted/twisted/blob/d7c19cd40d07c8c37f85... )
    - flexagoon 5mo ago
      > get_someAttribute Hell no
  - dxdm 5mo ago
    Maybe, but the position to not listen to him at all because he doesn't do that is still a much worse look in my book.
- lmm 4mo ago
  Wow, that's a much better reason to not take them seriously.

nayuki 5mo ago

Java made opaque types possible from the very start by private and package-private constructors. It's sad to see that many features regarding object-oriented programming and static typing are implemented worse in Python than Java. Various examples: __str__() vs. toString(); underscore vs. private; @staticmethod/@classmethod vs. static; generic types are so clunky in Python; types are not shown in the official Python standand library documentation; __init__() doesn't force you to call super() whereas it's mandatory in Java; @override (Python 3.12; year 2023) copying Java @Override (JDK 1.5; year 2004) very late; convention changing from duck typing (always available in Python) to structural typing (optional in Python, mandatory in Java).

causal 5mo ago
Python is much older than Java, and Java is a big OO-first language. It's a bit like saying Python doesn't do functional as well as Erlang.
- qrobit 5mo ago
  Much older? Wikipedia says[^1][^2] java appeared in 1995 (started in 1991), while python appeared in 1991 (started in late 1980s). 4 years doesn't seem too far apart, considering both language are >30 years old now. [^1]: https://en.wikipedia.org/wiki/Python_(programming_language) https://en.wikipedia.org/wiki/Python_(programming_language) [^2]: https://en.wikipedia.org/wiki/Java_(programming_language) https://en.wikipedia.org/wiki/Java_(programming_language)
  - nayuki 5mo ago
    Not only that, but Python had the benefit of doing a very painful break in version 3 (2008), when they had the option to cleaned up almost anything they wanted. (Some changes in Python 3 I can recall: bytes/str/unicode being the biggest one; fixing mutable variables in nested functions; changing some obscure behavior in class hierarchies and overload resolution; changing things like range() and map() to lazy evaluation.) For better or for worse, Java has maintained very good (not perfect) compatibility throughout, even with painful changes like generics in 1.5, lambdas in 8, modules in 9, eventual removal of applets and SecurityManager, etc. This also contrasts with C#/.NET, which I think had some breaking changes over the decades.

CreRecombinase 5mo ago

Why not just use a dictionary, or why not just leave the type unannotated? If you really can't (or don't want to) say anything about the type, then don't. Python is dynamically typed!

techn00 5mo ago
average python script writer
sdeframond 5mo ago
The point is to mark the constructor as "private" so that it is easy to spot unintended use during code reviews (or using linters).
MeetingsBrowser 5mo ago
The blog post does want to share some type information with users. They just want to prevent users from relying on a specific implementation of that type. They are basically describing a public API backed by a private type that they can extend, rearrange, or otherwise modify without breaking the public contract.

gorgoiler 5mo ago

An alternative to consider might be to accept a Literal[“fast”, “slow”] or an Enum FAST or SLOW, and then decode that into shipping options inside the shipping code. Only then are you truly putting a solid boundary between your library and the folks using your library. Everything else is just praying that you and only you have an underscore on your keyboard! :) And of course another alternative is to accept that there is no true private in Python other than defdef*, so you allow your ShippingOption to be publicly visible while also documenting that the helper-constructors are what should really be used. *”defdef” as in function definitions inside other function definitions — closures if you will, although I prefer to write mine as taking most if not all their parameters explicitly: def public(foo): def private(foo): … class Private: … # less common …

OutOfHere 4mo ago

Do not ever do this Javaesque nonsense in Python or anywhere really. Use ShippingOptions directly.

stephenlf 4mo ago

The conversation here is surprisingly devoid of comments on name mangling, which _almost_ enforces private properties. https://docs.python.org/3/reference/expressions.html https://docs.python.org/3/reference/expressions.html

deleted 4mo ago

[deleted]

hun3 4mo ago
_RealShipOpts has a constructor.