5 ms·

I suspect people will use these comments to share their wishlist of Python features, so let me add that I really wish Python had a safe navigation operator, for

I suspect people will use these comments to share their wishlist of Python features, so let me add that I really wish Python had a safe navigation operator, for when you are dealing with nested objects that could be None. I have been trying to parse a lot of XML and JSON in Python lately and a feature like that could really help reduce boilerplate checks. https://en.wikipedia.org/wiki/Safe_navigation_operator https://en.wikipedia.org/wiki/Safe_navigation_operator

nullwarp 3y ago
While not as short as a proper operator .get("key", {}) does work.
- woolion 3y ago
  Of course this only works at one level and not arbitrarily deeply, but you still need to check for None with this code; it may actually be better to write `x.get("key") or {}` so that you always get an empty dict. I write 'may' because the difference between None and an empty dict may be very subtle and rarely does the API specify with enough precision what are supposed to be the semantics of each case.
  - ddejohn 3y ago
    > Of course this only works at one level and not arbitrarily deeply It does though, you can chain as many of these as you want: some_dict.get("level_1_key", {}).get("level_2_key", {}).get("level_3_key", {})... edit: >> but you still need to check for None with this code > Not sure what you mean here. You only have to check for None if you use `.get("key")` and don't provide a fallback value. GP was talking about `{"foo": None}` and trying to drill deeper, which I misunderstood. Still, a simple try/except allows you to short-circuit the deeply nested access.
    - smilliken 3y ago
      The issue is if the dictionary has a key defined and the value is None. Your get expression will return None, causing your next access to raise an error.
      - ddejohn 3y ago
        Ah, I see. Still, I'd just catch that exception and move on because it's obvious you won't have anything more deeply nested anyway. Blowing up on `None` is an easy short-circuit.
    - kh_hk 3y ago
      >>> foo = {"foo": None} >>> print(foo.get('foo', {}).get('bar')) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'NoneType' object has no attribute 'get' >>> print((foo.get('foo') or {}).get('bar')) None
      - ddejohn 3y ago
        Right, see my other comment: > Blowing up on `None` is an easy short-circuit. The thing being discussed is attempting to access deeply nested values, so short-circuiting here is a win-win. I.e., you wouldn't want to unnecessarily traverse tons of empty dictionaries using the `.get() or {}` trick.
  - JimDabell 3y ago
    > x.get("key") or {} This has the side effect of replacing falsy values like False or 0 with an empty dictionary instead of giving you the actual value. For the exact intended behaviour, you do need to explicitly check for None instead of using a short circuit trick with or.
    - ddejohn 3y ago
      Yeah, when trying to drill down into deeply nested structures, it'd be a waste to just blindly nullish-coalesce any None objects into empty dicts instead of just immediately short-circuiting out of the deep access. It's a cute trick but would not pass code review in my shop.
    - woolion 3y ago
      The problem was underspecified, but I suggested that under the assumption that the value would be an Optional[dict]. If you have another falsy value, it means either Optional[Any] or something like Optional[Union[dict, list]] (I put list as an example, but you get the idea). I would say the only reasonable choice is Optional[dict], in which case it should be sufficient, but Python being Python, it could be anything. And in these cases you need to handle all cases way more carefully.
ivoflipse 3y ago
You can participate in the latest iteration of discussing this feature here: https://discuss.python.org/t/introducing-a-safe-navigation-operator-in-python/35480/65 https://discuss.python.org/t/introducing-a-safe-navigation-o...
GolDDranks 3y ago
Python, prouding (← is that a word?笑) itself to be the foremost language of duck typing, I'd really want to see that too.
- fritzo 3y ago
  'priding' is idiomatic English
- tipsytoad 3y ago
  Priding ;)
- esafak 3y ago
  touting
cameroncairns 3y ago
The syntax has a bit of a learning curve, but there's a library called glom that can be helpful in parsing heterogeneous JSON. Specifically it has a coalesce operator that let's you define multiple paths to get the data you are trying to get at, with a default value if none of the paths are valid. https://glom.readthedocs.io/en/latest/api.html#defaults-with-coalesce https://glom.readthedocs.io/en/latest/api.html#defaults-with... e.g. target = [{"a": {"b": "c"}}, {"a": {"c": "e"}}, {"a": {}}] results = glom(target, [Coalesce("a.b", "a.c", default="")]) # -> ["c", "e", ""]
ricree 3y ago
I believe PEP505 should cover what you're asking, but from what I can tell it seems stalled out. https://peps.python.org/pep-0505/ https://peps.python.org/pep-0505/
wodenokoto 3y ago
def safe_navigation(obj, path: List[str], default=None): try: for key in path: obj = obj[key] return obj except KeyError: return default Yeah. I thought it would just be a try-except, but that turned out pretty ugly. But really, how do you know if your path is missing or the object at the end of the path is None, without using exceptions for communicating this?
- o11c 3y ago
  I would write: def safe_navigation(obj: Any, *path: str, default: Any = None, strict: bool = False) -> Any: sentinel = object() if strict else None for key in path: obj = obj.get(key, sentinel) if obj is sentinel: return default return obj The only new disadvantage of this is that it only works on mappings and no longer works on sequences. But I really don't like having this much `Any`; it's generally a sign of poor design.
  - ddejohn 3y ago
    I wouldn't call a function that mutates the input "safe navigation", personally :)
    - o11c 3y ago
      Er, it doesn't? `.get()` (or `[]` in the original) doesn't change the object, and `obj =` only changes which object the local variable refers to? This isn't like C++ with its intrusive `=`.
      - ddejohn 3y ago
        You're right. Python is pass by reference and it's the reference being changed, not the object. This is what happens when you read code on the internet after drinks.
- ddejohn 3y ago
  There's certainly humor in a function called "safe navigation" mutating the data structure I'm supposedly safely navigating through :D
EdwardDiego 3y ago
I use Pydantic for this for JSON.
graton 3y ago
For JSON there is the `jmespath` library which might help. https://github.com/jmespath/jmespath.py https://github.com/jmespath/jmespath.py print(jmespath.search( expression="some.deep.nested.value", data={"some": {"deep": {"nested": {"value": 2}}}}, )) prints 2 print(jmespath.search( expression="some.deep.nested.value2", data={"some": {"deep": {"nested": {"value": 2}}}}, )) prints None
ziml77 3y ago
That's something I've wished for a long time that it had. I know you can use exceptions to get similar behavior but it's really ugly and I just generally don't like raising exceptions for cases that aren't actually exceptional (even though I know Python uses exceptions for normal flow with StopIteration). What's actually a bit odd about Python missing that operator is that its `or` operator acts as a rough equivalent of C#'s null-coalescing (`??`) operator. To me the safe navigation operator goes hand-in-hand with that.
- dragonwriter 3y ago
  > What's actually a bit odd about Python missing that operator is that its `or` operator acts as a rough equivalent of C#'s null-coalescing (`??`) operator. Python `or` is C# `||` plus type coercion. It is not even roughly `??` because important non-None Python values are falsey (0, False, empty collections).
  - ziml77 3y ago
    > python3 -c 'print(None or "hello")' hello It evaluates to the left if that's truthy otherwise it evaluates to the right, similar to how `??` evaluates to the left if that's non-null otherwise the right. In C# even if `||` automatically coerced the types, the output will always be a bool.
movpasd 3y ago
It's been stuck in discussions for a long time, unfortunately. The main reason that I remember gleaning from discussions is it's unclear just how "special" None should be treated as, and whether having special syntax for it is excessively "special". I saw some suggestions for overridable behaviour in the data model for None-like objects, but didn't scrutinise them closely. Either way, it seems controversial. On the one hand, the ability to override operators is really key to Python's design, but on the other it could lead to rather surprising behaviour from overeager overriding. (But to be fair, this is true of other, more common operators too.) So, for now we're stuck with `if _ is not None else`, I suppose!
rightbyte 3y ago
On a library level you could return an "empty" object which you could call methods on, returning empty objects and so on, for chaining.
- turboponyy 3y ago
  It seems that every language, given enough time, wants to tend towards Haskell!
nonameiguess 3y ago
I haven't even used Python for like four years now, but all of these suggestions to do try except and get.key() or {} seem to be ignoring that Python has had a defaultdict in the standard library for, I don't even know, at least the last decade and a half, as long as I've been paying attention. It's even written in C: https://github.com/python/cpython/blob/d9246c7b734b8958da03494045208681d95f5b74/Modules/_collectionsmodule.c#L1931 https://github.com/python/cpython/blob/d9246c7b734b8958da034....
- rwestergren 3y ago
  defaultdict doesn't seem to easily solve for the main topic of deeply nested values.
malodyets 3y ago
I created a "NoDict" class that does this awhile back: https://github.com/BlackEarth/bl/blob/main/bl/no.py https://github.com/BlackEarth/bl/blob/main/bl/no.py

nullwarp 3y ago

While not as short as a proper operator .get("key", {}) does work.

woolion 3y ago
Of course this only works at one level and not arbitrarily deeply, but you still need to check for None with this code; it may actually be better to write `x.get("key") or {}` so that you always get an empty dict. I write 'may' because the difference between None and an empty dict may be very subtle and rarely does the API specify with enough precision what are supposed to be the semantics of each case.
- ddejohn 3y ago
  > Of course this only works at one level and not arbitrarily deeply It does though, you can chain as many of these as you want: some_dict.get("level_1_key", {}).get("level_2_key", {}).get("level_3_key", {})... edit: >> but you still need to check for None with this code > Not sure what you mean here. You only have to check for None if you use `.get("key")` and don't provide a fallback value. GP was talking about `{"foo": None}` and trying to drill deeper, which I misunderstood. Still, a simple try/except allows you to short-circuit the deeply nested access.
  - smilliken 3y ago
    The issue is if the dictionary has a key defined and the value is None. Your get expression will return None, causing your next access to raise an error.
    - ddejohn 3y ago
      Ah, I see. Still, I'd just catch that exception and move on because it's obvious you won't have anything more deeply nested anyway. Blowing up on `None` is an easy short-circuit.
  - kh_hk 3y ago
    >>> foo = {"foo": None} >>> print(foo.get('foo', {}).get('bar')) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'NoneType' object has no attribute 'get' >>> print((foo.get('foo') or {}).get('bar')) None
    - ddejohn 3y ago
      Right, see my other comment: > Blowing up on `None` is an easy short-circuit. The thing being discussed is attempting to access deeply nested values, so short-circuiting here is a win-win. I.e., you wouldn't want to unnecessarily traverse tons of empty dictionaries using the `.get() or {}` trick.
- JimDabell 3y ago
  > x.get("key") or {} This has the side effect of replacing falsy values like False or 0 with an empty dictionary instead of giving you the actual value. For the exact intended behaviour, you do need to explicitly check for None instead of using a short circuit trick with or.
  - ddejohn 3y ago
    Yeah, when trying to drill down into deeply nested structures, it'd be a waste to just blindly nullish-coalesce any None objects into empty dicts instead of just immediately short-circuiting out of the deep access. It's a cute trick but would not pass code review in my shop.
  - woolion 3y ago
    The problem was underspecified, but I suggested that under the assumption that the value would be an Optional[dict]. If you have another falsy value, it means either Optional[Any] or something like Optional[Union[dict, list]] (I put list as an example, but you get the idea). I would say the only reasonable choice is Optional[dict], in which case it should be sufficient, but Python being Python, it could be anything. And in these cases you need to handle all cases way more carefully.

ivoflipse 3y ago

You can participate in the latest iteration of discussing this feature here: https://discuss.python.org/t/introducing-a-safe-navigation-operator-in-python/35480/65 https://discuss.python.org/t/introducing-a-safe-navigation-o...

GolDDranks 3y ago

Python, prouding (← is that a word?笑) itself to be the foremost language of duck typing, I'd really want to see that too.

fritzo 3y ago
'priding' is idiomatic English
tipsytoad 3y ago
Priding ;)
esafak 3y ago
touting

cameroncairns 3y ago

The syntax has a bit of a learning curve, but there's a library called glom that can be helpful in parsing heterogeneous JSON. Specifically it has a coalesce operator that let's you define multiple paths to get the data you are trying to get at, with a default value if none of the paths are valid. https://glom.readthedocs.io/en/latest/api.html#defaults-with-coalesce https://glom.readthedocs.io/en/latest/api.html#defaults-with... e.g. target = [{"a": {"b": "c"}}, {"a": {"c": "e"}}, {"a": {}}] results = glom(target, [Coalesce("a.b", "a.c", default="")]) # -> ["c", "e", ""]

ricree 3y ago

I believe PEP505 should cover what you're asking, but from what I can tell it seems stalled out. https://peps.python.org/pep-0505/ https://peps.python.org/pep-0505/

wodenokoto 3y ago

def safe_navigation(obj, path: List[str], default=None): try: for key in path: obj = obj[key] return obj except KeyError: return default Yeah. I thought it would just be a try-except, but that turned out pretty ugly. But really, how do you know if your path is missing or the object at the end of the path is None, without using exceptions for communicating this?

o11c 3y ago
I would write: def safe_navigation(obj: Any, *path: str, default: Any = None, strict: bool = False) -> Any: sentinel = object() if strict else None for key in path: obj = obj.get(key, sentinel) if obj is sentinel: return default return obj The only new disadvantage of this is that it only works on mappings and no longer works on sequences. But I really don't like having this much `Any`; it's generally a sign of poor design.
- ddejohn 3y ago
  I wouldn't call a function that mutates the input "safe navigation", personally :)
  - o11c 3y ago
    Er, it doesn't? `.get()` (or `[]` in the original) doesn't change the object, and `obj =` only changes which object the local variable refers to? This isn't like C++ with its intrusive `=`.
    - ddejohn 3y ago
      You're right. Python is pass by reference and it's the reference being changed, not the object. This is what happens when you read code on the internet after drinks.
ddejohn 3y ago
There's certainly humor in a function called "safe navigation" mutating the data structure I'm supposedly safely navigating through :D

EdwardDiego 3y ago

I use Pydantic for this for JSON.

graton 3y ago

For JSON there is the `jmespath` library which might help. https://github.com/jmespath/jmespath.py https://github.com/jmespath/jmespath.py print(jmespath.search( expression="some.deep.nested.value", data={"some": {"deep": {"nested": {"value": 2}}}}, )) prints 2 print(jmespath.search( expression="some.deep.nested.value2", data={"some": {"deep": {"nested": {"value": 2}}}}, )) prints None

ziml77 3y ago

That's something I've wished for a long time that it had. I know you can use exceptions to get similar behavior but it's really ugly and I just generally don't like raising exceptions for cases that aren't actually exceptional (even though I know Python uses exceptions for normal flow with StopIteration). What's actually a bit odd about Python missing that operator is that its `or` operator acts as a rough equivalent of C#'s null-coalescing (`??`) operator. To me the safe navigation operator goes hand-in-hand with that.

dragonwriter 3y ago
> What's actually a bit odd about Python missing that operator is that its `or` operator acts as a rough equivalent of C#'s null-coalescing (`??`) operator. Python `or` is C# `||` plus type coercion. It is not even roughly `??` because important non-None Python values are falsey (0, False, empty collections).
- ziml77 3y ago
  > python3 -c 'print(None or "hello")' hello It evaluates to the left if that's truthy otherwise it evaluates to the right, similar to how `??` evaluates to the left if that's non-null otherwise the right. In C# even if `||` automatically coerced the types, the output will always be a bool.

movpasd 3y ago

It's been stuck in discussions for a long time, unfortunately. The main reason that I remember gleaning from discussions is it's unclear just how "special" None should be treated as, and whether having special syntax for it is excessively "special". I saw some suggestions for overridable behaviour in the data model for None-like objects, but didn't scrutinise them closely. Either way, it seems controversial. On the one hand, the ability to override operators is really key to Python's design, but on the other it could lead to rather surprising behaviour from overeager overriding. (But to be fair, this is true of other, more common operators too.) So, for now we're stuck with `if _ is not None else`, I suppose!

rightbyte 3y ago

On a library level you could return an "empty" object which you could call methods on, returning empty objects and so on, for chaining.

turboponyy 3y ago
It seems that every language, given enough time, wants to tend towards Haskell!

nonameiguess 3y ago

I haven't even used Python for like four years now, but all of these suggestions to do try except and get.key() or {} seem to be ignoring that Python has had a defaultdict in the standard library for, I don't even know, at least the last decade and a half, as long as I've been paying attention. It's even written in C: https://github.com/python/cpython/blob/d9246c7b734b8958da03494045208681d95f5b74/Modules/_collectionsmodule.c#L1931 https://github.com/python/cpython/blob/d9246c7b734b8958da034....

rwestergren 3y ago
defaultdict doesn't seem to easily solve for the main topic of deeply nested values.

malodyets 3y ago

I created a "NoDict" class that does this awhile back: https://github.com/BlackEarth/bl/blob/main/bl/no.py https://github.com/BlackEarth/bl/blob/main/bl/no.py