4 ms·

But again: why do you need to care about the whole sorted state thing? "Sorting" a dictionary keyed by index is linear time. Like the cost of doing that isn't

But again: why do you need to care about the whole sorted state thing? "Sorting" a dictionary keyed by index is linear time. Like the cost of doing that isn't going to be greater than the existing cost of keyword arguments, which can already be arbitrarily ordered! Like, what about this is more "preposterous" than passing a massive dict to kwargs already?

necovek 3y ago
Passing a dict is passing by reference, and if the called method reads 5 elements from that dict, it's tiny constant time. Sorting of a large collection happening implicitly is too expensive for what would generally be a benign mistake.
- joshuamorton 3y ago
  But again: you don't actually need to sort anything. Like that's really important you don't need to sort anything! Passing kwargs requires resolving n references, one for each kwarg. Passing args by index requires passing n references. Like in code: def convert_dict_to_args(f, magic_kwargs): out_args = [] out_kwargs = {} for k, v in magic_kwargs: if isinstance(k, int): out_args[k] = v else: out_kwargs[k] = v return f(*out_args, **out_kwargs) There's no sorting necessary! There's potential particular optimizations that might need to happen to make this work with python's opcodes for fast function calls, since out_args isn't constructed as a tuple which is internally optimized in function calls, but afaik you're already not getting those optimizations/fast paths if you're passing by name and/or using kwargs, so this isn't a change. Edit: for context, >>> from timeit import timeit as t >>> t(stmt='f(1,2,3)', setup='def f(a, b, c): pass', number=int(1e8)) 5.765844099999981 >>> t(stmt='f(*(1,2,3))', setup='def f(a, b, c): pass', number=int(1e8)) 5.840746500000023 >>> t(stmt='f(a=1,b=2,c=3)', setup='def f(a, b, c): pass', number=int(1e8)) 7.400104499999998 >>> t(stmt='f(**d)', setup='d=dict(a=1,b=2,c=3)\ndef f(a, b, c): pass', number=int(1e8)) 20.779321399999958 >>> t(stmt='f(**d)', setup='d=dict()\ndef f(a=None, b=None, c=None): pass', number=int(1e8)) 9.583819900000094 So yeah using kwargs at all has a significant performance penalty, so you're already hitting performance penalties for using kwargs, keeping a sorted view isn't going to help sinificantly.
  - necovek 3y ago
    Since "sorting is not necessary", why don't you try: >> out_args = [] >> out_args[3] = 'foo' ? And if it's still not clear, pass magic_kwargs = { 'arg1': 'bar', 2: 'foo', 'arg2': 'yeah', 1: 'not', 0: 'really'} to that helper you wrote.
    - joshuamorton 3y ago
      > >> out_args = [] >> out_args[3] = 'foo' Yes, this should be a typeerror. I elided any kind of error checking, but you still don't need to sort to check that. You run into the same issue if you pass `*range(5)` to a function of one argument today. > magic_kwargs = { 'arg1': 'bar', 2: 'foo', 'arg2': 'yeah', 1: 'not', 0: 'really'} Again, this would be a TypeError (assuming the underlying function is defined as `def (arg1, arg2, ...) ` or something. And yet again, there's no need to sort anything to detect that error. I feel like the thing you're getting at here is that you get index out of range errors given the half-assed implementation I did, which is absolutely true. It's buggy, it's broken, it's demo code. But you solve that by finding the max value of the integers keys of the dict, but finding the maximum value at worst requires a second linear scan, not a sort. You never need to sort the keys, you can always place them or error in linear time. For a fuller implementation: def convert_dict_to_args(f, magic_kwargs): _max = 0 for k in magic_kwargs.keys(): if isinstance(k, int): if k < 0: raise TypeError if k > _max: _max = k elif not isinstance(k, str): raise TypeError _SENTINEL = object() out_args = [_SENTINEL for _ in range(_max) out_kwargs = {} for k, v in magic_kwargs.items(): if isinstance(k, int): out_args[k] = v else: out_kwargs[k] = v if _SENTINEL in out_args: raise TypeError return f(*out_args, **out_kwargs) And this version functions as expected, still without any sorting required: args = ('really', 'not', 'foo') kwargs = {'arg1': 'bar', 'arg2': 'yeah'}
      - necovek 3y ago
        You've basically sorted those int-keyed elements into a list, only optimized by making a few assumptions (eg. that all indexes are present up to a max: if you want to error check that like a real implementation would have to, you lose some of that optimization since you now have to keep track of missing elements, which is at least another loop through the list). All of these optimizations are applicable to sorting a sparse, bounded list of integers, but that's still sorting them.

necovek 3y ago

Passing a dict is passing by reference, and if the called method reads 5 elements from that dict, it's tiny constant time. Sorting of a large collection happening implicitly is too expensive for what would generally be a benign mistake.

joshuamorton 3y ago
But again: you don't actually need to sort anything. Like that's really important you don't need to sort anything! Passing kwargs requires resolving n references, one for each kwarg. Passing args by index requires passing n references. Like in code: def convert_dict_to_args(f, magic_kwargs): out_args = [] out_kwargs = {} for k, v in magic_kwargs: if isinstance(k, int): out_args[k] = v else: out_kwargs[k] = v return f(*out_args, **out_kwargs) There's no sorting necessary! There's potential particular optimizations that might need to happen to make this work with python's opcodes for fast function calls, since out_args isn't constructed as a tuple which is internally optimized in function calls, but afaik you're already not getting those optimizations/fast paths if you're passing by name and/or using kwargs, so this isn't a change. Edit: for context, >>> from timeit import timeit as t >>> t(stmt='f(1,2,3)', setup='def f(a, b, c): pass', number=int(1e8)) 5.765844099999981 >>> t(stmt='f(*(1,2,3))', setup='def f(a, b, c): pass', number=int(1e8)) 5.840746500000023 >>> t(stmt='f(a=1,b=2,c=3)', setup='def f(a, b, c): pass', number=int(1e8)) 7.400104499999998 >>> t(stmt='f(**d)', setup='d=dict(a=1,b=2,c=3)\ndef f(a, b, c): pass', number=int(1e8)) 20.779321399999958 >>> t(stmt='f(**d)', setup='d=dict()\ndef f(a=None, b=None, c=None): pass', number=int(1e8)) 9.583819900000094 So yeah using kwargs at all has a significant performance penalty, so you're already hitting performance penalties for using kwargs, keeping a sorted view isn't going to help sinificantly.
- necovek 3y ago
  Since "sorting is not necessary", why don't you try: >> out_args = [] >> out_args[3] = 'foo' ? And if it's still not clear, pass magic_kwargs = { 'arg1': 'bar', 2: 'foo', 'arg2': 'yeah', 1: 'not', 0: 'really'} to that helper you wrote.
  - joshuamorton 3y ago
    > >> out_args = [] >> out_args[3] = 'foo' Yes, this should be a typeerror. I elided any kind of error checking, but you still don't need to sort to check that. You run into the same issue if you pass `*range(5)` to a function of one argument today. > magic_kwargs = { 'arg1': 'bar', 2: 'foo', 'arg2': 'yeah', 1: 'not', 0: 'really'} Again, this would be a TypeError (assuming the underlying function is defined as `def (arg1, arg2, ...) ` or something. And yet again, there's no need to sort anything to detect that error. I feel like the thing you're getting at here is that you get index out of range errors given the half-assed implementation I did, which is absolutely true. It's buggy, it's broken, it's demo code. But you solve that by finding the max value of the integers keys of the dict, but finding the maximum value at worst requires a second linear scan, not a sort. You never need to sort the keys, you can always place them or error in linear time. For a fuller implementation: def convert_dict_to_args(f, magic_kwargs): _max = 0 for k in magic_kwargs.keys(): if isinstance(k, int): if k < 0: raise TypeError if k > _max: _max = k elif not isinstance(k, str): raise TypeError _SENTINEL = object() out_args = [_SENTINEL for _ in range(_max) out_kwargs = {} for k, v in magic_kwargs.items(): if isinstance(k, int): out_args[k] = v else: out_kwargs[k] = v if _SENTINEL in out_args: raise TypeError return f(*out_args, **out_kwargs) And this version functions as expected, still without any sorting required: args = ('really', 'not', 'foo') kwargs = {'arg1': 'bar', 'arg2': 'yeah'}
    - necovek 3y ago
      You've basically sorted those int-keyed elements into a list, only optimized by making a few assumptions (eg. that all indexes are present up to a max: if you want to error check that like a real implementation would have to, you lose some of that optimization since you now have to keep track of missing elements, which is at least another loop through the list). All of these optimizations are applicable to sorting a sparse, bounded list of integers, but that's still sorting them.