15 ms·
Data Classification: Does Python still have a need for class without dataclass?
- rollcat 4y agoI think: data Point3D(x, y, z) from Vector: ... is a terrible syntax, there are plenty of legitimate use cases where you'd have an overwhelming amount of fields in a class/structure, and listing them line by line (as well as explicitly specifying accessors to handle the couple special cases) would end up much more readable. I'd rather see something more along these lines: data Parrot(Animal): name age: int alive: bool = False purchased_in: Shop | None = field(default_factory=get_shop) ... (Move "field" to builtins, etc.) But overall this proposal seems like a no-brainer, even before the addition of the match statement. The "classic" __init__ is a jarring violation of DRY, baked right into one of the most common idioms in the language, and gets in the way of not just the "QoL" features like destructuring or type checking, but also of the efforts to build more efficient implementations that could pre-allocate object memory without speculative analysis, or optimise the dict lookup away. The question is not "shall we", but "why wasn't it done in 2008".
- JohnDeHope 4y agoYou made me think "I think: [all of python] is a terrible syntax". At some point in the past, python crossed over into perl or php territory for me. The semantics, runtime, culture, support, library availability, etc. are all great. But jiminy no syntax can keep up with the decades of layers of cruft these languages accumulate. I'm starting to look at C# the same way.
- rollcat 4y agoYou probably want a Lisp or a Forth instead then. Maybe (maybe!) Lua.
- kzrdude 4y agoPython is a multi-paradigm language. And not only that, I think there are many reasons to want to continue having non-dataclass classes. Dataclass fits best with record-type data and not with encapsulated data or classes with mostly private attributes.
- HelloNurse 4y agoThe Python baseline is that a class does not know, and doesn't pretend to know, what attributes it could contain, everything is welcome unless extra code is added to repress rogue assignments. Trivial getters and setters, __init__ methods that copy their parameters to members of the same name, and other boring Java-like patterns are certainly served well by convenience tools like the attrs library or dataclasses, but distinguishing records from serious classes requires significant skill.
- eru 4y ago> [...], everything is welcome unless extra code is added to repress rogue assignments. I think slots do that, don't they?
- HelloNurse 4y agoYes, and dataclasses offer an even nicer notation.
- kzrdude 4y agoDataclasses make it very easy to define such classes indeed but they don't disallow extra attributes. So they really compose with a lot of other techniques
- packetlost 4y agoI'm not sure what you mean here, GP was saying dataclasses themselves don't prevent rogue assignments, but they have a nicer syntax for generating a `__slots__` automatically based on type-hints/class attributes which absolutely does.
- kzrdude 4y agoI see, new in Py 3.10 and new to me
- HelloNurse 4y agoHaving to specify the names and, if you hate yourself, the types of class members up front is a major limitation. Only when it is possible to do so (because the class is a boring one) the repetitiveness of typical __init__ methods can become an issue.
- eru 4y agoHow is that a major limitation? You only have to specify them 'up front' in terms of how the code is laid out. But in terms of which order you write the code in, you can add them as you need them. Specifying the names and types is extremely useful for readers (and users) of your code.
- HelloNurse 4y agoAnything requiring e.g. setattr, getattr, delattr? Without looking far, https://github.com/python-attrs/attrs/blob/main/src/attr/_make.py https://github.com/python-attrs/attrs/blob/main/src/attr/_ma...
- lysecret 4y agoPlease, I am very curious about the kind of "non boring" classes you are talking about. If you could show an example. I couldn't come up with anything that I would define as reasonable where we couldn't define the types beforehand.
- kortex 4y agoThunks-style functors, where you bind a callback to a private variable, to be later called, often with kwargs. Like what most HTTP frameworks do. You can use Callable but the actual function prototype is unknown.
- half0wl 4y agoORMs come to mind. There's typically a lot of reflection and dynamic instantiation magic baked into its classes, which you really have to hack around typing to make it work nicely (or even at all).
- qsort 4y agoKind of a radical idea but IMHO in a high-level language (i.e. not C++ or Rust) the real mistake is structs/classes with mutable fields. None of those problems exist in that world. It's not very ergonomic, but if you start from that assumption you could have ad-hoc syntactic support that desugars to a limited version of an optics library.
- adenozine 4y agoSo, like Erlang? What’s an example of a language that does this by default?
- eru 4y agoCommon Lisp Object System might do something like that?
- qsort 4y agoIdiomatic usages of ML-family languages (Haskell, OCaml) are very close to that.
- packetlost 4y agoRust
- adenozine 4y agoI'd rather gouge my eyes out than work with Rust code. Horrific.
- lysecret 4y agoWe always use dataclasses with frozen=True to make them immutable.
- half0wl 4y agoSame. I treat them as immutable DTOs [0], or "value objects" if you will instead of "just a plain class". An unfortunate drawback is that not everyone is familiar with the concept of immutability, especially in the world of Python. It can also be confusing to explain how "frozen dataclasses are not actually the Python classes you know" :-/ [0] https://en.wikipedia.org/wiki/Data_transfer_object https://en.wikipedia.org/wiki/Data_transfer_object
- dec0dedab0de 4y agoI still don't understand why you would use classes without methods. That's what dicts are for. About 10 years ago someone who I thought was pretty smart said it was because dots are nicer to type, fair enough I guess. It just always seemed silly to me to go through setting up a class but not getting the benefits of using methods.
- bl0rg 4y ago1. Dots are nicer to type for sure 2. It's a bit easier to define types for class attributes than dictionaries IMO 3. You can absolutely have methods on dataclasses
- Liquid_Fire 4y agoThere is clear value in being able to pass around several pieces of information about the same thing without having to individually name them every time (less typing, you can decide to include more members at a later time, less overhead in passing by reference, users can treat it as an opaque type without needing to know the contents, etc).
- thanatropism 4y agoThis. I look at old code I wrote and cringe at things like def foo(...): ... return yhat, sse, aic, bic yhat, _, _, _ = foo(ytrue) plot(ytrue, yhat)
- eru 4y agoDicts are much more annoying to give types for. They are also slower.
- st4lz 4y agoThere is a slight distinction between a dictionary and a class. Dict is a mutable key-value storage, so can be used as a collection of objects (or values) and benefit from its dynamism. Dataclasses are meant to define an object, so the emphasis is on clearly declaring the purpose of it upfront, enhancing readability.
- 4y ago
- kodablah 4y agoUsing dataclass for all basic state classes whose primary purpose is public state with sensible init makes sense. But for the many other purposes of classes that need custom init, advanced field defaults, no equality, custom repr/str, advanced inheritance, etc dataclasses give you nothing but a bunch of magic to turn off or work around. Whether you're using Kotlin data class, a C# record, a Scala case class, or whatever, we've seen in practice that these are only good for (often immutable) public state and data composition. Business objects with more logic and more than simple state/member manipulation are only hampered by dataclass field approaches. If Python embraced immutability and composition more at its core, then using dataclasses in these cases would make more sense.
- dec0dedab0de 4y agoThat was similar to my initial reaction, but after trying some things in another comment I'm kind of sold on dataclasses. As long as it raised a syntax error if a class combined __init__ with arg: type, then I think you can still do everything else you would in a normal class. It's just a shortcut for when that's all you need.
- zzzeek 4y ago> But for the many other purposes of classes that need custom init, write your own `__init__` method then, as you would anyway. Dataclass' default init is way better than that of `object`. > advanced field defaults same > no equality set eq=False, all dataclass methods are optional > custom repr/str write __repr__ and __str__ methods, same as init, dataclass' default is much nicer than that of `object` > advanced inheritance What's the issue with inheritance? what's "advanced" inheritance, you mean mixins? > etc dataclasses give you nothing but a bunch of magic to turn off or work around. Python `object` has all kinds of default behaviors too, which are just worse. default behaviors aren't "magic". dataclasses currently manufactures its methods at runtime, so that's maybe "magic", but all of that could be built into Python, then it would just be, "default behavior".
- PartiallyTyped 4y agoRe __init__ and advanced defaults, these may be achieved/set using __post_init__.
- lprthx 4y agoI'm always astonished by Python. For ages "dataclasses" have been called struct or record in other languages and are fast first class values. In Python, they invent a weird, slow concept and discuss it for a decade, probably while writing 10 PEPs in the process. Through conferences, Stackoverflow, etc. people are slowly convinced that the concept is natural.
- empthought 4y agoPython has had named tuples since 2008 and slots for attributes since 1999, both of which are fast first class values. I’m always astonished by ignorant people feigning knowledge of their areas of ignorance.
- mejutoco 4y agoIt is the same concept. In rust structs are defined with the struct keyword, and can have methods. And there is no class keyword for classes. In haskell, structs are defined with the keyword data. Maybe that is where the inspiration comes from. https://lotz84.github.io/haskellbyexample/ex/structs https://lotz84.github.io/haskellbyexample/ex/structs In any case, it seems each language chooses equally confusing keywords. Syntax is not a big deal IMO if the concept is clear.
- eru 4y ago> In haskell, structs are defined with the keyword data. Maybe that is where the inspiration comes from. In Haskell, Rust and Python you can also just have tuples, if you want to. In Haskell you can use `newtype` or `type` to name your tuples and use them as records. Of course, as you say, you can also use the `data` keyword in Haskell. Or you can use church encoding to use functions as records.. Lots of (confusing) possibilities.
- mejutoco 4y agoIndeed. My intention was answering OPs claim that python is "weird" because > "dataclasses" have been called struct or record in other languages
- dec0dedab0de 4y agoWould switching all classes to the data class style mean losing the ability to differentiate between class variables and object variables? I'm going to go try it in a minute, but I'm specifically thinking of setting a mutable variable to use as a way to cache the results of a long running process across objects. Or even just any variable to use with class methods. I'll update soon edit: I tried it out and it looks like it works just fine @dataclass class MyCoolDataClass: arg: any _things = {'last_updated':0, 'data':[]} _things_timeout = 300 def _update_things(self): """some long process""" time.sleep(10) self._things['last_updated'] = time.time() self._things['data'] = ['some things'] @property def things(self): seconds_since = time.time() - self._things.get('last_updated', 0) if seconds_since >= self._things_timeout: self._update_things() return self._things.get('data',[]) which then works as expected, sharing the cache between objects. start_time = time.time() for i in range(10): obj = MyCoolDataClass(i) print(obj.things, int(time.time() - start_time)) ['some things'] 10 ['some things'] 10 ... So yeah, I think I might be on board with dataclasses by default. As long as we can still add __init__ for when we want to do something extra with the variable edit2: If we switch to dataclasses things like this would need to raise a syntax error: @dataclass class MyOtherDataClass: arg: any def __init__(self, special_arg:str): self.special_arg = special_arg.lower().strip() As it stands now defining __init__ completely removes any definitions set in the class. I think the killer feature would be if we added django forms style validate methods for validation and normalization. imagine if we could do this: class Drink: temp: str container: str def _validate_temp(self, temp): temp = temp.strip().lower() if temp not in ['hot', 'cold', 'room']: raise ValidationError(f'"{temp}" is not a valid temperature') return temp def _validate_container(self, container): container = temp.strip().lower() if container not in ['mug', 'glass' 'sippy cup']: raise ValidationError(f'"{container} is not a valid container") return container def _validate(self, kwargs): if kwargs['temp'] == 'cold' and kwargs['container'] == 'mug': raise ValidationError("Don't dirty a mug for a cold drink!") Sure, it goes against explicit is better than implicit, but that could be fixed with decorators. It would also be nice to have a concise way to say what values are valid, and a handful of standard normalization processes, but I think you get the idea.
- varispeed 4y agoMaybe I didn't have enough coffee, but what's really the point of "data" keyword? Seems like its only goal is to confuse. So you write that data thingy and later figure out you actually need a class, then what? What's the difference between data and class? Because of what it keeps? So why not have "methods" for classes that only have methods or "notdecided" if you don't yet know what sort of class you are going to end up with. I don't follow.
- dec0dedab0de 4y agoThe idea is that dataclasses are for when you just want to store some data in a pre-structured format. It is almost the same thing as a named tuple. It trades away the flexibility of processing the incoming data in __init__ to make defining the class simpler. I agree that the name is confusing, I think maybe @autoinit would have been better.
- varispeed 4y agoWhy Python interpreter can't figure it out like "oh this class only stores data, so it must be a data class". No need for decorators etc. What would be the consequences of non-data class that only contains data to treat is as a data class? Likely none.
- dec0dedab0de 4y agoData classes are just regular classes but with some extra syntactic sugar(magic) to automate some common stuff.
- kortex 4y agoGood luck introspecting a given python object and ascertaining "this class only stores data". It's literally impossible given the dynamicism of python. The consequences are exposing state that you don't necessarily want to expose, missed opportunities for optimization (lose fine-grained memory/performance tradeoff ability), imply something is serializable when it isn't, collisions with metaclasses/inheritance weirdness.
- 4y ago
- leoff 4y agoFor my general uses, pydantic and dataclasses are very similar, and I prefer pydantic over dataclass for the following reasons: 1. pydantic by default ignores extra fields - it's useful when I make an API call and want to extract and validate only certain fields from the response, while dataclasses throws errors if you don't specify all fields, and this behavior can't be disabled 2. pydantic is more customizable - I can overwrite the BaseModel if I have a custom need 3. pydantic seems to integrate better with python typing - I'm not sure how to explain this one, but it feels more natural and dynamic I'm not sure what the performance concerns are, though
- eru 4y agoThere's also dataclassy, which stays closer to the standard library dataclasses while improving on them.
- lysecret 4y agoPydantic is much much much slower than normal dataclasses. However, they have to be because they do validation and dataclasses do not. So, I use normal dataclasses for internal constructs, which need to be fast and pydantic for anything that comes externally, can't be trusted and has to be validated.
- stevesimmons 4y agoPydantic models also have a model.construct(*src) form which skips validation. According to Pydantic's docs [1], this makes it 30x faster. [1] https://docs.pydantic.dev/usage/models/#creating-models-without-validation https://docs.pydantic.dev/usage/models/#creating-models-with...
- animuchan 4y agoWhat I seriously love about pydantic is the ability to just write the default value of `tags: list[str] = []` and not worry about all of the instances of the class sharing the same single instance of the default list.
- 4y ago
- iandanforth 4y agoTo answer the question of the author, yes compatibility is sufficient reason not to do this. Please don't push this idea. I also don't agree with much of the reasoning or conclusions but others have identified many of the reasons elsewhere so I won't repeat them.
- jammycrisp 4y agoIntegrating `dataclass` more into the language builtins might be nice, if only that it may allow/encourage a more native and performant implementation. Using a dataclass right now results in slightly slower class operations than handwritten types, and much slower import times. I maintain another dataclass-like library[1] that's written fully as a C extension. Moving this code to C means these types are typically 5-10x faster for common operations[2]. It'd be nice if the builtin dataclasses were equally performant. [1]: https://github.com/jcrist/msgspec https://github.com/jcrist/msgspec [2]: https://jcristharif.com/msgspec/benchmarks.html#benchmark-structs https://jcristharif.com/msgspec/benchmarks.html#benchmark-st...
- smitty1e 4y agoI think of dataclasses as glorified tabular data that is not of size to justify numpy. Once the need to tart up the class methods or do any inheritance crops up, make a "proper" class. None of that has an technical basis, but there is value in being methodical with the coding choices.
- gotofritz 4y agoThis would have been a good discussion 10 years ago, but these days it's pydantic first, hardly ever use dataclass
- goodoldneon 4y agoDataclasses are nice but they have a big downside: extending a dataclass with a default property can raise an error. If you run this code: import dataclasses @dataclasses.dataclass class Base: a: int b: int = 1 @dataclasses.dataclass class Child(Base): c: int Then you'll get this error: TypeError: non-default argument 'c' follows default argument
- zzzeek 4y agothat's because you are expecting dataclasses to create a default `__init__()` for you, which it can't do with that declaration. set init=False if you don't want it to do that. import dataclasses @dataclasses.dataclass class Base: a: int b: int = 1 @dataclasses.dataclass(init=False) class Child(Base): c: int if you want the init to be generated for you, you have to tell it how, given that "b" has a default value. so "c" needs to be keyword only here: import dataclasses @dataclasses.dataclass class Base: a: int b: int = 1 @dataclasses.dataclass class Child(Base): _: dataclasses.KW_ONLY c: int
- goodoldneon 4y agoLooks like that works but it’s unintuitive. Pydantic can handle this situation without the weird KW_ONLY thing, so dataclasses should too
- zzzeek 4y agohm it implicitly decides to be KW_ONLY? OK does not work: from pydantic import dataclasses @dataclasses.dataclass class Base: a: int b: int = 1 @dataclasses.dataclass class Child(Base): c: int error File "pydantic/dataclasses.py", line 265, in pydantic.dataclasses.dataclass ... File "/opt/python-3.10.0/lib/python3.10/dataclasses.py", line 539, in _init_fn raise TypeError(f'non-default argument {f.name!r} ' TypeError: non-default argument 'c' follows default argument it works with KW_ONLY though awkward that pydantic.dataclasses doesn't supply it directly, need to split up imports among pydantic.dataclasses and dataclasses.
- rtpg 4y agoI think the difficulty here is that data classes beyond the most simple cases have awkward extension points, see the docs on post-initialization https://docs.python.org/3/library/dataclasses.html#post-init-processing https://docs.python.org/3/library/dataclasses.html#post-init... I am partial to a dataclass keyword but… it’s tough. The problem with declarative stuff like this is that sometimes you really just want to do things in a certain order and now you have to communicate that (the keyword argument hack is an example of this)
- planede 4y agoIf you are defining dataclasses, do yourself a favor and define them kw_only=True.
- wirrbel 4y agoHave been using python since python 1.x, I find data classes kind of nice bringing in convenience at the same time it is a weird hybrid. I would have rather have some native struct and variant (think rust neun) types which would have required new syntax In a way it feels like data classes are a strange hybrid. Similarly I see all kinds of type theoretical issues with type checking popping up.
- kortex 4y agoI enjoy novel ideas, they are fun to entertain, but this is quite spicy of a take. I kind of understand where OP is coming from, but if you start to understand python internals, you would quickly see why it's completely untenable. data Point3D(x, y, z) from Vector: is never, ever, ever, ever gonna happen. It's ugly, alien to the rest of python style, will be a massive pain to implement in the lexer/parser, adds another keyword and mechanism, and most importantly, makes the word "data" unusable as a variable name. Dataclasses are "just" decorator functions (there is some special optimization and meta magic under the hood, but still just a callable). There's a glaring reason why we can't "just make all classes dataclasses" - it would break the entire data model of the language. "class" keyword is basically syntactic sugar for the "type" builtin. Everything (with a few small exceptions) you touch in python is a subclass of object. Changing the behavior of class to by default behave like dataclass would have knock-on effects across the entire language on every single python3 codebase on the planet. Ok, even assuming we could surmount the insurmountable breaking changes, would we even want default classes to be dataclasses? Not in the slightest. Python may not have true private variables, but libraries/APIs leverage dunders, properties, and hidden state to do all kinds of OOPy things. You can't do that with dataclasses. Many times you have objects bound to attributes which don't have concrete types, such as endpoints in HTTP frameworks and anything with dynamic dispatch (yeah there's Callable, but it is tricky to type check). Dataclass also has a self-documentation effect: it says "this thing is kind of like a record". Not a primitive, not a function, not a router, view, thread, tree node, god-class or stack frame. If I see dataclass, I almost always expect to be able to de/serialize it to some sort of message. Complex state, anything unpickleable, or exotic types don't really belong in a dataclass.
- kortex 4y agoOff the top of my head, python classes which shouldn't/can't be dataclasses: The entire modules typing, os, sys, struct, inspect, json. Anything which is a metaclass or relies on metaclass behavior (you have to be careful combining dataclass with other metaclasses, it's possible but with footguns akimbo). Datetime actually should be a dataclass, but that api is so crufty and unintuitive that it's never gonna happen, and trying to shoehorn it to a dataclass would break oodles of libraries and applications. threading: Thread, Lock, Rlock anything whichs uses locks Numpy.ndarry, pandas.Series, pandas.Dataframe, torch.Tensor. Numerous scikit-learn models. Probably the vast majority of c extensions. pydantic.BaseModel, fastapi.FastAPI, fastapi.APIRouter (same with flask/django), celery.Task. I could go on but I think I've made my point. There are plenty of non-dataclass use cases out there.
- __mharrison__ 4y agoAs a Python instructor, classes are a pain in general to teach. OO seems to have a non-understandable aura around it. Especially for folks without a CS background. self is weird until you explain that __init__ is not really a constructor. We get through it but there are bumps in the road. I don't teach dataclasses until I teach intermediate classes because I like my students to understand decorators beforehand. This make me wonder if I should experiment with teaching dataclasses instead of __init__ (and just hand-wave decorators and types)...
- sireat 4y agoI've been a Python instructor for 5+ years and I've never taught to my students that __init__ is not a constructor. That is I teach that __init__ works just like a constructor. Sure technically (the best type of correct) it is not a constructor. Where would this be important? Maybe it could be taught as part of "language lawyer" course. I teach OOP around lesson 10. Each lesson is about 2hours. Complete beginnners to programming are able create basic classes and even inheritance. Some people like OOP some stick with imperative style. Some migrate to more functional style (which I personally advocate). Going through this old thread: https://stackoverflow.com/questions/6578487/init-as-a-constructor https://stackoverflow.com/questions/6578487/init-as-a-constr... puts me firmly into "this is hair-spliting" camp.
- bmelton 4y ago> Where would this be important FAANG interviews? Knowing that __new__() comes before __init__() is probably useful, as the deeper one gets into a language the more important it is to understand the lifecycle of an object. Otherwise, the only thing I can think of that's special is understanding that and how it works as a singleton.
- gautamdivgi 4y agoI’ve been through a couple of those. I’ve never been asked language specific questions. Most FAANG code interviews don’t care much about which language you choose. I’ve also not seen them care if you use OO or don’t. Either way works - as long as your algorithm is good. In short this is irrelevant for those interviews. It’s definitely relevant if you do python as a day job.
- ehutch79 4y agoIf we get rid of normal classes, how do things like Django ORM work? It's a pretty significant package, if you're proposing this kind of thing, I assume there's an answer to this...
- seanhunter 4y agoThere's all kinds of use cases for classes without dataclass. For example, just the other day I wrote a leaky bucket rate limiter class. It uses the __aenter__ and __aexit__ methods to do things in async with blocks. I love dataclasses, but there needs to be a compelling reason to break everyone's code and the reason can't just be "I don't want one import and one decorator".
- glyph 4y agowhy do you think you can't define `__aenter__` and `__aexit__` on a dataclass?
- seanhunter 4y agoI don't. But it's really not a dataclass - it doesn't need slots etc.
- saiojd 4y agoProbably a controversial opinion, but I would like `self` removed, and have the class body be an init function. Something like: ``` dataclass NormalizedPoint(x: float, y: float): _norm = math.sqrt(x\*2 + y\*2) x /= _norm y /= _norm def __add__(other: NormalizedPoint): return NormalizedPoint(x + other.x, y + other.y) ``` In other words, I would really like some construct which sticks as close as possible to a normal function, but that also returns a (typed) object with everything in its scope.
- dragonwriter 4y ago> Probably a controversial opinion, but I would like `self` removed, and have the class body be an init function. That would be weird in classes that override the constructor (__new__), which is what is actually called (not __init__, except contingent on the result of the constructor) when the class is used as a callable. Not to mention the need to define methods in the class body, and be able to distinguish them from local functions in the constructor/initializer, which is a pretty good argument against the class body being the constructor or initializer.
- saiojd 4y agoI agree that in the general case it does not work, but if we were to introduce an alternate dataclass/class definition syntax, I feel like something lightweight like this would be nice. (Local functions could just be prefixed with underscores, no need to distinguish them from private methods.)
- adenozine 4y agoDoes Java still have a use for variables without final? Does Javascript still have a use for functions without async/await? This is just a repackaging of an age-old argument, imo. When they invented the screw, it did not preclude the nail. /shrug
- unoti 4y agoFor one category of classes-- representing data-- there's little point in non-dataclass classes. But for other kinds of classes, it's important to be able to control the constructor/dependencies. For example, suppose I have a provider class that abstracts the interface to some external system. My system needs to operate in at least 3 different contexts for this data provider: 1) connecting to an external database, 2) connecting to the local filesystem for simulations and experiments, and 3) unit tests. I have several implementations of this class using different mechanisms. Each of these take different things in their constructor: one takes a database connection as a dependency, another takes a directory name for the local filesystem, and another takes an in-memory dictionary or maybe doesn't need anything in its constructor. Having each of these classes have a clearly defined type-annotated constructor makes a lot of sense and makes things more self-documented. If everything were a dataclass then this situation would not be explicitly stated in the constructors.
- vbtemp 4y agoIt's just unfortunately becoming clear Python is going the way of C++.