4 ms·
Pandas & Numpy are extremely useful DSLs in Python. Being able to add vectorization to the language and use structures like DataFrames in an R-like way is very
by goatlover 9y ago
Pandas & Numpy are extremely useful DSLs in Python. Being able to add vectorization to the language and use structures like DataFrames in an R-like way is very powerful. There are some good use case scenarios.
Language features like operator overloading do exist for a reason, and that's because the language designer was aware of situations where you do want to use those features.
- makmanalp 9y agoDo pandas and numpy count as DSLs? This always confuses me. I think it's just a library, with the same language and the same semantics, but different data structures.
- dmix 9y agoThis is a good question, at what point does an extensive abstraction, a library on top of a language to extend the language to make it accessible for a specialized use-case, turn into a "DSL"? I also never saw Numpy/Pandas as a DSL but rather a extensive layer on top of Python whose complexity is largely a result of the usecase rather than being the result of attempting to be a full DSL on top of the language ala Matlab for Python. This is likely one of those scenarios where DSL's are one of many options available to particular languages to solve a particular problem set but are hard to identify in practice. Not to mention the many times it doesn't make sense to develop a full DSL layer but regardless the ease of creating them in some languages makes it a commonly abused trope (as many OO-related concepts are applied to everything where other old solutions are far superior). It's difficult to differentiate between the functional utility vs purely aesthetic optimizations of various abstractions, so I wouldn't be quick to blame negligence as much as communicating the best tools for the job on a language-by-language basis.
- makmanalp 9y agoI'd even go as far as arguing that the fact that pandas / numpy isn't a DSL causes some of its awkwardnesses, e.g. the fact that you have to use & for `and` in pandas, and the fact that you have to parenthesize expressions like `df[(df.a == 7) & (df.b == 2)]` instead of `df[df.a == 7 & df.b == 2]` or python's wonky operator precedence will try to execute `7 & b` first. Also we could even have special dataframe scoping rules like `df[a == 7 and b == 2]`, but we have to do `df.a` instead, exactly because pandas is NOT a DSL.
- goatlover 9y agoThat makes sense. I do find that stuff to be awkward and sometimes wish the syntax could be simpler.
- goatlover 9y agoYou can't do this in regular Python: Numpy array * 10 Pandas column A + column B Pandas Dataframe[ column C < 10 ] Numpy array 1 / array 2 where the second array has 0s and NaNs in it. Numpy has overridden division to allow division by 0 and NaN (Numpy added data type) in addition to vectorization. Moreover, you're encouraged to not iterate (generally a lot slower) if you can help it when using these libraries.
- kevin_thibedeau 9y agoSubclass array.array and specialize the operators as you desire. All in pure, albeit slower, Python. Numpy is just a library.
- gnaritas 9y agoEmbedded DSL's are just libraries, what makes something an embedded DSL is that it attempts to be a literate fluent configuration language in the host languages native syntax. If it doesn't use the host langauge's syntax, it's not an embedded DSL, it's an external one.
- kevin_thibedeau 9y agoNumpy doesn't introduce new syntax. Novel operator behavior does not a DSL make.
- goatlover 9y agoI don't think you can introduce new syntax in Python and have it run as part of the language, so magic methods, decorators and metaclasses are as good as it gets. You'd have to write a parser to handle new syntax, and that makes it external, right?
- klibertp 9y agoYou can also use MacroPy[1] and create embedded DSL with a macro system inspired by Scheme and Elixir. You don't need to write a parser, btw, because the stdlib provides one for you (in `ast` module). [1] https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy
- dozzie 9y agoNo, because they're valid Python. Similarly, what Ruby kids call "DSLs" don't count as DSLs, at best they can be called DSAPIs.
- syngrog66 9y agothey are not DSLs. they are literally Python and they are APIs.
- goatlover 9y agoAn internal DSL would have to be part of the native language. Either Python doesn't (directly) support this, or magic methods partially allow the creation of DSLs by extending the operators.
- carapace 9y agoI get what you're trying to say, I think, but you should use a different term. Pandas and NumPy aren't DSLs unless you interpret the 'L' to mean Library. It is unusual but perfectly cromulent in Python to overload the magic methods on a class to provide whatever semantics you like though operators. So to me it doesn't seem like a DSL. There was a recipe on ActiveState's site for essentially creating new operators by defining classes that overrode default operator semantics in "both directions" if you will. So you could write: foo <<my_op>> bar And my_op could do whatever it wanted to with foo and bar, by overloading the left- and right-shift magic methods in the my_op class. Neat, eh? (But still not a DSL! Heh.)
- dragonwriter 9y agoBy definition, internal (or embedded) DSLs (a term with well established use) are valid host-language code, relying on whatever host language features exist that allow defining code that reads fluently for the application domain. That is what distinguishes them from external DSLs.
- carapace 9y agoI didn't know that. Thank you.