5 ms·

This is a nice story, but the reality is that popular languages and libraries tend to expose API surfaces that are broad and shallow, not ones that are narrow a

This is a nice story, but the reality is that popular languages and libraries tend to expose API surfaces that are broad and shallow, not ones that are narrow and deep. For example, a Python string object has something like 40 methods. I counted about 85 instance methods on org.joda.time.DateTime from Java.[1] Turning to JavaScript, every method in Underscore.js lives in a single gigantic namespace.[2] Experience suggests two related factors that favor the proliferation of big, shallow APIs. Modern IDEs offer rich auto-completion with pop-up docs. This makes it much easier to look at the available list of fields and methods and choose the one that's appropriate. A second, related reason is that IDEs generally don't have tooling that is nearly as rich to help you identify the chain of calls you need to make to obtain an instance of some type that has the method you want. Do you get a Connection from a ConnectionContext, a ConnectionContextManager, or a ConnectionContextManagerCache? And how do you get one of those? Shallow APIs and big namespaces tend to favor easy lookup. Deep APIs where every type only has a handful of methods make you memorize them. [1] http://joda-time.sourceforge.net/apidocs/org/joda/time/DateTime.html http://joda-time.sourceforge.net/apidocs/org/joda/time/DateT... [2] https://underscorejs.org/ https://underscorejs.org/

waheoo 6y ago
Shallow or flattened structures are always what I default to. The example given does look better, but it's far more complicated to actually use. Focus on flat denormalized structures as your default, its simpler and more discoverable. At some point it does get unwieldy, at some point it is worth to find a seam and refactor out a chunk of logic. The difference is you're using the codes modularity needs to guide what gets defined where, not some arbitrary idea of what looks neat and organised, because what looks neat and organised is usually not very flexible.
- loopz 6y ago
  The simplest flexible grouping of data is just a struct. The default approach would be to append members to that struct, and it would in most cases remain backwards-compatible. It is when you see the need to structure the members, or subgroups of them are used in very different contexts, that one need to refactor. This is precisely the right time and not any sooner, because now you know why you now need to do all that extra analysis and work! With OO it gets a little bit more complicated. If you use OO without needing the features of OO, you could, and maybe should, simplify by just using the struct. If you need to tie action with data, you also know which use cases this will represent in what contexts, what to utilize the encapsulation or polymorphism for. Ie. using OO to adress higher level concerns than data and action alone. The added complexity may require its own additional meta-logic in order to do that, which for pure data would've been solved more directly, but without the higher-level interface. For DDD one could solve most of this on paper before any code is written, because knowing the domain, the design can be made more explicitly upfront rather than growing something "organically" while trying to weed the results later. The approach really depend on the problem space and experience of the developer. With experience, there'd always be more ideas for improvement, even if we go backwards sometimes.
- TheOtherHobbes 6y ago
  I suspect OP is confusing number of attributes with number of abstraction layers. Attributes are not a problem as long as they're listed with intent and not just thrown into a bucket, and there's some rationale for their existence. It's much harder to trace flow down and up through multiple abstraction and encapsulation layers. And it's harder still to remember the layers for different classes/applications. And it's even harder to do this when all the class names are related and hard to tell apart.
  - pbw 6y ago
    Hierarchy and abstractions have costs, no doubt. I didn't get into that in the article, but the article was only 1500 words. I can imagine a long "yes, but..." section that explores the risks and downsides of small objects. It's possible to write total crap software using only small objects for sure. I don't advocate doing that. That's where the Bonsai metaphor comes in, that's as much about shaping and pruning as growing. Uncontrolled growth of small objects has a name: cancer.
didibus 6y ago
I agree, one of OOP's problem is conflating code organization with data-structure and access scopes. Especially true when each class must be its own file as well. In FP, in contrast, you might have one namespace that contains your entire set of functions. Each function is still just its input to output. So when you read the function, in your mind, there is nothing else you need to think about, just the function that fits in only 7+-2 line of code most likely. Now, if you find that the functions need some logical grouping to help you find them, or if you need to split things in more than one file to share the file with other projects or for multiple people to more easily work on the code, you can do that as well, and move some functions out to other namespaces, but doing so only affects your import statements, not the structure of your data or anything else of that sort. Similarly, in FP, things are immutable for the most part, which means you don't have to think about all the other parts of the code that might modify the value of your variable, or list, or map, etc. Another issue with OOP is touched in the article as well, the OOP structure doesn't map with everything you'll need to work with. That's where the whole ORM issue comes in. For example, tables are really hard to represent as objects. Which in turn makes working with them and manipulating them more difficult than it needs to be.
- kaens 6y ago
  i have found larger real world fp codebases to have essentially the same organizational problems as OOP codebases, just expressed in different ways. most of the problem wrt code organization has to do with communicating meaning to humans, fp codebases written by people with a weak and evolving understanding of their problem domain are prone to the same sort of spaghettish expression of intent as OOP codebases in similar spaces, albeit often less verbosely by LOC. the assumption that all you have to think about is the implementation of the transformation of the function you are currently looking at's inputs to its outputs is often incorrect, or only true in small or idealized cases. this isn't a diss on FP styles as a whole, i prefer them, but it is not in my experience a great model for everything nor do i find it to significantly aid clarity of purpose in many cases. it is well suited to a lot of shapes of problems that currently pop up often, particularly stream transformation / aggregation types of things. edit: for clarity, the thing that happened in this industry where "OOP" became synonymous with "how you write code / how you model problems" was a terrible mistake and the influence of fp mindsets on the whole has been a positive, especially considering that context. anything that disconnects seriously thinking about how what you're writing is being understood by humans and ran by machines and how that intersects with your domain is imo a mistake, however this is a bit idealistic of me.
emptyparadise 6y ago
Maybe instead of computer- or human-oriented programming, we need more human-oriented tools. Let there be lots of autocomplete and documentation.
- wruza 6y ago
  Yes. Usually there is just a list of names when you initiate autocompletion. But exploration could be more easy if instead of that list (or maybe as a separate mode of a completion, like <Dot><F1>) a panel popped up with a well-drawn cheatlist from which you could navigate and read synopsises easily. Most IDEs still do not even draw a separator between superclass borders.
- pbw 6y ago
  I think tools are great. I don't get into it in the article, but with both tools and humans you can in fact makes sense of complicated structures even if there are dozens of attributes per object, but the thing is it's hard, and that work is throw-away. If you spend three weeks studying a complex piece of code until it makes sense to you, unless you modify the code with your newfound understanding, the next person has to climb up that same ladder. So even if the tools are great, I think in maybe cases it's go to push that understanding back into the artifact, so the artifact improves. A counter-example might be legacy software where it's just too risky to make anythings. Then maybe you just push your understanding into really good documentation, or a 2nd parallel implementation.
Kinrany 6y ago
Flat APIs are not affected by the 7+-2 limitation. They do not depend on each other, so we don't need to think about all of them at the same time.
- pbw 6y ago
  I agree if attributes are global variables to the structured program that is the object, then methods are just functions and they are far less "dangerous". If you have a modest number of globals, like 3, you can have hundreds of different functions without much fuss. The only issue with tons of functions is really documentation and discoverability, teaching people what all you can do with the object. It's common to have a nested tree of objects to control state and keep the implementation clean, but then expose a flat API on top of all that. While it's more code, sometimes it's the best of both worlds. In fact creating objects with many attributes is kind walking down the primrose path, you think you're making the API easier to use, but you are making the implementation into a quagmire that you many never pull out of.
tchaffee 6y ago
The article talks about object properties, not methods. The problem described is that each property is like a global variable that all methods share. So the string object having 40 methods is not a counter example. As far as I can tell from the docs, Python strings have exactly one property: the sequence of characters that make up the string. And the api surface isn't strictly shallow either. Strings rely heavily on the Format class, right? Happy to be corrected.
- pbw 6y ago
  I wrote the article and you nailed it. In another comment I mention NumPy's ndarray has 15 attributes but 50 methods. Imagine the reverse, an object with 50 attributes? Even if the attributes were just booleans, that object could be in one of 2^50 different states! That's trillions upon trillions of states. Yes if attributes are global variables for the structured program that is the object, then the complexity of that object is very sensitive to the number attributes, but adding methods is mostly harmless. New methods are just new useful things you can do with that object, adding more methods does not degrade the existing methods, except maybe as far as discoverability and documentation. An object with a 1000 methods would be a challenge to learn.
joepie91_ 6y ago
> Turning to JavaScript, every method in Underscore.js lives in a single gigantic namespace. This is really not a great example. Code written with heavy use of Underscore/Lodash is notoriously difficult to understand and maintain, because it is difficult to keep a mental model of all the pieces of the library and how they interact. People mainly use these libraries when they come from a language with an extensive standard library, and want something familiar in JS. Experienced JS developers virtually never use them, for the above reason - it's bad for maintainability. > This is a nice story, but the reality is that popular languages and libraries tend to expose API surfaces that are broad and shallow, not ones that are narrow and deep. The unspoken assumption you're making here is that "what is popular, must also be the optimal thing to do". There's absolute mountains of evidence (eg. the tech hype cycle and ingrained cultural problems around NIHing) that suggest that this really isn't true. Indeed, grabbing for broad and monolithic tools seems to be primarily a cultural thing, and people commonly do it because people commonly do it. It's ingrained culture. This is especially painfully obvious in JS-land, which has all the tooling that's necessary to work with smaller, easier-to-reason-about, modular tools - but where I'm still spending half of my days in #Node.js trying to explain to people why it's a bad idea to try and replicate the Python standard library in JS. Yet those who take the advice seriously, consistently report that it has improved their development and maintenance process. Likewise, dependencies in JS are a popular thing to complain about, but virtually all of the complaints are based on assumptions from how other languages work (where they would hold true), but that do not hold true for JS - and people rarely bother to try and learn more about how it actually works in JS. It's ingrained culture, not optimization. In short, I don't think there's any merit to the claim that, paraphrased, "if everyone is doing it this way, it must be the better way". The referenced article makes a pretty good rational case, and it's a little odd to dismiss it purely based on "that's not what software looks like". > Modern IDEs offer rich auto-completion with pop-up docs. This makes it much easier to look at the available list of fields and methods and choose the one that's appropriate. Except it doesn't, because autocompletion is based on words, not on concepts. You still need to know what something is called, at least approximately, to get anything useful out of autocompletion. Otherwise it's just a big list of words. The reduction of concepts to smaller-scoped ones is precisely what helps there! > Shallow APIs and big namespaces tend to favor easy lookup. Deep APIs where every type only has a handful of methods make you memorize them. No, not really - it's actually the other way around. Lookup is actually much easier in smaller-scoped APIs, because you can do a 'refined search' - instead of having to find-by-word the thing you're looking for in a huge list, you can start by selecting the appropriate API subset from a much smaller list, and then drill down into specific methods and such. Big-scope APIs, on the other hand, require memorization; if you don't remember what a certain concept is called, you have no hope of ever finding it, short of a detour through some StackOverflow posts.
pbw 6y ago
I love flat APIs with many methods or functions. My article only talks about objects and attributes not methods or APIs. Numpy's ndarray object has around 15 attributes but over 50 functions. Imagine the reverse, if it had 50 attributes? Shudder. There's a huge difference between attributes and methods. As the article says attributes are just global variables in the structured program that is the object. Every attribute that you add increases the number of "globals" visible to every method, present and future. Consider if all your attributes were just booleans. If you have 4 attributes your object can be in one of 16 states. That means every method in theory has to account for each of those 16 states. If instead your object has 10 boolean attributes the object can now be in one of 1024 states. Every method, present and future, needs to work correctly no matter which of the 1024 states the object is in. This is why Functional Programming usually prohibits mutable state. 1024 states is insane enough, but an object which bounces around to different points in that state space over time? That's the stuff of nightmares. Many (most?) big OO programs grow over time into that exact nightmare scenario. Many people hate OO precisely because they've spent years shoveling in the illegal sapphire mine that is large badly written OO codebases. The reason bad OO is so common is some codebases become economically valuable right around when they start falling apart due to poor design, but you can hire more and more people if the software is bringing in more and more money. Adding an attribute to an object increases the complexity of every method in the object, because as the article says an attribute is just a global variable to the object. But adding a method is fairly harmless. It does not increase the object's complexity very much at all. Flat APIs are often nicer than nested. In fact it's super common to have a deeply nested object tree but then expose a single flat API for external consumption. That's the best of both worlds.

waheoo 6y ago

Shallow or flattened structures are always what I default to. The example given does look better, but it's far more complicated to actually use. Focus on flat denormalized structures as your default, its simpler and more discoverable. At some point it does get unwieldy, at some point it is worth to find a seam and refactor out a chunk of logic. The difference is you're using the codes modularity needs to guide what gets defined where, not some arbitrary idea of what looks neat and organised, because what looks neat and organised is usually not very flexible.

loopz 6y ago
The simplest flexible grouping of data is just a struct. The default approach would be to append members to that struct, and it would in most cases remain backwards-compatible. It is when you see the need to structure the members, or subgroups of them are used in very different contexts, that one need to refactor. This is precisely the right time and not any sooner, because now you know why you now need to do all that extra analysis and work! With OO it gets a little bit more complicated. If you use OO without needing the features of OO, you could, and maybe should, simplify by just using the struct. If you need to tie action with data, you also know which use cases this will represent in what contexts, what to utilize the encapsulation or polymorphism for. Ie. using OO to adress higher level concerns than data and action alone. The added complexity may require its own additional meta-logic in order to do that, which for pure data would've been solved more directly, but without the higher-level interface. For DDD one could solve most of this on paper before any code is written, because knowing the domain, the design can be made more explicitly upfront rather than growing something "organically" while trying to weed the results later. The approach really depend on the problem space and experience of the developer. With experience, there'd always be more ideas for improvement, even if we go backwards sometimes.
TheOtherHobbes 6y ago
I suspect OP is confusing number of attributes with number of abstraction layers. Attributes are not a problem as long as they're listed with intent and not just thrown into a bucket, and there's some rationale for their existence. It's much harder to trace flow down and up through multiple abstraction and encapsulation layers. And it's harder still to remember the layers for different classes/applications. And it's even harder to do this when all the class names are related and hard to tell apart.
- pbw 6y ago
  Hierarchy and abstractions have costs, no doubt. I didn't get into that in the article, but the article was only 1500 words. I can imagine a long "yes, but..." section that explores the risks and downsides of small objects. It's possible to write total crap software using only small objects for sure. I don't advocate doing that. That's where the Bonsai metaphor comes in, that's as much about shaping and pruning as growing. Uncontrolled growth of small objects has a name: cancer.

didibus 6y ago

I agree, one of OOP's problem is conflating code organization with data-structure and access scopes. Especially true when each class must be its own file as well. In FP, in contrast, you might have one namespace that contains your entire set of functions. Each function is still just its input to output. So when you read the function, in your mind, there is nothing else you need to think about, just the function that fits in only 7+-2 line of code most likely. Now, if you find that the functions need some logical grouping to help you find them, or if you need to split things in more than one file to share the file with other projects or for multiple people to more easily work on the code, you can do that as well, and move some functions out to other namespaces, but doing so only affects your import statements, not the structure of your data or anything else of that sort. Similarly, in FP, things are immutable for the most part, which means you don't have to think about all the other parts of the code that might modify the value of your variable, or list, or map, etc. Another issue with OOP is touched in the article as well, the OOP structure doesn't map with everything you'll need to work with. That's where the whole ORM issue comes in. For example, tables are really hard to represent as objects. Which in turn makes working with them and manipulating them more difficult than it needs to be.

kaens 6y ago
i have found larger real world fp codebases to have essentially the same organizational problems as OOP codebases, just expressed in different ways. most of the problem wrt code organization has to do with communicating meaning to humans, fp codebases written by people with a weak and evolving understanding of their problem domain are prone to the same sort of spaghettish expression of intent as OOP codebases in similar spaces, albeit often less verbosely by LOC. the assumption that all you have to think about is the implementation of the transformation of the function you are currently looking at's inputs to its outputs is often incorrect, or only true in small or idealized cases. this isn't a diss on FP styles as a whole, i prefer them, but it is not in my experience a great model for everything nor do i find it to significantly aid clarity of purpose in many cases. it is well suited to a lot of shapes of problems that currently pop up often, particularly stream transformation / aggregation types of things. edit: for clarity, the thing that happened in this industry where "OOP" became synonymous with "how you write code / how you model problems" was a terrible mistake and the influence of fp mindsets on the whole has been a positive, especially considering that context. anything that disconnects seriously thinking about how what you're writing is being understood by humans and ran by machines and how that intersects with your domain is imo a mistake, however this is a bit idealistic of me.

emptyparadise 6y ago

Maybe instead of computer- or human-oriented programming, we need more human-oriented tools. Let there be lots of autocomplete and documentation.

wruza 6y ago
Yes. Usually there is just a list of names when you initiate autocompletion. But exploration could be more easy if instead of that list (or maybe as a separate mode of a completion, like <Dot><F1>) a panel popped up with a well-drawn cheatlist from which you could navigate and read synopsises easily. Most IDEs still do not even draw a separator between superclass borders.
pbw 6y ago
I think tools are great. I don't get into it in the article, but with both tools and humans you can in fact makes sense of complicated structures even if there are dozens of attributes per object, but the thing is it's hard, and that work is throw-away. If you spend three weeks studying a complex piece of code until it makes sense to you, unless you modify the code with your newfound understanding, the next person has to climb up that same ladder. So even if the tools are great, I think in maybe cases it's go to push that understanding back into the artifact, so the artifact improves. A counter-example might be legacy software where it's just too risky to make anythings. Then maybe you just push your understanding into really good documentation, or a 2nd parallel implementation.

Kinrany 6y ago

Flat APIs are not affected by the 7+-2 limitation. They do not depend on each other, so we don't need to think about all of them at the same time.

pbw 6y ago
I agree if attributes are global variables to the structured program that is the object, then methods are just functions and they are far less "dangerous". If you have a modest number of globals, like 3, you can have hundreds of different functions without much fuss. The only issue with tons of functions is really documentation and discoverability, teaching people what all you can do with the object. It's common to have a nested tree of objects to control state and keep the implementation clean, but then expose a flat API on top of all that. While it's more code, sometimes it's the best of both worlds. In fact creating objects with many attributes is kind walking down the primrose path, you think you're making the API easier to use, but you are making the implementation into a quagmire that you many never pull out of.

tchaffee 6y ago

The article talks about object properties, not methods. The problem described is that each property is like a global variable that all methods share. So the string object having 40 methods is not a counter example. As far as I can tell from the docs, Python strings have exactly one property: the sequence of characters that make up the string. And the api surface isn't strictly shallow either. Strings rely heavily on the Format class, right? Happy to be corrected.

pbw 6y ago
I wrote the article and you nailed it. In another comment I mention NumPy's ndarray has 15 attributes but 50 methods. Imagine the reverse, an object with 50 attributes? Even if the attributes were just booleans, that object could be in one of 2^50 different states! That's trillions upon trillions of states. Yes if attributes are global variables for the structured program that is the object, then the complexity of that object is very sensitive to the number attributes, but adding methods is mostly harmless. New methods are just new useful things you can do with that object, adding more methods does not degrade the existing methods, except maybe as far as discoverability and documentation. An object with a 1000 methods would be a challenge to learn.

joepie91_ 6y ago

> Turning to JavaScript, every method in Underscore.js lives in a single gigantic namespace. This is really not a great example. Code written with heavy use of Underscore/Lodash is notoriously difficult to understand and maintain, because it is difficult to keep a mental model of all the pieces of the library and how they interact. People mainly use these libraries when they come from a language with an extensive standard library, and want something familiar in JS. Experienced JS developers virtually never use them, for the above reason - it's bad for maintainability. > This is a nice story, but the reality is that popular languages and libraries tend to expose API surfaces that are broad and shallow, not ones that are narrow and deep. The unspoken assumption you're making here is that "what is popular, must also be the optimal thing to do". There's absolute mountains of evidence (eg. the tech hype cycle and ingrained cultural problems around NIHing) that suggest that this really isn't true. Indeed, grabbing for broad and monolithic tools seems to be primarily a cultural thing, and people commonly do it because people commonly do it. It's ingrained culture. This is especially painfully obvious in JS-land, which has all the tooling that's necessary to work with smaller, easier-to-reason-about, modular tools - but where I'm still spending half of my days in #Node.js trying to explain to people why it's a bad idea to try and replicate the Python standard library in JS. Yet those who take the advice seriously, consistently report that it has improved their development and maintenance process. Likewise, dependencies in JS are a popular thing to complain about, but virtually all of the complaints are based on assumptions from how other languages work (where they would hold true), but that do not hold true for JS - and people rarely bother to try and learn more about how it actually works in JS. It's ingrained culture, not optimization. In short, I don't think there's any merit to the claim that, paraphrased, "if everyone is doing it this way, it must be the better way". The referenced article makes a pretty good rational case, and it's a little odd to dismiss it purely based on "that's not what software looks like". > Modern IDEs offer rich auto-completion with pop-up docs. This makes it much easier to look at the available list of fields and methods and choose the one that's appropriate. Except it doesn't, because autocompletion is based on words, not on concepts. You still need to know what something is called, at least approximately, to get anything useful out of autocompletion. Otherwise it's just a big list of words. The reduction of concepts to smaller-scoped ones is precisely what helps there! > Shallow APIs and big namespaces tend to favor easy lookup. Deep APIs where every type only has a handful of methods make you memorize them. No, not really - it's actually the other way around. Lookup is actually much easier in smaller-scoped APIs, because you can do a 'refined search' - instead of having to find-by-word the thing you're looking for in a huge list, you can start by selecting the appropriate API subset from a much smaller list, and then drill down into specific methods and such. Big-scope APIs, on the other hand, require memorization; if you don't remember what a certain concept is called, you have no hope of ever finding it, short of a detour through some StackOverflow posts.

pbw 6y ago

I love flat APIs with many methods or functions. My article only talks about objects and attributes not methods or APIs. Numpy's ndarray object has around 15 attributes but over 50 functions. Imagine the reverse, if it had 50 attributes? Shudder. There's a huge difference between attributes and methods. As the article says attributes are just global variables in the structured program that is the object. Every attribute that you add increases the number of "globals" visible to every method, present and future. Consider if all your attributes were just booleans. If you have 4 attributes your object can be in one of 16 states. That means every method in theory has to account for each of those 16 states. If instead your object has 10 boolean attributes the object can now be in one of 1024 states. Every method, present and future, needs to work correctly no matter which of the 1024 states the object is in. This is why Functional Programming usually prohibits mutable state. 1024 states is insane enough, but an object which bounces around to different points in that state space over time? That's the stuff of nightmares. Many (most?) big OO programs grow over time into that exact nightmare scenario. Many people hate OO precisely because they've spent years shoveling in the illegal sapphire mine that is large badly written OO codebases. The reason bad OO is so common is some codebases become economically valuable right around when they start falling apart due to poor design, but you can hire more and more people if the software is bringing in more and more money. Adding an attribute to an object increases the complexity of every method in the object, because as the article says an attribute is just a global variable to the object. But adding a method is fairly harmless. It does not increase the object's complexity very much at all. Flat APIs are often nicer than nested. In fact it's super common to have a deeply nested object tree but then expose a single flat API for external consumption. That's the best of both worlds.