7 ms·

It still astounds me that we haven't solved reusable modules in 2016. Sure, we have libraries, apis, package managers etc. but every time I read a code base, t

by hbt 11y ago

It still astounds me that we haven't solved reusable modules in 2016. Sure, we have libraries, apis, package managers etc. but every time I read a code base, there is always a utility function reinventing the wheel. Someone wrote it because it is still difficult to discover modular code and reuse it easily. It's pretty nuts when you think about it. Imagine mechanical engineers having to recreate the same CAD file because they can't find a component with the same functionality (which does happen but for other reasons e.g intellectual property). But we don't have the same intellectual property challenges and yet the best tools we have to discover and reuse code are 90s search engines that treat code as raw text with zero contextual understanding and often outdated code snippets. PS: I love your project btw.

thoradam 11y ago
This is one of the best qualities of languages like Haskell, the type system promotes discovery and true reuse of code. There has even been research into specifying dependencies as 'something that implements this signature' rather than specific packages.
- harry8 11y ago
  Yeah haskell is amazing. I'm still really looking forward to the day when somebody writes something actually useful in haskell. (Oh but pandoc, xmonad, shellcheck... in how many years?) ;-)
Tau_Zero 11y ago
One big reason I've seen people "reinventing" small utilities and modules is to avoid external dependencies, both for licensing and management hassle.
- jandrese 11y ago
  This is a big one. External dependencies are a liability, you need to be getting enough value out of them to offset their cost. For small one-off functions it's often better to just rewrite the thing than to pull in a library and have to worry about versioning or the library being abandoned or the API changing or the package manager on your target system not having that library or only having an old version or one that was compiled weirdly and doesn't have the function you need. There can also be interface friction between your code and the library--different syntactic style, or you have to convert the arguments from your native form into something the library can understand and then convert the result back. Don't get me wrong, libraries are wonderful and useful and you should be using them where appropriate, but like everything they have tradeoffs. I've seen way too many projects that try to just tie every single library they can find together without having to write any code. I know these projects because they're a nightmare to install and keep working and I end up spending way too much time trying to hack around all of the code rot to get them running again.
  - rhizome 11y ago
    Libraryism is the sinister flipside to Not-Invented-Here Syndrome, the same kind of conceptual teeter-totter as arguments around thin vs. thick clients.
anexprogrammer 11y ago
It gets more daft when every time someone tries to solve the problem, they make it worse. See the 200 version of visual C++ restributable you end up installed. I'm tempted to say developers shouldn't have been able to target specific dll versions, but that would cause different problems.
- lmm 11y ago
  > See the 200 version of visual C++ restributable you end up installed. In the grand scale of things that are wrong with programming that's somewhere between non-problem and minor irritation. Disk space is cheap.
incepted 11y ago
But we have this in software. On the JVM, Maven repos are essentially an enormous repository of existing modules that you can pull in your project with a one line addition to your build file. This covers small utility libraries to manipulate strings or dates (like the one you allude to) to entire ecosystems like web containers and everything in-between. I don't understand what part of this doesn't qualify as a "reusable module" to you.
- hbt 11y ago
  npm and maven modules are not easily discoverable - How do you search for a function other than by description or name. Assuming the developer classified properly and you even happen to use the same domain language as the writer. - When multiple modules are returned, similar packages, how do you compare and select efficiently without wasting hours reading the code, evaluating it, checking if it is still maintained and hoping there are no hidden bugs etc. It is easy to share code. Package manager even facilitate updating code. But it is still a nightmare to find/discover code and reuse it It's like building a lego set. You know the building block you want but if you have to spend hours looking for it, you give up. I believe competent developers write shitty code not because they are lazy or inexperienced but because the cost of writing great code is very high.
  - Tyr42 11y ago
    Hoogle helps for Haskell projects, as you can search by Type, and the Types are expressive enough that that actually finds what you're looking for most of the time.
    - mintplant 11y ago
      You can also search by function and module name.
    - eru 11y ago
      Neil Mitchell, the author of hoogle, has some idiosyncratic ideas about what to search over in hoogle. If you want a more expansive search, give hayoo a try.
zingermc 11y ago
Haskell has Hoogle[1], which allows you to search for functions using a type signature. This is surprisingly effective. Let's say you want that `contains` function from the original post. You'd search for `(Eq a) => a -> [a] -> Bool`, which describes a function that takes two parameters and returns a boolean. The first result[2] is the `elem` function, which is exactly what we wanted! This is a bit of a contrived example, but I have honestly been surprised by how effective searching by type signature is in Haskell. I wonder if it is possible for a language with a weaker type system, like JavaScript. [1] https://www.haskell.org/hoogle/ https://www.haskell.org/hoogle/ [2] https://www.haskell.org/hoogle/?hoogle=%28Eq+a%29+%3D%3E+a+-%3E+%5Ba%5D+-%3E+Bool https://www.haskell.org/hoogle/?hoogle=%28Eq+a%29+%3D%3E+a+-...
- Fargren 11y ago
  Does the order of the parameters matter? It's been a while since I've touched Haskell, so maybe it's obvious that this isn't [a] -> a -> Bool for some idiomatic reason. I guess that would be `isContained` instead of `contains`, so it probably wouldn't be the first thing I search for, but there's at least some potential for ambiguity.
  - zingermc 11y ago
    Order doesn't matter! And it still returns similar results without the `Eq` typeclass constraint.
  - cormacrelf 11y ago
    It's almost a convention in Haskell. The idea is, since currying functions is easy and common, you stick the argument that you'd want the least in a curried version last. So if you wanted to find out if x was in ten lists, you'd just: map (elem x) [list1, list2, ..., list10] or, for folds, the function is the one least likely to change, so you put it first, and the list is most likely to change, so you put it last: foldl (+) 0 [1,2,3] map (foldl (-) 100) [[1,2],[2,3],[4,19]] Of course, it's always debatable if you can find some situation where you wanted to curry in a different order, but generally you pick in order to reduce forced named lambda parameters. As far as I can tell, that's how it's done. Edit: if you saw the sneaky edit, you'll know that this particular convention isn't easy to follow!
skybrian 11y ago
What makes you think that utility function you saw wasn't copied from somewhere?
- lmm 11y ago
  Maybe it was, but that's a problem if so - it's not going to benefit from updates, and it will have to be maintained on its own indefinitely.
  - dsp1234 11y ago
    but that's a problem if so It's not necessarily a problem. It actually cuts both ways. The code is not going to benefit from updates, but the code is not also going to be harmed by updates. Think API or behavior changes. Additionally, if the use case for a dependency changes, sometimes the "new" solution doesn't match well. Ex: I work with jscript in ASP sometimes. A javascript library changed such that instead of doing iterative processing of nested items (pushing an item on to an array, then looping, popping off the item and processing it, then repeating), they changed to use node's nexttick with some logic like 'we want to move to nested function calls for processing, and doing that iteratively would blow the stack, so we'll just use nexttick so it won't have an increasing stack'. Well, jscript in ASP doesn't have nexttick or any equivalent timer. So while the original code itself worked flawlessly, the use case of the authors moved and we pulled that code in as an external dependency. That's obviously an extreme case, but I can't really count the number of times that an API change in an NPM package for node has meant modifying code, without a change in functionality. So yes, you get updates, and sometimes those are going to be security updates and real bug fixes. Other times though that update is going be adding new functionality (and possibly new attack vectors), or dropping support for your use case, etc. Like many things in our field, it's wisest to look at the risks in all cases, evaluate them for the specific situation at hand, and then choose the appropriate one, rather than cargo-culting one 'best practice'.
    - dandelany 11y ago
      Sure, but if you use strict versioning in your package.json this isn't really an issue...
EGreg 11y ago
What are you talking about? Every framework has reusable modules. Perhaps you mean package management?
- hanniabu 11y ago
  I think that's part of the problem he's talking about. Why do new modules have to be made for a new framework? The framework should play nicely with standard modules written vanilla for that specific language. This way we're not staying over whenever a new framework is made.
  - EGreg 11y ago
    Well there is CommonJS, and AMD modules. Many modules implement that in JS.
    - coldtea 11y ago
      Those should delegate all the implementation to libs that work across languages (e.g. in C) and just provide a JS API for them. But they don't work that way, except for a very small number...
jerf 11y ago
We've come a lot farther than you may realize. We've come far enough that, having removed a lot of the accidental complexities of importing external modules, we've discovered collectively than the essential complexity is non-zero. Bringing a module into your project is non-trivially expensive. Bringing multiple modules into your project grows in expense super-linearly. (Not "exponentially", but definitely something greater than linearly.) There probably isn't a nirvana just waiting for That One Great Tool. Or, alternatively, if there is, it probably involves having to switch to something like a dependently-typed system or the very bleeding edge of where the Haskell community is right now, which is probably still not really quite where it needs to be yet for this. And that would still involve a lot of cost for that switch. Oh, and this is one of those places where I perhaps may be justified in pointing out again that for all the sound and fury in software development, in reality our field does not move that quickly. There's a ton of package managers out there, but broadly speaking, they're the same set of features shuffled into various combinations. Nothing wrong with that, per se. Just a lot less innovation than may initially meet the eye.
- coldtea 11y ago
  >We've come a lot farther than you may realize. We've come far enough that, having removed a lot of the accidental complexities of importing external modules, we've discovered collectively than the essential complexity is non-zero. Bringing a module into your project is non-trivially expensive. Bringing multiple modules into your project grows in expense super-linearly. (Not "exponentially", but definitely something greater than linearly.) That's because those modules also have multiple (and competing) dependencies though. If most modules in all languages depended on the same, lower level modules (with rigidly fixed APIs) that wouldn't the case.
lunchTime42 11y ago
Maybe should add that as a feature to a reshare IDE - if it identifys functional identical code to library functions- it offers a refactoring.
dsp1234 11y ago
but every time I read a code base, there is always a utility function reinventing the wheel. Sometimes the wheel is invented because the existing wheel isn't built for the same purpose (ex: off-road). ex: I recently needed to check a buffer to see if it's contents were valid UTF-8 in nodejs. I wanted to do this without having to run it through something like StringDecoder, because it's an intermediate buffer, and I didn't want to have the overhead of actually converting it since it wasn't for my use. So I do 'npm install is-utf8'. Then: let isUtf8 = require('is-utf8') isUtf8(new Buffer('\u0000')) //false isUtf8(new Buffer('\u000B')) //false isUtf8(new Buffer('foobar')) //true So the package is-utf8 is really more like "Is a printable UTF-8 string". Which is certainly a reasonable choice, and a possibly reasonable default, but was not what I needed.
coldtea 11y ago
True. Stuff like common functionality (regex engine, math functions, talk to postgres, etc) should be written once, and consumed with higher level interfaces from all other languages. The problem is we don't have a safe, easy to build and package with everything else, and easy to consume (without performance penalty) language. The best we have is C, which is not enough.

by hbt 11y ago

thoradam 11y ago

This is one of the best qualities of languages like Haskell, the type system promotes discovery and true reuse of code. There has even been research into specifying dependencies as 'something that implements this signature' rather than specific packages.

harry8 11y ago
Yeah haskell is amazing. I'm still really looking forward to the day when somebody writes something actually useful in haskell. (Oh but pandoc, xmonad, shellcheck... in how many years?) ;-)

Tau_Zero 11y ago

One big reason I've seen people "reinventing" small utilities and modules is to avoid external dependencies, both for licensing and management hassle.

jandrese 11y ago
This is a big one. External dependencies are a liability, you need to be getting enough value out of them to offset their cost. For small one-off functions it's often better to just rewrite the thing than to pull in a library and have to worry about versioning or the library being abandoned or the API changing or the package manager on your target system not having that library or only having an old version or one that was compiled weirdly and doesn't have the function you need. There can also be interface friction between your code and the library--different syntactic style, or you have to convert the arguments from your native form into something the library can understand and then convert the result back. Don't get me wrong, libraries are wonderful and useful and you should be using them where appropriate, but like everything they have tradeoffs. I've seen way too many projects that try to just tie every single library they can find together without having to write any code. I know these projects because they're a nightmare to install and keep working and I end up spending way too much time trying to hack around all of the code rot to get them running again.
- rhizome 11y ago
  Libraryism is the sinister flipside to Not-Invented-Here Syndrome, the same kind of conceptual teeter-totter as arguments around thin vs. thick clients.

anexprogrammer 11y ago

It gets more daft when every time someone tries to solve the problem, they make it worse. See the 200 version of visual C++ restributable you end up installed. I'm tempted to say developers shouldn't have been able to target specific dll versions, but that would cause different problems.

lmm 11y ago
> See the 200 version of visual C++ restributable you end up installed. In the grand scale of things that are wrong with programming that's somewhere between non-problem and minor irritation. Disk space is cheap.

incepted 11y ago

But we have this in software. On the JVM, Maven repos are essentially an enormous repository of existing modules that you can pull in your project with a one line addition to your build file. This covers small utility libraries to manipulate strings or dates (like the one you allude to) to entire ecosystems like web containers and everything in-between. I don't understand what part of this doesn't qualify as a "reusable module" to you.

hbt 11y ago
npm and maven modules are not easily discoverable - How do you search for a function other than by description or name. Assuming the developer classified properly and you even happen to use the same domain language as the writer. - When multiple modules are returned, similar packages, how do you compare and select efficiently without wasting hours reading the code, evaluating it, checking if it is still maintained and hoping there are no hidden bugs etc. It is easy to share code. Package manager even facilitate updating code. But it is still a nightmare to find/discover code and reuse it It's like building a lego set. You know the building block you want but if you have to spend hours looking for it, you give up. I believe competent developers write shitty code not because they are lazy or inexperienced but because the cost of writing great code is very high.
- Tyr42 11y ago
  Hoogle helps for Haskell projects, as you can search by Type, and the Types are expressive enough that that actually finds what you're looking for most of the time.
  - mintplant 11y ago
    You can also search by function and module name.
  - eru 11y ago
    Neil Mitchell, the author of hoogle, has some idiosyncratic ideas about what to search over in hoogle. If you want a more expansive search, give hayoo a try.

zingermc 11y ago

Haskell has Hoogle[1], which allows you to search for functions using a type signature. This is surprisingly effective. Let's say you want that `contains` function from the original post. You'd search for `(Eq a) => a -> [a] -> Bool`, which describes a function that takes two parameters and returns a boolean. The first result[2] is the `elem` function, which is exactly what we wanted! This is a bit of a contrived example, but I have honestly been surprised by how effective searching by type signature is in Haskell. I wonder if it is possible for a language with a weaker type system, like JavaScript. [1] https://www.haskell.org/hoogle/ https://www.haskell.org/hoogle/ [2] https://www.haskell.org/hoogle/?hoogle=%28Eq+a%29+%3D%3E+a+-%3E+%5Ba%5D+-%3E+Bool https://www.haskell.org/hoogle/?hoogle=%28Eq+a%29+%3D%3E+a+-...

Fargren 11y ago
Does the order of the parameters matter? It's been a while since I've touched Haskell, so maybe it's obvious that this isn't [a] -> a -> Bool for some idiomatic reason. I guess that would be `isContained` instead of `contains`, so it probably wouldn't be the first thing I search for, but there's at least some potential for ambiguity.
- zingermc 11y ago
  Order doesn't matter! And it still returns similar results without the `Eq` typeclass constraint.
- cormacrelf 11y ago
  It's almost a convention in Haskell. The idea is, since currying functions is easy and common, you stick the argument that you'd want the least in a curried version last. So if you wanted to find out if x was in ten lists, you'd just: map (elem x) [list1, list2, ..., list10] or, for folds, the function is the one least likely to change, so you put it first, and the list is most likely to change, so you put it last: foldl (+) 0 [1,2,3] map (foldl (-) 100) [[1,2],[2,3],[4,19]] Of course, it's always debatable if you can find some situation where you wanted to curry in a different order, but generally you pick in order to reduce forced named lambda parameters. As far as I can tell, that's how it's done. Edit: if you saw the sneaky edit, you'll know that this particular convention isn't easy to follow!

skybrian 11y ago

What makes you think that utility function you saw wasn't copied from somewhere?

lmm 11y ago
Maybe it was, but that's a problem if so - it's not going to benefit from updates, and it will have to be maintained on its own indefinitely.
- dsp1234 11y ago
  but that's a problem if so It's not necessarily a problem. It actually cuts both ways. The code is not going to benefit from updates, but the code is not also going to be harmed by updates. Think API or behavior changes. Additionally, if the use case for a dependency changes, sometimes the "new" solution doesn't match well. Ex: I work with jscript in ASP sometimes. A javascript library changed such that instead of doing iterative processing of nested items (pushing an item on to an array, then looping, popping off the item and processing it, then repeating), they changed to use node's nexttick with some logic like 'we want to move to nested function calls for processing, and doing that iteratively would blow the stack, so we'll just use nexttick so it won't have an increasing stack'. Well, jscript in ASP doesn't have nexttick or any equivalent timer. So while the original code itself worked flawlessly, the use case of the authors moved and we pulled that code in as an external dependency. That's obviously an extreme case, but I can't really count the number of times that an API change in an NPM package for node has meant modifying code, without a change in functionality. So yes, you get updates, and sometimes those are going to be security updates and real bug fixes. Other times though that update is going be adding new functionality (and possibly new attack vectors), or dropping support for your use case, etc. Like many things in our field, it's wisest to look at the risks in all cases, evaluate them for the specific situation at hand, and then choose the appropriate one, rather than cargo-culting one 'best practice'.
  - dandelany 11y ago
    Sure, but if you use strict versioning in your package.json this isn't really an issue...

EGreg 11y ago

What are you talking about? Every framework has reusable modules. Perhaps you mean package management?

hanniabu 11y ago
I think that's part of the problem he's talking about. Why do new modules have to be made for a new framework? The framework should play nicely with standard modules written vanilla for that specific language. This way we're not staying over whenever a new framework is made.
- EGreg 11y ago
  Well there is CommonJS, and AMD modules. Many modules implement that in JS.
  - coldtea 11y ago
    Those should delegate all the implementation to libs that work across languages (e.g. in C) and just provide a JS API for them. But they don't work that way, except for a very small number...

jerf 11y ago

We've come a lot farther than you may realize. We've come far enough that, having removed a lot of the accidental complexities of importing external modules, we've discovered collectively than the essential complexity is non-zero. Bringing a module into your project is non-trivially expensive. Bringing multiple modules into your project grows in expense super-linearly. (Not "exponentially", but definitely something greater than linearly.) There probably isn't a nirvana just waiting for That One Great Tool. Or, alternatively, if there is, it probably involves having to switch to something like a dependently-typed system or the very bleeding edge of where the Haskell community is right now, which is probably still not really quite where it needs to be yet for this. And that would still involve a lot of cost for that switch. Oh, and this is one of those places where I perhaps may be justified in pointing out again that for all the sound and fury in software development, in reality our field does not move that quickly. There's a ton of package managers out there, but broadly speaking, they're the same set of features shuffled into various combinations. Nothing wrong with that, per se. Just a lot less innovation than may initially meet the eye.

coldtea 11y ago
>We've come a lot farther than you may realize. We've come far enough that, having removed a lot of the accidental complexities of importing external modules, we've discovered collectively than the essential complexity is non-zero. Bringing a module into your project is non-trivially expensive. Bringing multiple modules into your project grows in expense super-linearly. (Not "exponentially", but definitely something greater than linearly.) That's because those modules also have multiple (and competing) dependencies though. If most modules in all languages depended on the same, lower level modules (with rigidly fixed APIs) that wouldn't the case.

lunchTime42 11y ago

Maybe should add that as a feature to a reshare IDE - if it identifys functional identical code to library functions- it offers a refactoring.

dsp1234 11y ago

but every time I read a code base, there is always a utility function reinventing the wheel. Sometimes the wheel is invented because the existing wheel isn't built for the same purpose (ex: off-road). ex: I recently needed to check a buffer to see if it's contents were valid UTF-8 in nodejs. I wanted to do this without having to run it through something like StringDecoder, because it's an intermediate buffer, and I didn't want to have the overhead of actually converting it since it wasn't for my use. So I do 'npm install is-utf8'. Then: let isUtf8 = require('is-utf8') isUtf8(new Buffer('\u0000')) //false isUtf8(new Buffer('\u000B')) //false isUtf8(new Buffer('foobar')) //true So the package is-utf8 is really more like "Is a printable UTF-8 string". Which is certainly a reasonable choice, and a possibly reasonable default, but was not what I needed.

coldtea 11y ago

True. Stuff like common functionality (regex engine, math functions, talk to postgres, etc) should be written once, and consumed with higher level interfaces from all other languages. The problem is we don't have a safe, easy to build and package with everything else, and easy to consume (without performance penalty) language. The best we have is C, which is not enough.