4 ms·

> it just doesn't work at an enterprise level Elaborate?

by bpicolo 11y ago

yenda 11y ago
Judging by his comment I have the feeling that entreprise level means huge projects with thousands of lines of bloat code and comments which need vast amount of underpaid programmers to analyse and debug just to add minor features.
- grayclhn 11y ago
  Why do you assume they're underpaid? :)
daxfohl 11y ago
Big, not-very-interesting CRUD app. Every time the data model changes, you've got to go through manually and find all the places the code needs changed, and hope you didn't miss anything. It probably didn't help that it was running on MongoDB, so two levels of dynamic typing and impossible to remember what went where, and a DB that takes whatever you give without complaining. So no compile errors and often no runtime errors either, just incorrect behavior. Switched to Scala and SQL, and code is almost as compact but typed relations are far easier to reason about; if a relation changes then you change the core type and follow the compile errors. Once it compiles, it usually works. We've not missed Clojure in the least. Scala (and even Java) simply feels more structured, with relations between types, the resulting app more "designed", which seems necessary at the enterprise level. In my 2 years, Clojure constantly felt like just hacking. Would love to see some examples of Clojure being used well at an enterprise level so I could understand better how to do so, but CircleCI's recent flopping between core.typed and prismatic just makes me think it's a square peg in a round hole.
- RussianCow 11y ago
  > Every time the data model changes, you've got to go through manually and find all the places the code needs changed, and hope you didn't miss anything. Isn't that what tests are for?
  - daxfohl 11y ago
    No, that's what types are for. Types are exactly for this. They're far less verbose than tests, they prevent compilation completely on error, they're integrated with your code, they're 100% comprehensive, and they don't contain bugs. Contrarily, tests don't work at all for model changes because the test still retains the old model anyway.
    - catnaroek 11y ago
      I agree with your sentiment - why bother testing something that we can get the compiler to prove, once and for all? However, in Java, a typeful model of a program's specification is bound to be: (0) Imprecise, because of the holes in Java's static safety. You can establish a convention that statically unsafe features are verboten, but it will remain just that - a convention. (1) Verbose, because Java can't express even the simplest ideas without using at least 5 classes: Finite case analysis? Visitors. Higher-order functions? Strategies. Persistent data structures? Memento. Things don't get better if we move to a higher architectural level: Embedded DSLs? You need a separate class for each production rule in your abstract syntax. Type-safe database queries? Java has no structurally typed records, so you need to make a custom POJO class for every query that returns different columns. These things are incredibly unpleasant to deal with, and while it might be true that "unpleasant work builds character", I don't think it should come as a surprise that some people say "enough is enough", and seek alternatives that minimize or at least reduce their pain (like dynamically checked conformance to a schema), even at the cost of sacrificing some type safety (dynamic checks can't enforce higher-order invariants).
      - paulddraper 11y ago
        I agree with you that Java is painful. But if "enough is enough", abandoning all types is throwing out the baby with bathwater. You make a compelling argument for languages with type inference and richer typing systems. There are many such languages. For the JVM, there is Scala.
        catnaroek 11y ago
- deleted 11y ago
  [deleted]
- dmichulke 11y ago
  I'm using postgres and clojure for quite a few enterprises and my typical approach is to keep the raw sql files somewhere (resources/sql/) so that I can execute it via CLI (say psql) as well as from within clojure (by parsing the files and putting them in a JDBC transaction) From this I derive both the database schema itself as well as implementation details (e.g., check what columns are in a table or see if the current DB schema agrees with the current SQL schema in resources/sql) For program internal representation I have my own "classes" (hash maps with a few pre/post assertions on key points) and I just use a single db.clj file that needs to be adapted when the SQL representation changes. I also worked a lot with Java and Hibernate and it just feels verbose to change a simple datatype definition in both the application and the db. In a way, with clojure I get DB / App consistency that I wouldn't get with Java and/or types. I haven't had the chance to use scala, however. I also worked with MongoDB (when I started with clojure) and I agree, there it is very easy to make mistakes but maybe I would do things now differently if I used it again.
  - daxfohl 11y ago
    Yeah I'm using raw JDBC for the same reason; I like knowing the exact SQL that's going to be used and it's nice to be able to freely use non-SQL-standard things that your particular database supports. It's trivial to write a custom wrapper to make things easier, and simple case classes in Scala to represent the DB records make for a single line code change when the DB model changes.
- michel-slm 11y ago
  Using Mongo certainly didn't help, we're in the same situation ourselves and having various regrets over different aspects of it... What do you use to interface with SQL on the Scala side? Squeryl?
  - daxfohl 11y ago
    JDBC. I tried a few libraries and found they all provide either too much or too little. It's not so hard to write your own wrapper around it to make JDBC more friendly. Plus then you get to use features specific to your DB rather than just the lowest common denominator. Another thing that helped was to use underscore_naming for any case-class in Scala that maps directly to a query, so it's easier to identify them.
  - daxfohl 11y ago
    Just out of curiosity, have you run into the "how did (our / users') (password / private key) get into the ABC data record that gets returned on XYZ API call?" regret (even though it's not displayed by the GUI, still it's transmitted)? I'm thinking it's a total potential for exploit on web apps backed by mongodb and any dynamic language.
- tahssa 11y ago
  Well, in lisp languages you just write an abstraction layer to account for such problem areas. It's really no different than the OO argument. If one wants OO functionality in lisp it can be implemented with the introduction of a few functions. But by using an OO only language you have just forced that abstraction to the surface leaving you with only one way to write your code even when it's not suitable for the problem space. So the same could be said for Types. Without Type Inference you're locking yourself in to model/framework that has a list of disadvantages too. I think the real problem is when people don't recognize they need to account for those problem areas, and they don't take the time to write the abstraction layer when needed and naturally they end up with bad results and feel like it's just hacking, because, well, they are.
  - catnaroek 11y ago
    Implementing statically checked abstractions in Lisp or Clojure using `defmacro` is a royal pain in the rear hole. Error messages from macros being used wrong range from "confusing" at best, to "WTF is this %#$&%#$&?" at worst. Racket supports your argument much better. And Racket is not Lisp: https://news.ycombinator.com/item?id=10763584 https://news.ycombinator.com/item?id=10763584
    - tahssa 11y ago
      I wouldn't go that far. The problem outlined was in dealing with data model changes. So the problem as I see is in managing the alignment of parameter types used in code with database field types. For that I've written a schema-template abstraction that manages that relationship. This template engine is about 100 lines of code and for every point the code intersect to/from the data it goes through the engine, looks up the a template and uses that for access/processing (a hook if you will). This allows automation for finding all the code entry spots. Bottom line is I don't hit these issues because I identified the problem and wrote an abstraction layer to handle it and for the rest of my code I can happily rely on type inference.
      - catnaroek 11y ago
        What exactly are you calling "type inference"? It cannot be ML-like type reconstruction, because doing that requires either whole program analysis or a prohibition from interacting with untyped code.

> it just doesn't work at an enterprise level Elaborate?

by bpicolo 11y ago

> it just doesn't work at an enterprise level Elaborate?

yenda 11y ago

Judging by his comment I have the feeling that entreprise level means huge projects with thousands of lines of bloat code and comments which need vast amount of underpaid programmers to analyse and debug just to add minor features.

grayclhn 11y ago
Why do you assume they're underpaid? :)

daxfohl 11y ago

Big, not-very-interesting CRUD app. Every time the data model changes, you've got to go through manually and find all the places the code needs changed, and hope you didn't miss anything. It probably didn't help that it was running on MongoDB, so two levels of dynamic typing and impossible to remember what went where, and a DB that takes whatever you give without complaining. So no compile errors and often no runtime errors either, just incorrect behavior. Switched to Scala and SQL, and code is almost as compact but typed relations are far easier to reason about; if a relation changes then you change the core type and follow the compile errors. Once it compiles, it usually works. We've not missed Clojure in the least. Scala (and even Java) simply feels more structured, with relations between types, the resulting app more "designed", which seems necessary at the enterprise level. In my 2 years, Clojure constantly felt like just hacking. Would love to see some examples of Clojure being used well at an enterprise level so I could understand better how to do so, but CircleCI's recent flopping between core.typed and prismatic just makes me think it's a square peg in a round hole.

RussianCow 11y ago
> Every time the data model changes, you've got to go through manually and find all the places the code needs changed, and hope you didn't miss anything. Isn't that what tests are for?
- daxfohl 11y ago
  No, that's what types are for. Types are exactly for this. They're far less verbose than tests, they prevent compilation completely on error, they're integrated with your code, they're 100% comprehensive, and they don't contain bugs. Contrarily, tests don't work at all for model changes because the test still retains the old model anyway.
  - catnaroek 11y ago
    I agree with your sentiment - why bother testing something that we can get the compiler to prove, once and for all? However, in Java, a typeful model of a program's specification is bound to be: (0) Imprecise, because of the holes in Java's static safety. You can establish a convention that statically unsafe features are verboten, but it will remain just that - a convention. (1) Verbose, because Java can't express even the simplest ideas without using at least 5 classes: Finite case analysis? Visitors. Higher-order functions? Strategies. Persistent data structures? Memento. Things don't get better if we move to a higher architectural level: Embedded DSLs? You need a separate class for each production rule in your abstract syntax. Type-safe database queries? Java has no structurally typed records, so you need to make a custom POJO class for every query that returns different columns. These things are incredibly unpleasant to deal with, and while it might be true that "unpleasant work builds character", I don't think it should come as a surprise that some people say "enough is enough", and seek alternatives that minimize or at least reduce their pain (like dynamically checked conformance to a schema), even at the cost of sacrificing some type safety (dynamic checks can't enforce higher-order invariants).
    - paulddraper 11y ago
      I agree with you that Java is painful. But if "enough is enough", abandoning all types is throwing out the baby with bathwater. You make a compelling argument for languages with type inference and richer typing systems. There are many such languages. For the JVM, there is Scala.
      - catnaroek 11y ago
deleted 11y ago
[deleted]
dmichulke 11y ago
I'm using postgres and clojure for quite a few enterprises and my typical approach is to keep the raw sql files somewhere (resources/sql/) so that I can execute it via CLI (say psql) as well as from within clojure (by parsing the files and putting them in a JDBC transaction) From this I derive both the database schema itself as well as implementation details (e.g., check what columns are in a table or see if the current DB schema agrees with the current SQL schema in resources/sql) For program internal representation I have my own "classes" (hash maps with a few pre/post assertions on key points) and I just use a single db.clj file that needs to be adapted when the SQL representation changes. I also worked a lot with Java and Hibernate and it just feels verbose to change a simple datatype definition in both the application and the db. In a way, with clojure I get DB / App consistency that I wouldn't get with Java and/or types. I haven't had the chance to use scala, however. I also worked with MongoDB (when I started with clojure) and I agree, there it is very easy to make mistakes but maybe I would do things now differently if I used it again.
- daxfohl 11y ago
  Yeah I'm using raw JDBC for the same reason; I like knowing the exact SQL that's going to be used and it's nice to be able to freely use non-SQL-standard things that your particular database supports. It's trivial to write a custom wrapper to make things easier, and simple case classes in Scala to represent the DB records make for a single line code change when the DB model changes.
michel-slm 11y ago
Using Mongo certainly didn't help, we're in the same situation ourselves and having various regrets over different aspects of it... What do you use to interface with SQL on the Scala side? Squeryl?
- daxfohl 11y ago
  JDBC. I tried a few libraries and found they all provide either too much or too little. It's not so hard to write your own wrapper around it to make JDBC more friendly. Plus then you get to use features specific to your DB rather than just the lowest common denominator. Another thing that helped was to use underscore_naming for any case-class in Scala that maps directly to a query, so it's easier to identify them.
- daxfohl 11y ago
  Just out of curiosity, have you run into the "how did (our / users') (password / private key) get into the ABC data record that gets returned on XYZ API call?" regret (even though it's not displayed by the GUI, still it's transmitted)? I'm thinking it's a total potential for exploit on web apps backed by mongodb and any dynamic language.
tahssa 11y ago
Well, in lisp languages you just write an abstraction layer to account for such problem areas. It's really no different than the OO argument. If one wants OO functionality in lisp it can be implemented with the introduction of a few functions. But by using an OO only language you have just forced that abstraction to the surface leaving you with only one way to write your code even when it's not suitable for the problem space. So the same could be said for Types. Without Type Inference you're locking yourself in to model/framework that has a list of disadvantages too. I think the real problem is when people don't recognize they need to account for those problem areas, and they don't take the time to write the abstraction layer when needed and naturally they end up with bad results and feel like it's just hacking, because, well, they are.
- catnaroek 11y ago
  Implementing statically checked abstractions in Lisp or Clojure using `defmacro` is a royal pain in the rear hole. Error messages from macros being used wrong range from "confusing" at best, to "WTF is this %#$&%#$&?" at worst. Racket supports your argument much better. And Racket is not Lisp: https://news.ycombinator.com/item?id=10763584 https://news.ycombinator.com/item?id=10763584
  - tahssa 11y ago
    I wouldn't go that far. The problem outlined was in dealing with data model changes. So the problem as I see is in managing the alignment of parameter types used in code with database field types. For that I've written a schema-template abstraction that manages that relationship. This template engine is about 100 lines of code and for every point the code intersect to/from the data it goes through the engine, looks up the a template and uses that for access/processing (a hook if you will). This allows automation for finding all the code entry spots. Bottom line is I don't hit these issues because I identified the problem and wrote an abstraction layer to handle it and for the rest of my code I can happily rely on type inference.
    - catnaroek 11y ago
      What exactly are you calling "type inference"? It cannot be ML-like type reconstruction, because doing that requires either whole program analysis or a prohibition from interacting with untyped code.