9 ms·
I feel like this is something I have to keep repeating, "the data always outlives the app code." If app code is required to make sense of the data, you are goin
by matwood 10y ago
I feel like this is something I have to keep repeating, "the data always outlives the app code." If app code is required to make sense of the data, you are going to have problems. Of course NoSQL has a place, but only in very certain cases should it be your primary datastore. SQL (and the RDBMSs that leverage it) are built to store, protect, and provide structure to the data.
The second annoyance I have is the push for schemaless. Schemaless does not exist. There is always a schema, except in a schemaless data store, the schema has been moved to app code and then my earlier comment applies.
One thing I do agree with is that most ORMs are not very good. The best ones I have used are very thin layers over the sql, like jOOQ. I think the lack of understanding of SQL and bad ORM experiences (Hibernate WTF) is what led people to think SQL/RDMBs were the problem when in fact they were not.
- fnord123 10y ago>The second annoyance I have is the push for schemaless. Schemaless does not exist. There is always a schema, except in a schemaless data store, the schema has been moved to app code and then my earlier comment applies. This is a great point. It's like types for programming languages. They always exist; it's just a matter of whether you have the compiler managing it or whether you need to keep it in your head when you're hacking. > I think the lack of understanding of SQL and bad ORM experiences (Hibernate WTF) is what led people to think SQL/RDMBs were the problem when in fact they were not. Again, like types in programming languages, I think the ability to iterate quickly without having to predict how the data will be used in the future leads to 'schemaless' approaches which can be kicked out the door more quickly (maybe) than ponderous schema laden tables.
- mindcrime 10y agoI think the lack of understanding of SQL and bad ORM experiences (Hibernate WTF) I don't understand this comment. I've never had a bad experience with Hibernate. If anything, Hibernate has saved me tons of time over the years.
- matwood 10y agoI'd be curious about the size of the systems you have used with Hibernate. Do you know off the top of your head what will cause Hibernate to issue a flush? How about the order of operations in a batch? I've read the entire Hibernate book multiple times while trying to squeeze out speed, and finally decided enough. The problem with Hibernate is I have to learn all of these Hibernate quirks to maybe get decent speed out of a system, when I could have just written SQL and been done. This isn't even including debugging when Hibernate decides to issue N+1s.
- gerbilly 10y agoSpeed? I found Hibernate is great for CRUD. Like where you have dozens and dozens of domain objects that you want to persist, and you don't want to spend all your time writing DAO code with SQL in in that is all the same but different. Enterprise applications are rife with those. For speed write some hand written SQL to handle those few cases (like batch imports maybe). Also I found a handy rule: Only create an association between entities when the entity makes no sense without the other. For all other cases split them up and perhaps create aggregates by hand.
- mindcrime 10y agoThat jibes with my experience. We don't have tons of complicated associations, and do a lot of fairly straightforward CRUD operations. Maybe that's just the sweet spot for using Hibernate.
- cygned 10y agoI had to use Hibernate in a project. We were loading an entity from the database, someone invoked a setter and the set value was updated in the database automatically. Totally blew my mind. Never touched it again since then (not doing much Java anyway).
- cakes 10y agoThis mirrors the experience I have as well. I (currently) don't know what will cause Hibernate to flush but I certainly do know that I had to know this exactly and had to do some really painful debugging at points. The worst one I was on was just early prototyping of an app that used Hibernate and we had so many problems with performance and Hibernate(isms) that the whole thing seemed destined to fail even though our use case was exactly what ORM/Hibernate are for...yet we still have to control when changed flushed because otherwise things would get wonky really quickly.
- rwj 10y agoThis should be emphasised more. Schemaless is good for productivity during prototyping (like dynamically typed languages), but long term a well-defined schema will help maintenance (like statically typed languages).
- kbenson 10y ago> It's like types for programming languages. They always exist; it's just a matter of whether you have the compiler managing it or whether you need to keep it in your head when you're hacking. While true to a degree, it's worth noting that some languages also reduce the number of types considerably at the same time, and maybe even provide some other conveniences when dealing with types (e.g. automatic interpretation as numeric when used in a numeric operation). You still have to keep types in your head, but the reduction of types generally makes this non-problematic. Although, as your code grows and you define your own types (either formally through an included system or as ad-hoc data structures), you start needing to keep more types in your head, and this starts eating away at any benefit you had from not having to define them in the first place.
- dllthomas 10y agoYou and the parent are not using quite the same definition of "types", I think.
- kbenson 10y agoI think we are, if you look at it from a high level. C has some well known types most people are familiar with. Perl has a few types too, scalar, array and hash (and a few more lesser known or used ones). This makes Perl very easy to use initially, because you can just treat strings as string and numbers as numbers, and Perl will mostly do what you want based on the operators involved. You create complex data structures by just creating them (ad-hoc) or by actually defining a class, which actually gives you a new type. As you create any non-trivial program, you'll likely create some types, whether in C or Perl (as classes). What was initially a benefit in Perl, since you didn't have many typed to keep track of, becomes a liability, as there are more types but still no formal way to ensure they are being used in the correct locations and as expected arguments without runtime checking (and manual checking at that, unless you use a good module to handle a lot of that for you). That some languages keep types fairly low level (C, with typed relating fairly closely to machine representation) and some are high level (Perl, with each type being a higher level container) is really irrelevant. In the end, they are just labels that codify behavior (size, acceptable methods of use). In SQL and NoSQL systems, you have the same trade-offs. Do I define everything up-front so I know (presumably) when there's a problem, or do I create the structure ad-hoc as needed and enforce the structure through the function of the application that creates the data?
- exclusiv 10y agoSometimes it really is schemaless, although usually only schemaless for certain fields. Imagine a case where users can create custom fields or store arbitrary data in some fields.
- techno_modus 10y ago> Schemaless does not exist. There is always a schema... I also think so but the issue has several important aspects which justify the use of the separate term "schemaless". One of them is that schema elements (say, column names) can be stored as normal data. For example, instead of having normal columns like Name, Age, Department, we could introduce a column storing these strings in 3 rows. As a result, DBMS is simply unaware of the schema - the schema exists only in our head (and in the app). As a consequence, DBMS cannot help us too much in managing data, and instead our app becomes responsible for these tasks, hence we get problems you mentioned. But the major problem is that currently there is no technology that allows us to say that this table column stores actually column names which can be used in queries and have to be treated as normal columns.
- MarHoff 10y agoUse jsonb inside a Database and use constraints to ensure all requested json attributes (store name,etc..) are present at INSERT/UPDATE time? That don't seems like rocket science to me... duh... PS: But of course don't store ID in an object value or you will lost major benefit of RDMS relational constraints.
- jrochkind1 10y agoWell, now you're just as 'schemaless' as any NoSQL, so that's neither an advantage nor a disadvantage over it. There are other differences, both advantages and disadvantages to doing this with Postgres (which is the only db with a type called 'jsonb' I think?), vs some kind of non-SQL document store. I agree that as Postgres json(b) gets better and better, the disadvantages decrease. And one of the main advantages to me is simply not having to have another system to run and understand, since postgres is probably already there. But it's always trade-offs and choices, that you make better with more experience and better understanding of your domain. It's definitely not a "duh, it's always obvious" thing. Unless your domain is so simple that it doesn't hardly matter.
- SmellTheGlove 10y ago> PS: But of course don't store ID in an object value or you will lost major benefit of RDMS relational constraints. Or anything that you might want to use as a foreign key down the road. It requires you to know that now, or modify later, so we might as well declare a schema and manage it unless we're totally certain we won't need any additional keys. But if we're certain of that, we could just use a schema.
- jasode 10y ago>Schemaless does not exist. There is always a schema, except in a schemaless data store, the schema has been moved to app code I get what you're saying but keep in mind that "schemaless" isn't a philosophical statement about denying ontology[1]. Instead, it's an industry label for avoiding database schema operations. E.g. a rapidly-evolving app doesn't know ahead of time all the rigid columns they need so they use "schemaless" db strategy to avoid "ALTER TABLE ADD COLUMN X" or avoid an export/import to new v2,v3,etc tables. From the relative point-of-view of the db engine, it doesn't see a "schema" when the so-called "columns" are embedded as strings inside of generic fields. [1] https://en.wikipedia.org/wiki/Ontology https://en.wikipedia.org/wiki/Ontology
- VLM 10y agoAnd without the adult supervision that "ALTER TABLE" theoretically implies, you end up with 15 columns named "t1" and the like, no idea what any of it means, and source code to match (assuming its not gone and you're trying to reverse engineer the schema-less data) There is a place for unorganized, disorganized, unengineered, unplanned data structures, but its not in the DB.
- agentultra 10y agoChange management is not something that should be feared if given careful consideration! I don't always know the "perfect" schema design up front but there are great tools in the modern SQL tool belt for managing abstractions and evolving schemas. In PostgreSQL: CREATE SCHEMA crm; The "crm" schema can store all of my customer relationship management related tables. I may not know all of the data I need for them right now and will need to evolve them over time. However if I start with a solid design for my public schema I can use views in the public schema: CREATE VIEW public.customers AS SELECT c.id, c.contact_name, c.email FROM crm.customers AS c; This is a simple view. You applications can CREATE/UPDATE/INSERT against the view in the public schema with no changes to your code. However if you want to evolve the schema of customers you can do so in the `crm` schema adding tables, columns, etc. CREATE VIEW public.customers AS SELECT c.id, c.contact_name, c.email FROM crm.customers AS c WHERE deleted = false; You can then evolve your public schema to include more information as needed by updating the view. The caveat is that if your view starts materializing data from more than one table you'll have to intercept INSERT/UPDATE/DELETE and replace them with your own functions to put the data in the right place. This works quite well for implementing nice features like administrative flags without having to update your application code or spread that logic out: it's all kept in the database. You still get the power of indexes, a strong query language, and structured data. PostgreSQL is great as an application server allowing you to comment all of your objects and having deep introspection facilities allows you to not only be flexible with your data models but also consistent. Update: Added an example of changing the public schema to add a filter to the query for an administrative flag
- emilecantin 10y agoYeah, I think the good ORMs make the simple case _really_ simple (like User.getById(1234)) while letting you drop down to raw SQL easily when you need it. Any ORM that doesn't let you do that is not worth using, IMO.
- ZenoArrow 10y agoThank you for recommending jOOQ. I find many ORMs seem unnecessarily complicated compared to raw SQL (with a few exceptions), nice to find another one that offers readable code. Is the performance generally decent too?
- matwood 10y agoThe performance with jOOQ is great. I find it on par with raw SQL. It also has quite a few advanced ways to map the results that can take more complex queries and map them into usable data easily. The main jOOQ programmer really knows databases and has a very informative blog. My only complaint (and it's a minor one that would be hard to get around) is the need to run a code generator for jOOQ after every schema change.
- antoaravinth 10y agoJust a noob question on jooq. How do you construct error message which you use to send back to user on SQL exceptions. Like in case of duplicate item, you send back "User already exists". How your handling it? I know jooq just sends DataAccessException, now the question is how your converting that exception to a meaningful message which needs to be sent to say UI?
- lukaseder 10y agoThat's a really interesting question, and the best answer is: You have to write this manually. The SQL/JDBC/jOOQ layer can only generate system exceptions, like a constraint violation exception in this case. But there are many different types of constraints, and many different things that a constraint violation can mean. Only you and your service layer can possibly know how to translate such a violation to a meaningful (localised?) user error message. Having said so, you could hook in an ExecuteListener into jOOQ to handle errors in a single location: https://www.jooq.org/javadoc/latest/org/jooq/ExecuteListener.html https://www.jooq.org/javadoc/latest/org/jooq/ExecuteListener... An example that uses the Spring Exception translator: https://github.com/jOOQ/jOOQ/blob/master/jOOQ-examples/jOOQ-spring-example/src/main/java/org/jooq/example/spring/exception/ExceptionTranslator.java https://github.com/jOOQ/jOOQ/blob/master/jOOQ-examples/jOOQ-...
- Sir_Cmpwn 10y agoI ended up using NoSQL for ephemeral data and SQL for persistent data in most of my applications.
- sixdimensional 10y agoI find a simple analogy for schemaless in the OO world as well - an object without a class definition (for example a dynamic/anonymous object) and/or an interface has unknown properties and methods and is thus of highly limited use. Just like a table without a schema... because after all, it's still likely objects all the way down. There is no such thing as working without a "schema", because software simply can't function without data and corresponding metadata to describe it.
- fortpoint 10y agoI saw Mike Stonebraker while he was on his VoltDB "tour" a few years back. He makes the same arguments. In his talk he walked through the history of database technology and discussed a similar "war" between CODASYL and SQL. He framed that debate in terms of the split between separating the concerns of your data's structure (schema) from application logic. Fairly or not, he suggested the current fight between noSQL/Schemaless vs SQL/RDBMS was being fought in ignorance of all that went on in the 70s.
- ams6110 10y agoAgree so much. The first thing I do for almost any new system is design the data model. Once that satisfies all the requirements, building an application and UI on top of that is usually pretty straightforward.
- matwood 10y agoExactly what I do. I think in terms of data. UIs seem to change weekly, but the data lives forever.
- lukaseder 10y ago> There is always a schema It has a name. Schema-on-write (static schema) vs. schema-on-read (dynamic schema).