5 ms·

I was never able to get on board with Meteor because of the Mongo dependency. Not to get too far off topic, but Mongo is guaranteed to lose data under certain w

I was never able to get on board with Meteor because of the Mongo dependency. Not to get too far off topic, but Mongo is guaranteed to lose data under certain well-defined conditions. I completely fail to understand why some developers seem to be allergic to learning about data structures and the means of query-ing them. It's not really that hard. If you can JS, you can SQL.

minimaxir 11y ago
> I completely fail to understand why some developers seem to be allergic to learning about data structures and the means of query-ing them I'm not a fan of Meteor's dependency of Mongo either, but proper database architecture for a rDBMS is not an obvious skill. Even with an ORM (Rails/Django), it still takes care to set up PK/FK relationships and that's not necessarily taught in a CS curriculum. Mongo would be sufficient if the data loss issues were resolved. (JSON in PostgreSQL may be a best-of-both-worlds scenario)
- ianamartin 11y ago
  I might be speaking to a sparse audience here, but I don't come from a CS curriculum. I'm a violinist and a music theorist who happens to have learned enough to make a better living by writing code. I don't know what gets taught in CS degree plans. But I will say that if I can learn how to model, query, and display data, then pretty much anyone can. Especially those with the benefit of a CS degree. I don't use an ORM beyond certain limited functions. I use SQLAlchemy for a lightweight connection to the database and for migrations with Alembic. (apologies and accolades to zzzeek) Beyond that I have a wrapper around database stored procedures that objectifies results from the proc. I wrote it once. It has worked really well many times. It's not quite as easy as doing everything in your code. You have to support and maintain your SQL in addition to your web app code. But it is blazingly fast to do things this way, and in addition, you are going to have to do it this way eventually. You must implement business rules in the database, or you will suffer at some point. I disagree with you that Mongo will ever be sufficient, even if the Mongo team does manage the data loss problem (which, in my understanding, is that it's a feature--not a bug). I don't know of a method in the Mongo system that allows you to enforce business rules the way that SQL Server or PostgreSQL or Oracle does. But I would happily be enlightened. This could be a case where I just don't know enough.
  - uptownJimmy 11y ago
    I have no use for a dev stack that is tied firmly to a db that is incapable of handling relational data properly. Many, many devs agree.
  - squeaky-clean 11y ago
    The latest version of Mongo added support for document validation on insert/update. https://docs.mongodb.org/manual/core/document-validation/ https://docs.mongodb.org/manual/core/document-validation/
  - JonnieCache 11y ago
    >But I will say that if I can learn how to model, query, and display data, then pretty much anyone can. This isn't really true. >This could be a case where I just don't know enough. It sounds like you've picked up some very important things pretty well tbh. Better than a lot of people in the industry. I mean, your code could be a complete mess for all I know. But what you're describing is a potentially very powerful and robust design. You're likely not giving yourself enough credit. Hey, if you can navigate the tangled nightmare of conflicting redundancy that is music theory, modelling business objects and doing CRUD on them is surely a piece of cake ;)
extrapickles 11y ago
I currently use Meteor for a production app, but only use Mongo as a scratch pad (holding data until the user is mostly done changing their mind and other minor UI state). Data is then packaged and sent to MS SQL Server for long term storage and reporting. As an example, a user can start updating a piece of data (eg: sales order, item, etc), with everyone able to see the changes as they happen (no hard need for locking, easy to support temporary network loss). Once they are done with changes, they are sent to SQL. The advantage of doing it this way, is that the data stored in SQL Server can be completely optimized for reporting/storage and doesn't need the schema to support UI requirements. This makes things much simpler for people writing the UI as they don't have to change the SQL schema often to support the shinny new UI widget of the day and the report writers don't have to sift through a million useless to them tables/columns or partially entered rows to find the data they need.
- rashkov 11y ago
  This is very interesting. Could you give a little more details on how you accomplish this? How do you serialize unstructured Mongo data into MS SQL? Does this system break down when front-end people change the data structures in Mongo? I'm not familiar with Meteor but maybe you could explain in broad strokes
  - extrapickles 11y ago
    There are several ways we go about it depending on the needs of the data to be stored in SQL. All of our data in SQL and Mongo has a uuid as the primary key and effectively lets us use merge-style replication techniques, and support temporary offline use of our application. If the Mongo data is flat, its very easy to go from Mongo->SQL, only a type mapper is needed. The lumpy data case gets more complicated, as its dependent on the data. The complex data generally ends up translating into master/detail tables or similar ideas. Various pieces of data that is nice to have, but changes with the UI/customer needs, gets stuffed into several key/value tables (one for each type). For the completely dynamic Mongo data we store in SQL, we alter the SQL table programmatically to meet the new schema (in part of our application, customers must be able to drive the schema). This way works in our specific application, but it requires a lot of work to make sure your users cant paint themselves into corners with their schema. So far we haven't experienced any issues with the front-end people changing data structures, as the changes they generally make only impact fields that are not stored in SQL. Completely new features generally require the addition of new columns/tables to SQL, but every application has that issue. During startup of the server it double checks the schema of all of the tables used and updates as needed.
tomcam 11y ago
> If you can JS, you can SQL Reactive SQL in the Meteor style is tricky because who has time to reverse engineer DDA, and rolling it yourself would mean guaranteeing that you ended up doing it not the Meteor way.

minimaxir 11y ago

> I completely fail to understand why some developers seem to be allergic to learning about data structures and the means of query-ing them I'm not a fan of Meteor's dependency of Mongo either, but proper database architecture for a rDBMS is not an obvious skill. Even with an ORM (Rails/Django), it still takes care to set up PK/FK relationships and that's not necessarily taught in a CS curriculum. Mongo would be sufficient if the data loss issues were resolved. (JSON in PostgreSQL may be a best-of-both-worlds scenario)

ianamartin 11y ago
I might be speaking to a sparse audience here, but I don't come from a CS curriculum. I'm a violinist and a music theorist who happens to have learned enough to make a better living by writing code. I don't know what gets taught in CS degree plans. But I will say that if I can learn how to model, query, and display data, then pretty much anyone can. Especially those with the benefit of a CS degree. I don't use an ORM beyond certain limited functions. I use SQLAlchemy for a lightweight connection to the database and for migrations with Alembic. (apologies and accolades to zzzeek) Beyond that I have a wrapper around database stored procedures that objectifies results from the proc. I wrote it once. It has worked really well many times. It's not quite as easy as doing everything in your code. You have to support and maintain your SQL in addition to your web app code. But it is blazingly fast to do things this way, and in addition, you are going to have to do it this way eventually. You must implement business rules in the database, or you will suffer at some point. I disagree with you that Mongo will ever be sufficient, even if the Mongo team does manage the data loss problem (which, in my understanding, is that it's a feature--not a bug). I don't know of a method in the Mongo system that allows you to enforce business rules the way that SQL Server or PostgreSQL or Oracle does. But I would happily be enlightened. This could be a case where I just don't know enough.
- uptownJimmy 11y ago
  I have no use for a dev stack that is tied firmly to a db that is incapable of handling relational data properly. Many, many devs agree.
- squeaky-clean 11y ago
  The latest version of Mongo added support for document validation on insert/update. https://docs.mongodb.org/manual/core/document-validation/ https://docs.mongodb.org/manual/core/document-validation/
- JonnieCache 11y ago
  >But I will say that if I can learn how to model, query, and display data, then pretty much anyone can. This isn't really true. >This could be a case where I just don't know enough. It sounds like you've picked up some very important things pretty well tbh. Better than a lot of people in the industry. I mean, your code could be a complete mess for all I know. But what you're describing is a potentially very powerful and robust design. You're likely not giving yourself enough credit. Hey, if you can navigate the tangled nightmare of conflicting redundancy that is music theory, modelling business objects and doing CRUD on them is surely a piece of cake ;)

extrapickles 11y ago

I currently use Meteor for a production app, but only use Mongo as a scratch pad (holding data until the user is mostly done changing their mind and other minor UI state). Data is then packaged and sent to MS SQL Server for long term storage and reporting. As an example, a user can start updating a piece of data (eg: sales order, item, etc), with everyone able to see the changes as they happen (no hard need for locking, easy to support temporary network loss). Once they are done with changes, they are sent to SQL. The advantage of doing it this way, is that the data stored in SQL Server can be completely optimized for reporting/storage and doesn't need the schema to support UI requirements. This makes things much simpler for people writing the UI as they don't have to change the SQL schema often to support the shinny new UI widget of the day and the report writers don't have to sift through a million useless to them tables/columns or partially entered rows to find the data they need.

rashkov 11y ago
This is very interesting. Could you give a little more details on how you accomplish this? How do you serialize unstructured Mongo data into MS SQL? Does this system break down when front-end people change the data structures in Mongo? I'm not familiar with Meteor but maybe you could explain in broad strokes
- extrapickles 11y ago
  There are several ways we go about it depending on the needs of the data to be stored in SQL. All of our data in SQL and Mongo has a uuid as the primary key and effectively lets us use merge-style replication techniques, and support temporary offline use of our application. If the Mongo data is flat, its very easy to go from Mongo->SQL, only a type mapper is needed. The lumpy data case gets more complicated, as its dependent on the data. The complex data generally ends up translating into master/detail tables or similar ideas. Various pieces of data that is nice to have, but changes with the UI/customer needs, gets stuffed into several key/value tables (one for each type). For the completely dynamic Mongo data we store in SQL, we alter the SQL table programmatically to meet the new schema (in part of our application, customers must be able to drive the schema). This way works in our specific application, but it requires a lot of work to make sure your users cant paint themselves into corners with their schema. So far we haven't experienced any issues with the front-end people changing data structures, as the changes they generally make only impact fields that are not stored in SQL. Completely new features generally require the addition of new columns/tables to SQL, but every application has that issue. During startup of the server it double checks the schema of all of the tables used and updates as needed.

tomcam 11y ago

> If you can JS, you can SQL Reactive SQL in the Meteor style is tricky because who has time to reverse engineer DDA, and rolling it yourself would mean guaranteeing that you ended up doing it not the Meteor way.