5 ms·

I have recently been thinking about doing microservices by simply having a shared MySql DB that all modules connect to. So for example one developer can create

by TekMol 9y ago

I have recently been thinking about doing microservices by simply having a shared MySql DB that all modules connect to. So for example one developer can create the backend and one developer can create the frontend. The codebases can be completely independent. One could use PHP+Laravel and the other one PHP+Symfony for example. Frontend and backend would live on their own servers. And simply having the IP, login and PW of the shared DB set in their project. What do you guys think about such an approach?

fnbr 9y ago
That's what I did for a recent side project. It worked really well, although coordinating the different services was a bit of a pain. We ended up using a cronjob to run everything repeatedly, which was inefficient but worked. If we had the time we would have used a proper RPC framework or some sort of queue.
- TekMol 9y ago
  What do you mean by 'run everything repeatedly'?
  - fnbr 9y ago
    For instance, say I had 2 services, a scraper and then a ML algorithm that classified the scraped data. You only need to run the ML classifier when there's new data. I didn't have anything set up to tell the classifier "hey! There's new data to classify!" Instead, I just ran it every 5 minutes, which was sufficient latency for the project.
wmf 9y ago
That sounds like the integration database pattern and it's generally not considered service-oriented (of which microservices is a subset).
0xCMP 9y ago
(Written quickly on my phone) Issue ends up being how to keep applications from ruining assumptions made in other services. There’s no single code base which you can walk through and see possible changes. At that point this isn’t making things easier to understand or maintain so it’s not the best approach.
Boxxed 9y ago
I think that sounds like a bad idea. I could see pain arising from the shared usage of the database -- e.g., during migrations or heavy usage. I'm not really sure what the benefit is, either: each team will likely want to be the owner of their own database anyway. If you're talking about the separate microservices using the same tables in the database, that's probably an even worse idea. Either your services are way too tightly coupled, or shouldn't have been split into separate services in the first place.
dvt 9y ago
> shared MySql DB This is already a smell. Usually, microservices don't need to access global state.
saryant 9y ago
I've been down that road. Eventually it turns into a nightmare because evolving your DB means making simultaneous changes to multiple applications. "We think we can drop this column, someone figure out which of our eight apps using this DB might be using it still" I much prefer putting a single service layer in front of the DB that speaks thrift or protobuf and letting all clients interface with that instead. Evolving thrift services is fairly straightforward and allows you to make changes without needing every app the keep up.
- TekMol 9y ago
  How does an additional layer make changes to the DB easier? If a column is dropped what will your layer do when a request for data from that column comes in? What if the request joins it with other columns from other tables?
  - cookiecaper 9y ago
    It's better because if you have one application that has authority over that area, you only have to answer this question once. The answer will depend on the data in question, of course; maybe it is fine to serve stale data for a while, maybe you need to write to one DB, read from another, and combine in-process, etc., until the change fully propagates. But the impact needs to be localized to whatever extent is possible. If you have several applications accessing the database directly, it makes the database everyone's problem, instead of just the one thing's problem. Then everyone has to know about the downtime and come up with their own strategy to mitigate. They can't say "Well we'll just trust what we get from Service A", because they don't actually get info from service A; they get info from service A's underlying datastore. Worse, in most cases like this, there will just be one global database for everything, so schema changes, database restarts, etc., necessary for one thing can have negative effects, both direct and indirect, across the entire ecosystem. If Bob's Service decides it needs to do a massive reindexing and Alice's Service is on the same DB, even if they're using completely independent tables, etc., the performance hit is going to affect both. If Bob changes his schema and Alice reads or writes directly to those tables (e.g., Alice's service updates a column in records originally inserted by Bob's service), now Alice has to know about the change, plan for it, and coordinate her deployment in sync with Bob, etc. That kind of thing is what people mean when they say "distributed monolith". There is no real "private" and "public" space where one service provider could reasonably offer a stable API but change things as necessary on the back-end. Nothing is really independent. All you've done is make a monolith that is much harder to coordinate, manage, debug, and understand.
laurencei 9y ago
The problem is the codebases are still not "completely independent". If the backend dev needs to change a field in the database, or add a new one, then can might affect the front end dev. Also - simple things like validation rules - i.e. what can the front end dev allow users to store into the DB vs what can the back end dev use/need.
- TekMol 9y ago
  What would be an example of a situation where the backend dev needs to change a field in the database?
  - sfifs 9y ago
    Say you are migrating password algorithms in a login system. Typical way to do it would be to add columns for new algorithm hash and ask people to change passwords once they login with the old one. Front end only needs to know to show the change password screen.
buttscicles 9y ago
If multiple services are coupled to the same database schema then any changes to the schema must be coordinated between all services, so you've got something of a distributed monolith instead of a set of independent microservices. This might be fine, and maybe it could even be a good choice in your situation, but I would probably avoid the added complexity.
- TekMol 9y ago
  I would say it is less complexity then adding another API on top of the database.
toast0 9y ago
Everything that accesses one table should be in one place. If you're sharing the database because you don't have enough use of the tables to justify splitting it up, that's fine. Do yourself a favor and create different MySQL users that can only access the tables needed. If you're sharing the database because sometimes you want to access a table from the frontend and sometimes you want to access that same table from your backend layer, you're probably setting yourself up for pain later.
cookiecaper 9y ago
At the very least, each service should "own" its area of concern, which means only one application should ever write to the underlying structures, and those structures should support that application alone. What happens if you have an app that needs a certain setting on the database, but you can't restart to make it effective because 9 other apps are using the same DB? It's hard enough to coordinate this kind of maintenance when there is only one OLTP-style user of the database. If you get a lot of interdependent units, where app A needs to read table B and app C needs to write to table D, it's nigh impossible to do it without global downtime. "Microservices" require a large amount of control and discipline to implement properly. In almost all cases, a straight-up monolith ends up being much saner. "Microservices" are often seen as a license to run hog wild and disregard everything outside of one's immediate area of concern / team. This is convenient at the time because of Conway's Law, but it is terrible for long-term maintenance, overall consistency, and employee sanity.
Floegipoky 9y ago
By most definitions of microservices I've heard there's a 1:many relationship between microservices:datastores. Multiple microservices can't talk to the same database. For example: > While monolithic applications prefer a single logical database for persistant data, enterprises often prefer a single database across a range of applications... Microservices prefer letting each service manage its own database, either different instances of the same database technology, or entirely different database systems https://martinfowler.com/articles/microservices.html https://martinfowler.com/articles/microservices.html Not to say that pattern can't be successfully implemented, just that it wouldn't be considered microservices. It sounds like you're describing a three-tier architecture: https://en.wikipedia.org/wiki/Multitier_architecture#Three-tier_architecture https://en.wikipedia.org/wiki/Multitier_architecture#Three-t...
deleted 9y ago
[deleted]
nostrademons 9y ago
That's how we used to build systems back in the 3-tier architecture (DB, app server, UI) days. Usually backend servers would be hooked up to a workflow engine or message bus like Celery, JMQ, or plain old crontab to do batch tasks. It's not a bad system - it worked for a technology generation, from around 1997-2012 - but it's somewhat limited in the problem domains it can solve effectively. Anything real-time or with high change volume becomes problematic, because each service needs to poll the DB for work to do. The DB can quickly become a scaling bottleneck. It's not as well-suited to thick clients, because you need an app server in front of the DB anyway to check & validate requests so the entire world doesn't get access to all your data. On the plus side, state coherency becomes much easier, because all state is in the DB and you can use transactions to update it atomically. I also disagree with the posters who say that updates & versioning are a nightmare: they are, but updates are a nightmare in any distributed system, and sticking an app server in front of the DB just makes it easier to know when you've made a breaking change, it doesn't make the change itself easier. I've worked on large (= Google) protobuf-based SOAs before, and changing anything that touches backend protocols or storage is difficult regardless of which architecture you use. I'm a big fan of the "solve the problem first, scale the architecture later" approach. For your first version, just store everything in hashtables/lists in RAM, in a single process, and serve out RPCs to client code (or run a webserver) as necessary. At this point, you don't have a useful product anyway, so it doesn't matter if it goes down and loses all data. Once you have something that's useful, work on moving state out to external systems: at this point, you have a pretty good idea what the schema should be, so you can save on a lot of migrations. Split into separate services when the computational resources exceed the ability of a single process to service them; before then, just split it up into libraries.
nickserv 9y ago
That's not a pure microservices approach, since you are sharing state (the database). It means that any development which changes or is impacted by a change in the DB has to be coordinated between teams. The microservice approach is to have each service have its own database (MySQL) or schema (PostgreSQL). Another solution is the use of a modular architecture, where separate modules are each responsible for their database tables' schema. Tables are not shared across modules; accessing data from a table requires calling its module's API. Each module can be developed independently, and combined into one large service or several smaller services. It's helpful to be able to separate modules into their own service for load balancing and optimization, while having all modules in the same service eases administration of the system. From previous experience, you start by lumping all modules together, then use monitoring software (i.e. ELK stack, collectd/Grafana) you decide where to split into services. Because the reality is you'll not really know where to split until you are running in production. This is the approach of Django for example and works quite well for large projects.
brown9-2 9y ago
Shared state across different services is a bad bad idea.

by TekMol 9y ago

fnbr 9y ago

That's what I did for a recent side project. It worked really well, although coordinating the different services was a bit of a pain. We ended up using a cronjob to run everything repeatedly, which was inefficient but worked. If we had the time we would have used a proper RPC framework or some sort of queue.

TekMol 9y ago
What do you mean by 'run everything repeatedly'?
- fnbr 9y ago
  For instance, say I had 2 services, a scraper and then a ML algorithm that classified the scraped data. You only need to run the ML classifier when there's new data. I didn't have anything set up to tell the classifier "hey! There's new data to classify!" Instead, I just ran it every 5 minutes, which was sufficient latency for the project.

wmf 9y ago

That sounds like the integration database pattern and it's generally not considered service-oriented (of which microservices is a subset).

0xCMP 9y ago

(Written quickly on my phone) Issue ends up being how to keep applications from ruining assumptions made in other services. There’s no single code base which you can walk through and see possible changes. At that point this isn’t making things easier to understand or maintain so it’s not the best approach.

Boxxed 9y ago

I think that sounds like a bad idea. I could see pain arising from the shared usage of the database -- e.g., during migrations or heavy usage. I'm not really sure what the benefit is, either: each team will likely want to be the owner of their own database anyway. If you're talking about the separate microservices using the same tables in the database, that's probably an even worse idea. Either your services are way too tightly coupled, or shouldn't have been split into separate services in the first place.

dvt 9y ago

> shared MySql DB This is already a smell. Usually, microservices don't need to access global state.

saryant 9y ago

I've been down that road. Eventually it turns into a nightmare because evolving your DB means making simultaneous changes to multiple applications. "We think we can drop this column, someone figure out which of our eight apps using this DB might be using it still" I much prefer putting a single service layer in front of the DB that speaks thrift or protobuf and letting all clients interface with that instead. Evolving thrift services is fairly straightforward and allows you to make changes without needing every app the keep up.

TekMol 9y ago
How does an additional layer make changes to the DB easier? If a column is dropped what will your layer do when a request for data from that column comes in? What if the request joins it with other columns from other tables?
- cookiecaper 9y ago
  It's better because if you have one application that has authority over that area, you only have to answer this question once. The answer will depend on the data in question, of course; maybe it is fine to serve stale data for a while, maybe you need to write to one DB, read from another, and combine in-process, etc., until the change fully propagates. But the impact needs to be localized to whatever extent is possible. If you have several applications accessing the database directly, it makes the database everyone's problem, instead of just the one thing's problem. Then everyone has to know about the downtime and come up with their own strategy to mitigate. They can't say "Well we'll just trust what we get from Service A", because they don't actually get info from service A; they get info from service A's underlying datastore. Worse, in most cases like this, there will just be one global database for everything, so schema changes, database restarts, etc., necessary for one thing can have negative effects, both direct and indirect, across the entire ecosystem. If Bob's Service decides it needs to do a massive reindexing and Alice's Service is on the same DB, even if they're using completely independent tables, etc., the performance hit is going to affect both. If Bob changes his schema and Alice reads or writes directly to those tables (e.g., Alice's service updates a column in records originally inserted by Bob's service), now Alice has to know about the change, plan for it, and coordinate her deployment in sync with Bob, etc. That kind of thing is what people mean when they say "distributed monolith". There is no real "private" and "public" space where one service provider could reasonably offer a stable API but change things as necessary on the back-end. Nothing is really independent. All you've done is make a monolith that is much harder to coordinate, manage, debug, and understand.

laurencei 9y ago

The problem is the codebases are still not "completely independent". If the backend dev needs to change a field in the database, or add a new one, then can might affect the front end dev. Also - simple things like validation rules - i.e. what can the front end dev allow users to store into the DB vs what can the back end dev use/need.

TekMol 9y ago
What would be an example of a situation where the backend dev needs to change a field in the database?
- sfifs 9y ago
  Say you are migrating password algorithms in a login system. Typical way to do it would be to add columns for new algorithm hash and ask people to change passwords once they login with the old one. Front end only needs to know to show the change password screen.

buttscicles 9y ago

If multiple services are coupled to the same database schema then any changes to the schema must be coordinated between all services, so you've got something of a distributed monolith instead of a set of independent microservices. This might be fine, and maybe it could even be a good choice in your situation, but I would probably avoid the added complexity.

TekMol 9y ago
I would say it is less complexity then adding another API on top of the database.

toast0 9y ago

Everything that accesses one table should be in one place. If you're sharing the database because you don't have enough use of the tables to justify splitting it up, that's fine. Do yourself a favor and create different MySQL users that can only access the tables needed. If you're sharing the database because sometimes you want to access a table from the frontend and sometimes you want to access that same table from your backend layer, you're probably setting yourself up for pain later.

cookiecaper 9y ago

At the very least, each service should "own" its area of concern, which means only one application should ever write to the underlying structures, and those structures should support that application alone. What happens if you have an app that needs a certain setting on the database, but you can't restart to make it effective because 9 other apps are using the same DB? It's hard enough to coordinate this kind of maintenance when there is only one OLTP-style user of the database. If you get a lot of interdependent units, where app A needs to read table B and app C needs to write to table D, it's nigh impossible to do it without global downtime. "Microservices" require a large amount of control and discipline to implement properly. In almost all cases, a straight-up monolith ends up being much saner. "Microservices" are often seen as a license to run hog wild and disregard everything outside of one's immediate area of concern / team. This is convenient at the time because of Conway's Law, but it is terrible for long-term maintenance, overall consistency, and employee sanity.

Floegipoky 9y ago

By most definitions of microservices I've heard there's a 1:many relationship between microservices:datastores. Multiple microservices can't talk to the same database. For example: > While monolithic applications prefer a single logical database for persistant data, enterprises often prefer a single database across a range of applications... Microservices prefer letting each service manage its own database, either different instances of the same database technology, or entirely different database systems https://martinfowler.com/articles/microservices.html https://martinfowler.com/articles/microservices.html Not to say that pattern can't be successfully implemented, just that it wouldn't be considered microservices. It sounds like you're describing a three-tier architecture: https://en.wikipedia.org/wiki/Multitier_architecture#Three-tier_architecture https://en.wikipedia.org/wiki/Multitier_architecture#Three-t...

deleted 9y ago

[deleted]

nostrademons 9y ago

That's how we used to build systems back in the 3-tier architecture (DB, app server, UI) days. Usually backend servers would be hooked up to a workflow engine or message bus like Celery, JMQ, or plain old crontab to do batch tasks. It's not a bad system - it worked for a technology generation, from around 1997-2012 - but it's somewhat limited in the problem domains it can solve effectively. Anything real-time or with high change volume becomes problematic, because each service needs to poll the DB for work to do. The DB can quickly become a scaling bottleneck. It's not as well-suited to thick clients, because you need an app server in front of the DB anyway to check & validate requests so the entire world doesn't get access to all your data. On the plus side, state coherency becomes much easier, because all state is in the DB and you can use transactions to update it atomically. I also disagree with the posters who say that updates & versioning are a nightmare: they are, but updates are a nightmare in any distributed system, and sticking an app server in front of the DB just makes it easier to know when you've made a breaking change, it doesn't make the change itself easier. I've worked on large (= Google) protobuf-based SOAs before, and changing anything that touches backend protocols or storage is difficult regardless of which architecture you use. I'm a big fan of the "solve the problem first, scale the architecture later" approach. For your first version, just store everything in hashtables/lists in RAM, in a single process, and serve out RPCs to client code (or run a webserver) as necessary. At this point, you don't have a useful product anyway, so it doesn't matter if it goes down and loses all data. Once you have something that's useful, work on moving state out to external systems: at this point, you have a pretty good idea what the schema should be, so you can save on a lot of migrations. Split into separate services when the computational resources exceed the ability of a single process to service them; before then, just split it up into libraries.

nickserv 9y ago

That's not a pure microservices approach, since you are sharing state (the database). It means that any development which changes or is impacted by a change in the DB has to be coordinated between teams. The microservice approach is to have each service have its own database (MySQL) or schema (PostgreSQL). Another solution is the use of a modular architecture, where separate modules are each responsible for their database tables' schema. Tables are not shared across modules; accessing data from a table requires calling its module's API. Each module can be developed independently, and combined into one large service or several smaller services. It's helpful to be able to separate modules into their own service for load balancing and optimization, while having all modules in the same service eases administration of the system. From previous experience, you start by lumping all modules together, then use monitoring software (i.e. ELK stack, collectd/Grafana) you decide where to split into services. Because the reality is you'll not really know where to split until you are running in production. This is the approach of Django for example and works quite well for large projects.

brown9-2 9y ago

Shared state across different services is a bad bad idea.