6 ms·

I'm usually the guy arguing against microservices, but I think this is a good example of why they shine. It's much easier to manage the updating of these synth

I'm usually the guy arguing against microservices, but I think this is a good example of why they shine. It's much easier to manage the updating of these synthesized numbers if you have a discrete service that limits the things you can do at a higher level. So instead of worrying about who is updating the data store directly, you push it up to a higher service-level interface where you have a finite list of operations that are supported, and the scope is small enough that one architect can understand the system in its entirety and make smart decisions about how to compute/store/cache these synthesized data. That said, if you have less than 10 engineers this shouldn't even be thinking about this. Way too many young engineers get hung up on scalability when they don't even have anything on the horizon that will require scaling. Couple that with a general ignorance of SQL because it's old and boring, and not realizing how much performance you can eke out of a well tuned RDBMS with proper schema and queries. A SQL DB is an incredible hedge against an uncertain future, keep your data tidy and you can keep pivoting until you find something you actually have to scale.

buckbova 10y ago
That's not to mention simple app caching techniques for computed values.
coldtea 10y ago
>So instead of worrying about who is updating the data store directly, you push it up to a higher service-level interface where you have a finite list of operations that are supported, and the scope is small enough that one architect can understand the system in its entirety and make smart decisions about how to compute/store/cache these synthesized data. Nothing about this requires a microservice. A module / package / etc barrier in a "monolithic" app through a limited exposed interface is totally enough.
- dasil003 10y ago
  Yes yes, I know, I've made all the same arguments myself. But you have to admit, the isolation of a code base with a discrete test suite and a discrete team that has to think about nothing else gives a level of awareness to the subtleties on the potential issue that you don't get in a monolith. SOA is about scaling people more than anything.
derefr 10y ago
Speaking of RDBMSes, you can achieve this "limited API" by just creating an RDBMS user with limited privileges (can query a subset of tables; can make changes through a fixed set of stored procedures.) Personally, I never understood why everyone thought it was necessary to create HTTP API layers in front of every database, rather than just standardizing a public-Internet SQL wire-protocol between different RDBMSes and giving them all the ability to enforce per-user constraints on things like maximum query execution cost.
- s_kilk 10y ago
  > I never understood why everyone thought it was necessary to create HTTP API layers in front of every database Mostly because you usually need to do some kind of logic that's either difficult or impossible to represent in SQL views or functions. For example, how would your SQL view access something like redis? or any kind of external cache? or an HTTP service? This is the kind of stuff that's pretty common in HTTP Controller functions, and would be a nightmare to move into the database. (Note: I'm generally in favor of using SQL views/functions for data-facing logic, just, y'know, not a silver bullet.)
  - derefr 10y ago
    > For example, how would your SQL view access something like redis? or any kind of external cache? or an HTTP service? Smug comeback: https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers But your point still stands; no other RDBMS has those, so it can't really be an assumption. Still, there is a mismatch of paradigms here. The people who built the RDBMSes assumed that everyone would import data into them from wherever else. You, the user (or you, the application developer) shouldn't need to query a third-party web service when doing your database query; something else was supposed to do that query in expectation of your need, and shove the results into the database, so you could just treat that data as another table of useful information to JOIN on. People who front databases with web APIs tend to treat the database itself as a dumb store—one among many—that's supposed to quickly return "raw" data from single tables so that the web service can chew it together with other data to compute answers to questions. But, by doing so, they're in effect creating their own bespoke Database Management System: the web service becomes the thing that does efficient OnLine Analytical Processing of manifold backing datasets. (Or, at least, attempts to.) A DBMS's job is to ingest data and process queries to spit out answers. It's not your job; it's not your app's job; it's the DBMS's job. The DBMS has decades of optimization for that use-case; and in fact, that's the meaning of the word "database management system", when defined in opposition to a "flat-file database" like BDB or LevelDB. The fact that there's a daemon sitting there between you and your databases allows the daemon to index and join and expose views and run checks and enforce rules and on and on and on. It allows the daemon to do asynchronous, streaming imports of data-sets from remote sources, so that its data is always up-to-date. It allows the daemon to be there for things like crawler agents to dump their results into, rather than dumping them to something like an S3 bucket where you later have to be the one to import them in. In other words, being a daemon allows the DBMS to be the service that you just come to and ask questions—arbitrarily-complex analytical queries—and get answers. But that only works if the database management system itself is given access methods to all the databases that contain the data to answer your question. And, in modern practice, we've essentially forgotten that that is a thing that we ever wanted to do. Who would teach a database management system anything? Database connector libraries don't belong in DBMSes; they belong scattered throughout our toppling edifice of RESTful API microservices! (And if you argue that we still need those RESTful APIs for things like browser-clients wanting to do queries using AJAX: there's no reason DBMSes—not just document stores but full-fledged OLAP DBMSes—can't speak HTTP as well. CouchDB is a great but strangely-unique example. Adapter gateways also exist, like PostgREST.)

buckbova 10y ago

That's not to mention simple app caching techniques for computed values.

coldtea 10y ago

>So instead of worrying about who is updating the data store directly, you push it up to a higher service-level interface where you have a finite list of operations that are supported, and the scope is small enough that one architect can understand the system in its entirety and make smart decisions about how to compute/store/cache these synthesized data. Nothing about this requires a microservice. A module / package / etc barrier in a "monolithic" app through a limited exposed interface is totally enough.

dasil003 10y ago
Yes yes, I know, I've made all the same arguments myself. But you have to admit, the isolation of a code base with a discrete test suite and a discrete team that has to think about nothing else gives a level of awareness to the subtleties on the potential issue that you don't get in a monolith. SOA is about scaling people more than anything.

derefr 10y ago

Speaking of RDBMSes, you can achieve this "limited API" by just creating an RDBMS user with limited privileges (can query a subset of tables; can make changes through a fixed set of stored procedures.) Personally, I never understood why everyone thought it was necessary to create HTTP API layers in front of every database, rather than just standardizing a public-Internet SQL wire-protocol between different RDBMSes and giving them all the ability to enforce per-user constraints on things like maximum query execution cost.

s_kilk 10y ago
> I never understood why everyone thought it was necessary to create HTTP API layers in front of every database Mostly because you usually need to do some kind of logic that's either difficult or impossible to represent in SQL views or functions. For example, how would your SQL view access something like redis? or any kind of external cache? or an HTTP service? This is the kind of stuff that's pretty common in HTTP Controller functions, and would be a nightmare to move into the database. (Note: I'm generally in favor of using SQL views/functions for data-facing logic, just, y'know, not a silver bullet.)
- derefr 10y ago
  > For example, how would your SQL view access something like redis? or any kind of external cache? or an HTTP service? Smug comeback: https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers But your point still stands; no other RDBMS has those, so it can't really be an assumption. Still, there is a mismatch of paradigms here. The people who built the RDBMSes assumed that everyone would import data into them from wherever else. You, the user (or you, the application developer) shouldn't need to query a third-party web service when doing your database query; something else was supposed to do that query in expectation of your need, and shove the results into the database, so you could just treat that data as another table of useful information to JOIN on. People who front databases with web APIs tend to treat the database itself as a dumb store—one among many—that's supposed to quickly return "raw" data from single tables so that the web service can chew it together with other data to compute answers to questions. But, by doing so, they're in effect creating their own bespoke Database Management System: the web service becomes the thing that does efficient OnLine Analytical Processing of manifold backing datasets. (Or, at least, attempts to.) A DBMS's job is to ingest data and process queries to spit out answers. It's not your job; it's not your app's job; it's the DBMS's job. The DBMS has decades of optimization for that use-case; and in fact, that's the meaning of the word "database management system", when defined in opposition to a "flat-file database" like BDB or LevelDB. The fact that there's a daemon sitting there between you and your databases allows the daemon to index and join and expose views and run checks and enforce rules and on and on and on. It allows the daemon to do asynchronous, streaming imports of data-sets from remote sources, so that its data is always up-to-date. It allows the daemon to be there for things like crawler agents to dump their results into, rather than dumping them to something like an S3 bucket where you later have to be the one to import them in. In other words, being a daemon allows the DBMS to be the service that you just come to and ask questions—arbitrarily-complex analytical queries—and get answers. But that only works if the database management system itself is given access methods to all the databases that contain the data to answer your question. And, in modern practice, we've essentially forgotten that that is a thing that we ever wanted to do. Who would teach a database management system anything? Database connector libraries don't belong in DBMSes; they belong scattered throughout our toppling edifice of RESTful API microservices! (And if you argue that we still need those RESTful APIs for things like browser-clients wanting to do queries using AJAX: there's no reason DBMSes—not just document stores but full-fledged OLAP DBMSes—can't speak HTTP as well. CouchDB is a great but strangely-unique example. Adapter gateways also exist, like PostgREST.)