7 ms·

Microservices – Demo Project in Python

35 points by prince_af_123 11y ago

bulatb 11y ago
The thing I really want to see is how a good authentication and authorization system works, for both the user-facing services and interservice calls. I've looked around a bit and never found a resource that addresses that important problem.
- zepolen 11y ago
  You wouldn't want authorization for internal interservice calls, way too much overhead and a firewall is more secure.
  - newman314 11y ago
    Firewall != AAA != encrypted transport They have different usages; I wouldn't say one is more secure than the other.
    - zepolen 11y ago
      For interservice calls a firewall is much more secure than any auth system, way less vectors of attack. Encrypted transport on public networks should be a given anyway, although if you have a private network then that is unnecessary also. The only place auth is needed is for services that interact with users, and that works well in the microservice philosophy having an auth service which talks to all services as in user <-> service <-> auth You can scale the auth system indirectly to the other services, and since auth is required almost everywhere means that you'll have a large part of your resources devoted to that. Where it gets interesting is when you use external microservices for auth like stormpath - is it a good idea to talk to that directly, or use a mediation auth service of your own - I usually prefer the 2nd method since while there is an extra hop - it makes the api more static and allows you to swap out what your app uses for auth much more easily.
      - newman314 11y ago
        I disagree. Given what we now know post Snowden, it's certainly advisable to encrypt internal transports whenever possible. As an additional example, lots of people tend to mistake MPLS as being encrypted as opposed to just being a private network so it's definitely advisable to make sure traffic is encrypted whenever possible inside or out.
        zepolen 11y ago
        Good point.
  - BillinghamJ 11y ago
    That assumes you have complete trust and confidence in the integrity and security of every single process running within your internal network. I personally prefer to maintain a "need to know" approach to communication between all systems on my network (including services, databases, infrastructure). This means that, for example, if my public-facing vehicle lookup service is compromised, my users' data is still a long way away. This is enforced by: (1) not providing an address to the service, (2) not providing the keys/passwords required to authenticate with the service. Not doing authorization internally means that the moment a single process running on your network is compromised (which almost certainly is possible), your attacker now has access to the entirety of your system, and you're fucked. Also, the overheads can be extremely low - to the point that they're not noticeable. It just depends how you do it.
zepolen 11y ago
This shows a few of the drawbacks of microservices one of which is memory. This project used about 15mb per python instance = 60mb total when the entire stack is launched. Once you load a few largish python libraries like sqlalchemy you can get up to 100mb per instance x 4 = 400mb. What would be interesting is a framework that allows you to run the entire microservice stack in a single interpreter (shouldn't be hard with flask blueprints) and break off the subapps as necessary when the need arises. You would need a way of managing the current endpoints for each service vs a requests.get('http://localhost:ARBITRARY_PORT' http://localhost:ARBITRARY_PORT') in the code, but if done correctly would make managing a cluster much simpler. Does anyone know if such a thing exists?
- z0mbix 11y ago
  Have you looked at Kong? (https://getkong.org https://getkong.org)
- prince_af_123 11y ago
  > What would be interesting is a framework that allows you to run the entire microservice stack in a single interpreter (shouldn't be hard with flask blueprints) and break off the subapps as necessary when the need arises. The idea behind microservices is to have small services running on their own processes and communicating using a light-weight mechanism. This means you could write a service in Java, another in Python, Node.js etc. (i.e. pick the most appropriate language for the task)
kpmah 11y ago
Number 1 problem with microservices here: https://github.com/umermansoor/microservices/blob/master/services/user.py#L64 https://github.com/umermansoor/microservices/blob/master/ser... N+1 HTTP requests. Ouch. Wouldn't it be nice to be able to do a database join?
- nickknissen 11y ago
  or just retrieve a list of movies movies_resp = requests.get("http://127.0.0.1:5001/movies?id=1&id=2&id=3")
  - kpmah 11y ago
    Right, so you implement this multi-get for every service, you work around the GET URL length limit by using POST or something, you do this slow HTTP join in parallel with any other service data you need if possible, in your application code, and you get what a single SQL JOIN gives you.
- peterhunt 11y ago
  There's a lot of benefits to this approach over a database join if you have async IO. Specifically, sharding becomes trivial and you can cache each movie individually, so if one movie changes you don't need to invalidate the cache for all of them.
  - kpmah 11y ago
    Yes, it might make sense if you're Facebook or Amazon to do this stuff. For the rest of us, a single (reasonably optimised) database can scale up to thousands of hits per second. Probably more. Also there's another downside to this approach: you lose the ability to do atomic transactions on the application's data.
    - peterhunt 11y ago
      Sure, but if you ever need to shard, you can no longer assume you can do atomic transactions anyway, which means you need to rewrite a lot of stuff. By designing in this way, your scaling problems change from "the entire app fell over" to "the movie service fell over", which really helps with availability. Not that there aren't disadvantages with microservices (overhead, distributed tracing, etc), I just think app architecture isn't one of them.
trymas 11y ago
``` Requirements Python 2.7 ``` oh sad day :(
- cdnsteve 11y ago
  Flask works just fine on 3.5 so you could easily update any breaking differences in this demo repo.
Endurance1 11y ago
Hello for everyone! Here are new great gadgets!Would you please tell your opinion about them.Had you ever used some similar devices?Thanks for everyone! Lasers and engraving machines made by Endurance. https://youtu.be/GzUtTyspTgQ https://youtu.be/GzUtTyspTgQ Endurance SelfieBot. Smart gadget that connects hearts. https://youtu.be/ccFt0MII1uc https://youtu.be/ccFt0MII1uc Here the official wedsite of the greatest Russian American company that is specializing in development and sales of high technology production, mainly, lasers, diode laser engravers, telepresence robots and robotized systems declares the pre-seed round of investments attraction to the project closed. http://endurancerobots.com/about-endurance/ http://endurancerobots.com/about-endurance/
jone673 11y ago
Isn't Microservices a fancy new term for SOA? The positioning machine in action...

35 points by prince_af_123 11y ago

The thing I really want to see is how a good authentication and authorization system works, for both the user-facing services and interservice calls. I've looked around a bit and never found a resource that addresses that important problem.

zepolen 11y ago
You wouldn't want authorization for internal interservice calls, way too much overhead and a firewall is more secure.
- newman314 11y ago
  Firewall != AAA != encrypted transport They have different usages; I wouldn't say one is more secure than the other.
  - zepolen 11y ago
    For interservice calls a firewall is much more secure than any auth system, way less vectors of attack. Encrypted transport on public networks should be a given anyway, although if you have a private network then that is unnecessary also. The only place auth is needed is for services that interact with users, and that works well in the microservice philosophy having an auth service which talks to all services as in user <-> service <-> auth You can scale the auth system indirectly to the other services, and since auth is required almost everywhere means that you'll have a large part of your resources devoted to that. Where it gets interesting is when you use external microservices for auth like stormpath - is it a good idea to talk to that directly, or use a mediation auth service of your own - I usually prefer the 2nd method since while there is an extra hop - it makes the api more static and allows you to swap out what your app uses for auth much more easily.
    - newman314 11y ago
      I disagree. Given what we now know post Snowden, it's certainly advisable to encrypt internal transports whenever possible. As an additional example, lots of people tend to mistake MPLS as being encrypted as opposed to just being a private network so it's definitely advisable to make sure traffic is encrypted whenever possible inside or out.
      - zepolen 11y ago
        Good point.
- BillinghamJ 11y ago
  That assumes you have complete trust and confidence in the integrity and security of every single process running within your internal network. I personally prefer to maintain a "need to know" approach to communication between all systems on my network (including services, databases, infrastructure). This means that, for example, if my public-facing vehicle lookup service is compromised, my users' data is still a long way away. This is enforced by: (1) not providing an address to the service, (2) not providing the keys/passwords required to authenticate with the service. Not doing authorization internally means that the moment a single process running on your network is compromised (which almost certainly is possible), your attacker now has access to the entirety of your system, and you're fucked. Also, the overheads can be extremely low - to the point that they're not noticeable. It just depends how you do it.

zepolen 11y ago

This shows a few of the drawbacks of microservices one of which is memory. This project used about 15mb per python instance = 60mb total when the entire stack is launched. Once you load a few largish python libraries like sqlalchemy you can get up to 100mb per instance x 4 = 400mb. What would be interesting is a framework that allows you to run the entire microservice stack in a single interpreter (shouldn't be hard with flask blueprints) and break off the subapps as necessary when the need arises. You would need a way of managing the current endpoints for each service vs a requests.get('http://localhost:ARBITRARY_PORT' http://localhost:ARBITRARY_PORT') in the code, but if done correctly would make managing a cluster much simpler. Does anyone know if such a thing exists?

z0mbix 11y ago
Have you looked at Kong? (https://getkong.org https://getkong.org)
prince_af_123 11y ago
> What would be interesting is a framework that allows you to run the entire microservice stack in a single interpreter (shouldn't be hard with flask blueprints) and break off the subapps as necessary when the need arises. The idea behind microservices is to have small services running on their own processes and communicating using a light-weight mechanism. This means you could write a service in Java, another in Python, Node.js etc. (i.e. pick the most appropriate language for the task)

kpmah 11y ago

Number 1 problem with microservices here: https://github.com/umermansoor/microservices/blob/master/services/user.py#L64 https://github.com/umermansoor/microservices/blob/master/ser... N+1 HTTP requests. Ouch. Wouldn't it be nice to be able to do a database join?

nickknissen 11y ago
or just retrieve a list of movies movies_resp = requests.get("http://127.0.0.1:5001/movies?id=1&id=2&id=3")
- kpmah 11y ago
  Right, so you implement this multi-get for every service, you work around the GET URL length limit by using POST or something, you do this slow HTTP join in parallel with any other service data you need if possible, in your application code, and you get what a single SQL JOIN gives you.
peterhunt 11y ago
There's a lot of benefits to this approach over a database join if you have async IO. Specifically, sharding becomes trivial and you can cache each movie individually, so if one movie changes you don't need to invalidate the cache for all of them.
- kpmah 11y ago
  Yes, it might make sense if you're Facebook or Amazon to do this stuff. For the rest of us, a single (reasonably optimised) database can scale up to thousands of hits per second. Probably more. Also there's another downside to this approach: you lose the ability to do atomic transactions on the application's data.
  - peterhunt 11y ago
    Sure, but if you ever need to shard, you can no longer assume you can do atomic transactions anyway, which means you need to rewrite a lot of stuff. By designing in this way, your scaling problems change from "the entire app fell over" to "the movie service fell over", which really helps with availability. Not that there aren't disadvantages with microservices (overhead, distributed tracing, etc), I just think app architecture isn't one of them.

trymas 11y ago

``` Requirements Python 2.7 ``` oh sad day :(

cdnsteve 11y ago
Flask works just fine on 3.5 so you could easily update any breaking differences in this demo repo.

Endurance1 11y ago

Hello for everyone! Here are new great gadgets!Would you please tell your opinion about them.Had you ever used some similar devices?Thanks for everyone! Lasers and engraving machines made by Endurance. https://youtu.be/GzUtTyspTgQ https://youtu.be/GzUtTyspTgQ Endurance SelfieBot. Smart gadget that connects hearts. https://youtu.be/ccFt0MII1uc https://youtu.be/ccFt0MII1uc Here the official wedsite of the greatest Russian American company that is specializing in development and sales of high technology production, mainly, lasers, diode laser engravers, telepresence robots and robotized systems declares the pre-seed round of investments attraction to the project closed. http://endurancerobots.com/about-endurance/ http://endurancerobots.com/about-endurance/

jone673 11y ago

Isn't Microservices a fancy new term for SOA? The positioning machine in action...