10 ms·

A List of Useful Backend Architectures

98 points by yarapavan 13y ago

asperous 13y ago
Completely dominated by open source projects. This helps my theory that when you release open source code, you are improving the baseline technology for the entire world.
- thejosh 13y ago
  Yep. And when companies release the products THEY use and find helpful, everyone improves it and releases it back. Memcache springs to mind.
  - zimbatm 13y ago
    Not necessarily. I have tons of github repos where nobody contributes to. I don't see why it wouldn't be the same for a company. It depends if your project is useful to others and if you're able to market it properly.
    - lifeisstillgood 13y ago
      Yes, Yes, Yes. OSS needs marketing, and the big ones recently have almost all sprung from world famous companies (twitter bootstrap, facebook and phabricator etc etc). It will become harder and harder to get noticed, but this is truly a open competitive market, and the best becomes top dog in most cases. CEOs who spout support for the Free Market - beware, this is what it looks like. :-)
- krenoten 13y ago
  There is an extreme case of the availability heuristic at work here. There are a ton of incredibly successful companies that use little or no OSS.
presty 13y ago
architectures != stacks
- jaggederest 13y ago
  Agreed. This is a list of buzzwords, not a list of architectures. An architecture would be something like 'all major work done via job queues distributed across cheap machines, all pages statically built and pushed to CMS', or 'a dozen bare-metal machines with serious hardware running one-machine sharded app/cache/db slices and hot backups of each shard', fun stuff like that. relatively technology agnostic.
- gizzlon 13y ago
  Agreed, but it's still a nice list with a lot of information. Check out the first chapter of "The Architecture of Open Source Applications (Volume Two)": "Scalable Web Architecture and Distributed Systems" http://www.aosabook.org/en/distsys.html http://www.aosabook.org/en/distsys.html
camus 13y ago
A a lot of these start-up (except for Facebook maybe ) are using python in their stack, why is that ? all I hear about around me is "Java Java Java". What makes python better ? ( it is just a question ). Or is python just for mvp then the stack moves to Java ( like Twitter did with rails )?
- kaoD 13y ago
  I'd advise against Python. There are too many caveats in it for a language that's supposed not to surprise me. My biggest gripe: it's easy to shoot yourself in the foot using Unicode. The system is cumbersome and the lack of typing makes it easy to slip a regular ASCII string somewhere, which will jump at your face when you expect it the least morphed into a stack trace. The data structures' API is the worst and most inconsistent API I've ever seen. Why is 'dict' not 'Dict' ? A comma ending a line actually makes a tuple, that's asking for trouble. I've witnessed more than one bug caused by this. It is slow and resource hungry. Django is too[1]. GlobalInterpreterLock[2]. The package system is horrible, and pip won't save your day either. I love Clojure and Node.js just because of Leiningen/NPM. I couldn't use PyOpenGL in Windows from some unknown reason. I gave up! Discourages functional programming, while encouraging ugly inline list comprehensions. The solution to the half-assed closures in Python2.7[3] was to introduce 'nonloncal'. Lists are mutable. Object Orientation feels clunky to me and using super is awkward. "Batteries included" but core libraries won't take you too far. Unfortunately they're still bloating the language. Concurrency is hard. Twisted solves it a bit but... meh... What I like: - Proper nested comprehensions (CoffeeScript completely blew this one...) - I like the minimal syntax, it helps me visualize the program better. - PyLint/PyFlakes highlights lines that don't conform to PEP8. - Generators. [1] http://www.techempower.com/benchmarks/#section=data-r5 http://www.techempower.com/benchmarks/#section=data-r5 [2] http://ynniv.com/blog/2007/08/closures-in-python.html http://ynniv.com/blog/2007/08/closures-in-python.html (section Caveats) [3] http://wiki.python.org/moin/GlobalInterpreterLock http://wiki.python.org/moin/GlobalInterpreterLock
  - jzwinck 13y ago
    What would you use instead of Python? It should be roughly comparable in terms of having a wide array of well-used libraries, and of course a good REPL. I've been using Python for a couple years and it's mostly joyful. It is unfortunate that not even the various containers within "collections" use a uniform naming convention, I'll give you that. And some of the scoping stuff is weird (nonlocal which nobody really uses? and seemingly-but-not constant default function arguments). But Twisted is a pretty good evented programming system, and NumPy plus multiprocessing let you process a lot of data efficiently, GIL or no. I cannot agree that "the core libraries won't take you too far"--they've taken me farther than the comparable bits in any other language.
dagw 13y ago
This is like discussing the architecture of buildings by listing the principal building materials
- wmblaettler 13y ago
  It's at least a good jumping off point to read more about how to use the materials to build the type of thing you want. I haven't even heard of many of them, but I work in the .NET world these days.
- pietro 13y ago
  That wouldn't be a bad thing, actually. The choice of building material has implications far beyond what you'd imagine.
- Duelist 13y ago
  Agreed. At the very least, it would be helpful to know the specific applications of the languages in projects that use more than one.
davidradcliffe 13y ago
What's with "my-sql". Shouldn't it be "MySQL"?
arturhoo 13y ago
http://highscalability.com/ http://highscalability.com/ is also a great blog for those interested in scaling backend architectures, they have great use cases
hellopat 13y ago
While these aren't so much architectures as they are a list of tools, it's fascinating to see which ones successful companies are using. I like that Trello has such a short list and still runs incredibly fast with a high number of users.
- twic 13y ago
  You say that like picking a small number of decent building blocks and then putting your energy into building high-quality code rather than wrangling a circus of services is a bad way to build scalable systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.
anuragramdasan 13y ago
This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.

98 points by yarapavan 13y ago

Completely dominated by open source projects. This helps my theory that when you release open source code, you are improving the baseline technology for the entire world.

thejosh 13y ago
Yep. And when companies release the products THEY use and find helpful, everyone improves it and releases it back. Memcache springs to mind.
- zimbatm 13y ago
  Not necessarily. I have tons of github repos where nobody contributes to. I don't see why it wouldn't be the same for a company. It depends if your project is useful to others and if you're able to market it properly.
  - lifeisstillgood 13y ago
    Yes, Yes, Yes. OSS needs marketing, and the big ones recently have almost all sprung from world famous companies (twitter bootstrap, facebook and phabricator etc etc). It will become harder and harder to get noticed, but this is truly a open competitive market, and the best becomes top dog in most cases. CEOs who spout support for the Free Market - beware, this is what it looks like. :-)
krenoten 13y ago
There is an extreme case of the availability heuristic at work here. There are a ton of incredibly successful companies that use little or no OSS.

presty 13y ago

architectures != stacks

jaggederest 13y ago
Agreed. This is a list of buzzwords, not a list of architectures. An architecture would be something like 'all major work done via job queues distributed across cheap machines, all pages statically built and pushed to CMS', or 'a dozen bare-metal machines with serious hardware running one-machine sharded app/cache/db slices and hot backups of each shard', fun stuff like that. relatively technology agnostic.
gizzlon 13y ago
Agreed, but it's still a nice list with a lot of information. Check out the first chapter of "The Architecture of Open Source Applications (Volume Two)": "Scalable Web Architecture and Distributed Systems" http://www.aosabook.org/en/distsys.html http://www.aosabook.org/en/distsys.html

camus 13y ago

A a lot of these start-up (except for Facebook maybe ) are using python in their stack, why is that ? all I hear about around me is "Java Java Java". What makes python better ? ( it is just a question ). Or is python just for mvp then the stack moves to Java ( like Twitter did with rails )?

kaoD 13y ago
I'd advise against Python. There are too many caveats in it for a language that's supposed not to surprise me. My biggest gripe: it's easy to shoot yourself in the foot using Unicode. The system is cumbersome and the lack of typing makes it easy to slip a regular ASCII string somewhere, which will jump at your face when you expect it the least morphed into a stack trace. The data structures' API is the worst and most inconsistent API I've ever seen. Why is 'dict' not 'Dict' ? A comma ending a line actually makes a tuple, that's asking for trouble. I've witnessed more than one bug caused by this. It is slow and resource hungry. Django is too[1]. GlobalInterpreterLock[2]. The package system is horrible, and pip won't save your day either. I love Clojure and Node.js just because of Leiningen/NPM. I couldn't use PyOpenGL in Windows from some unknown reason. I gave up! Discourages functional programming, while encouraging ugly inline list comprehensions. The solution to the half-assed closures in Python2.7[3] was to introduce 'nonloncal'. Lists are mutable. Object Orientation feels clunky to me and using super is awkward. "Batteries included" but core libraries won't take you too far. Unfortunately they're still bloating the language. Concurrency is hard. Twisted solves it a bit but... meh... What I like: - Proper nested comprehensions (CoffeeScript completely blew this one...) - I like the minimal syntax, it helps me visualize the program better. - PyLint/PyFlakes highlights lines that don't conform to PEP8. - Generators. [1] http://www.techempower.com/benchmarks/#section=data-r5 http://www.techempower.com/benchmarks/#section=data-r5 [2] http://ynniv.com/blog/2007/08/closures-in-python.html http://ynniv.com/blog/2007/08/closures-in-python.html (section Caveats) [3] http://wiki.python.org/moin/GlobalInterpreterLock http://wiki.python.org/moin/GlobalInterpreterLock
- jzwinck 13y ago
  What would you use instead of Python? It should be roughly comparable in terms of having a wide array of well-used libraries, and of course a good REPL. I've been using Python for a couple years and it's mostly joyful. It is unfortunate that not even the various containers within "collections" use a uniform naming convention, I'll give you that. And some of the scoping stuff is weird (nonlocal which nobody really uses? and seemingly-but-not constant default function arguments). But Twisted is a pretty good evented programming system, and NumPy plus multiprocessing let you process a lot of data efficiently, GIL or no. I cannot agree that "the core libraries won't take you too far"--they've taken me farther than the comparable bits in any other language.

dagw 13y ago

This is like discussing the architecture of buildings by listing the principal building materials

wmblaettler 13y ago
It's at least a good jumping off point to read more about how to use the materials to build the type of thing you want. I haven't even heard of many of them, but I work in the .NET world these days.
pietro 13y ago
That wouldn't be a bad thing, actually. The choice of building material has implications far beyond what you'd imagine.
Duelist 13y ago
Agreed. At the very least, it would be helpful to know the specific applications of the languages in projects that use more than one.

davidradcliffe 13y ago

What's with "my-sql". Shouldn't it be "MySQL"?

arturhoo 13y ago

http://highscalability.com/ http://highscalability.com/ is also a great blog for those interested in scaling backend architectures, they have great use cases

hellopat 13y ago

While these aren't so much architectures as they are a list of tools, it's fascinating to see which ones successful companies are using. I like that Trello has such a short list and still runs incredibly fast with a high number of users.

twic 13y ago
You say that like picking a small number of decent building blocks and then putting your energy into building high-quality code rather than wrangling a circus of services is a bad way to build scalable systems.

anuragramdasan 13y ago

This was disappointing. I came here actually expecting to see something about architecture but ended up looking at product stacks. having said that, I am somewhat glad to see django there. for all the inconsistencies and speed issues that have already mentioned about python and django, it is great to see that the framework still gives pretty good performance on large systems.

anuragramdasan 13y ago