7 ms·

Everything Is an X

109 points by polm23 6y ago

petethepig 6y ago
When thinking about modern distributed systems one abstraction I love to use is a Queueing System. If you think about it, everything can be described as a system with an input, some buffer (queue), a bunch of workers doing work in parallel, then output: * webserver (e.g nginx) * database server * network interface in your computer * all the networking hardware in between * the whole disk IO stack in your OS * cpu (instructions queue) I wrote about it [1] at some point in more detail if anyone's interested. [1] http://blog.dfilimonov.com/2020/04/24/everything-is-a-queueing-system.html http://blog.dfilimonov.com/2020/04/24/everything-is-a-queuei...
- asplake 6y ago
  Works for human-powered processes too. Everything is a queuing system, and then everything is flow. And again, on its own it takes you only so far...
- choeger 6y ago
  I think you are oversimplifying quite a bit. Looks to me like you are viewing things thought the lens of the typical Microservice architecture. But here are a few more dimensions of complexity: * State between communication. If service (worker/function) A needs to talk to B, A usually needs to keep some state until B answers. Maintaining this state (what if the message gets lost, what if we redeploy A, what if the storage fails) adds a new dimension IMO. * Circular dependencies/recursion. It's astonishing how things like A communicating with A, directly or indirectly seems to be implicitly missing from that typical architecture. * Message growth. What happens when you have an O(n^2) or worse growth in messages? How do you track and manage that? Generally speaking, it looks like microservices behave a lot like the early procedural programming languages by not considering the actually complicated stuff.
  - jstimpfle 6y ago
    I've been thinking a lot about these things lately. I still haven't found solutions to all the problems, but here goes... The alternative to asynchronous interfaces (which is basically what GP describes) are blocking interfaces - like system calls, or just waits for a specific event. And these are never an answer, unless it is _guaranteed_ that the blocking call will return withing a given timeframe and you also know that you will have absolutely nothing else to do meanwhile. > State between communication. There are two ways to keep state - locally on the stack or in an explicit data structure. As always in programming, you have to clean up when you destroy an object/process/stateful thing. > what if the message gets lost This absolutely should not happen, unless the sender doesn't expect an answer and the message can clean up itself. The latter is the case for example when sending a message means just copying it towards the destination, like in computer networks. The former is the case in particular in UDP connections. > what if we redeploy A, what if the storage fails You absolutely need to answer all messages that require an answer (called "IO completion" elsewhere). Of course, the answer can be "cancelled" or "failed". Before destruction or reset, you need to synchronize with all users that hold a direct handle to the object being destroyed. That could just be done by having only a single owner who is responsible to wait for a "cleaned up" event and to then destroy the object. Think Unix processes - processes that exited still appear in the process table until their parent has wait()ed for them. > Message growth. What happens when you have an O(n^2) or worse growth in messages? How do you track and manage that? In general, asynchronous IO is achieved with queues (which are what GP discussed). With queues you have the choice to limit their size right in the queue (if the queue is full, block sending, or reject it temporarily and notify when there is progress). Or you can allow unlimited queues and push responsibility for memory management (and allocation policies) to the users of the queue. For example, users can allocate messages on their own and just link them into the queue - not additional memory allocation needed.
  - layoutIfNeeded 6y ago
    >Generally speaking, it looks like microservices behave a lot like the early procedural programming languages by not considering the actually complicated stuff. +1 In my experience “modern distributed systems” is codeword for handwaving away the complex questions like transactionality and consistency and thinking mostly about the happy path behaviour.
- jstimpfle 6y ago
  Have read your blog post, and I want to finally take the plunge and try Factorio. By the way, another great game here is Opus Magnum.
- emmelaich 6y ago
  You should probably read https://en.wikipedia.org/wiki/Neil_J._Gunther#Universal_Law_of_Computational_Scalability https://en.wikipedia.org/wiki/Neil_J._Gunther#Universal_Law_... and Neil Gunther's other works.
- uyt 6y ago
  I've also seen it described as a log: "The Log: What every software engineer should know about real-time data's unifying abstraction" https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying"[1] https://engineering.linkedin.com/distributed-systems/log-wha... [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
  - whoisthemachine 6y ago
    I enjoyed this article quite a bit, thanks for sharing!
- whoisthemachine 6y ago
  It is a very true abstraction to hold to - add on top of that the ability to replay queues from some type of memory and you have the basics of computer architecture.
l0b0 6y ago
The RDBMS idea of using `insert`/`update`/`delete` to replace DDL is really interesting - it seems like a natural simplification of SQL. Maybe this is a case where the standardization of SQL has actually worked against such innovations? Or are there sound architectural reasons why creating a schema shouldn't be just `insert into meta.(schemas|tables|columns) …`? It would enormously simplify the amount of syntax to remember, at the cost of having to know the relevant meta-schema.
- andreareina 6y ago
  It would also mean creating a table goes from a single statement to a bunch of them.
2sk21 6y ago
And TCL: "everything is a string"
julvo 6y ago
We could add Deep Learning to the list as "everything is a vector". Love how the article can be interpreted as making statements about itself. The article states everything on this list is an implementation of 'everything is an X'. Or put differently, everything is an X, where X = 'everything is an X'.
- clircle 6y ago
  Better yet, in R everything is a vector.
Cloudef 6y ago
Plan9, everything is a file(system). There is no concept of client or server.
janpot 6y ago
Something I noticed digging through the webpack source code: everything is a Webpack plugin.
mfDjB 6y ago
I've noticed when your organisation gets to a certain size, it is inevitable that you will end up working with someone who believes in an "everything is X" fervently enough to do damage. The most common one I've seen is backend developers trying to force frontend to adopt backend paradigms (such as everything is a service, even libraries, and must be communicated to over RPC calls). They tend not to care about the collateral damage or nuances, they want to order the software according to their beliefs. Once you recognise this you can see the same pattern repeating in a lot of other non-software disciplines. I guess it's just a personality type.
- ktpsns 6y ago
  This mentality is also called: "if you have a hammer, every problem is a nail". I guess this paradigm is much older then programming :-)
  - scoot 6y ago
    Close. "If all you have is a hammer, everything looks like a nail." https://en.wiktionary.org/wiki/if_all_you_have_is_a_hammer,_everything_looks_like_a_nail https://en.wiktionary.org/wiki/if_all_you_have_is_a_hammer,_...
- generalizations 6y ago
  Seeems like the problem is when it's misapplied - I don't doubt what you're saying, but it has seemed like *nix got pretty far with the 'everything is a file' paradigm.
  - whatshisface 6y ago
    'nix doesn't actually think everything is a file. At most that's a user/programmer interface thing. Imagine how bad it would be if some insane zealot demanded that everything inside the kernel had to be a file!
    - generalizations 6y ago
moron4hire 6y ago
Sounds like my parents when I was growing up: "everything is an argument". Or in my own house, now that I have kids of my own: "everything is sticky". But (a little) more back on topic, I find a lot of open source software to be: "everything is a missing dependency hunt". Some days, C++ feels like: "everything is undefined behavior".
- donaldihunter 6y ago
  Everything is bloat. Everything is being reinvented again.
- whatshisface 6y ago
  I am so thankful that Rust has Cargo. It is the only place where missing dependency hunts never happen. One day I hope every other language can have something that good.
  - moron4hire 6y ago
    In the .NET world, we have NuGet, which was basically useless until Microsoft finally turned on "restore packages on build" by default about 10 years ago, and now it's absolutely wonderful.
armanboyaci 6y ago
Reminded me this tweet: https://twitter.com/dotMudge/status/739502344073797632 https://twitter.com/dotMudge/status/739502344073797632
asddubs 6y ago
php - everything is an array. and an array is also everything
cercatrova 6y ago
Any non-programming examples?
- godelzilla 6y ago
  Econ: Everything is a market.
realschnabler 6y ago
iOS/UIKit interface development: Everything is a UICollectionView. (UITableView back in the days)
jadar 6y ago
I thought this was going to talk about all the products that just stick "X" on the end of the name.
lambdatronics 6y ago
>If you extended the meta-powers, you could create/delete tables by inserting into/deleting from the “tables table”, and similarly add/remove/alter columns by manipulating the “columns table”. Your schema then becomes data, and your power has become a super-power. I don’t know if any DB has implemented this, or if the advantages would be really compelling over normal DDL. That's interesting. Has anyone heard of such a meta-relational database?
- dragonwriter 6y ago
  > Has anyone heard of such a meta-relational database? Schema-as-data is a central part of the relational model, and is implemented (usually with some limitations) in most RDBMSs. OTOH, DDL manipulation is usually more convenient than DML against catalog tables, even when you can in theory acheive the same result either way.

109 points by polm23 6y ago

When thinking about modern distributed systems one abstraction I love to use is a Queueing System. If you think about it, everything can be described as a system with an input, some buffer (queue), a bunch of workers doing work in parallel, then output: * webserver (e.g nginx) * database server * network interface in your computer * all the networking hardware in between * the whole disk IO stack in your OS * cpu (instructions queue) I wrote about it [1] at some point in more detail if anyone's interested. [1] http://blog.dfilimonov.com/2020/04/24/everything-is-a-queueing-system.html http://blog.dfilimonov.com/2020/04/24/everything-is-a-queuei...

asplake 6y ago
Works for human-powered processes too. Everything is a queuing system, and then everything is flow. And again, on its own it takes you only so far...
choeger 6y ago
I think you are oversimplifying quite a bit. Looks to me like you are viewing things thought the lens of the typical Microservice architecture. But here are a few more dimensions of complexity: * State between communication. If service (worker/function) A needs to talk to B, A usually needs to keep some state until B answers. Maintaining this state (what if the message gets lost, what if we redeploy A, what if the storage fails) adds a new dimension IMO. * Circular dependencies/recursion. It's astonishing how things like A communicating with A, directly or indirectly seems to be implicitly missing from that typical architecture. * Message growth. What happens when you have an O(n^2) or worse growth in messages? How do you track and manage that? Generally speaking, it looks like microservices behave a lot like the early procedural programming languages by not considering the actually complicated stuff.
- jstimpfle 6y ago
  I've been thinking a lot about these things lately. I still haven't found solutions to all the problems, but here goes... The alternative to asynchronous interfaces (which is basically what GP describes) are blocking interfaces - like system calls, or just waits for a specific event. And these are never an answer, unless it is _guaranteed_ that the blocking call will return withing a given timeframe and you also know that you will have absolutely nothing else to do meanwhile. > State between communication. There are two ways to keep state - locally on the stack or in an explicit data structure. As always in programming, you have to clean up when you destroy an object/process/stateful thing. > what if the message gets lost This absolutely should not happen, unless the sender doesn't expect an answer and the message can clean up itself. The latter is the case for example when sending a message means just copying it towards the destination, like in computer networks. The former is the case in particular in UDP connections. > what if we redeploy A, what if the storage fails You absolutely need to answer all messages that require an answer (called "IO completion" elsewhere). Of course, the answer can be "cancelled" or "failed". Before destruction or reset, you need to synchronize with all users that hold a direct handle to the object being destroyed. That could just be done by having only a single owner who is responsible to wait for a "cleaned up" event and to then destroy the object. Think Unix processes - processes that exited still appear in the process table until their parent has wait()ed for them. > Message growth. What happens when you have an O(n^2) or worse growth in messages? How do you track and manage that? In general, asynchronous IO is achieved with queues (which are what GP discussed). With queues you have the choice to limit their size right in the queue (if the queue is full, block sending, or reject it temporarily and notify when there is progress). Or you can allow unlimited queues and push responsibility for memory management (and allocation policies) to the users of the queue. For example, users can allocate messages on their own and just link them into the queue - not additional memory allocation needed.
- layoutIfNeeded 6y ago
  >Generally speaking, it looks like microservices behave a lot like the early procedural programming languages by not considering the actually complicated stuff. +1 In my experience “modern distributed systems” is codeword for handwaving away the complex questions like transactionality and consistency and thinking mostly about the happy path behaviour.
jstimpfle 6y ago
Have read your blog post, and I want to finally take the plunge and try Factorio. By the way, another great game here is Opus Magnum.
emmelaich 6y ago
You should probably read https://en.wikipedia.org/wiki/Neil_J._Gunther#Universal_Law_of_Computational_Scalability https://en.wikipedia.org/wiki/Neil_J._Gunther#Universal_Law_... and Neil Gunther's other works.
uyt 6y ago
I've also seen it described as a log: "The Log: What every software engineer should know about real-time data's unifying abstraction" https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying"[1] https://engineering.linkedin.com/distributed-systems/log-wha... [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- whoisthemachine 6y ago
  I enjoyed this article quite a bit, thanks for sharing!
whoisthemachine 6y ago
It is a very true abstraction to hold to - add on top of that the ability to replay queues from some type of memory and you have the basics of computer architecture.

l0b0 6y ago

The RDBMS idea of using `insert`/`update`/`delete` to replace DDL is really interesting - it seems like a natural simplification of SQL. Maybe this is a case where the standardization of SQL has actually worked against such innovations? Or are there sound architectural reasons why creating a schema shouldn't be just `insert into meta.(schemas|tables|columns) …`? It would enormously simplify the amount of syntax to remember, at the cost of having to know the relevant meta-schema.

andreareina 6y ago
It would also mean creating a table goes from a single statement to a bunch of them.

2sk21 6y ago

And TCL: "everything is a string"

julvo 6y ago

We could add Deep Learning to the list as "everything is a vector". Love how the article can be interpreted as making statements about itself. The article states everything on this list is an implementation of 'everything is an X'. Or put differently, everything is an X, where X = 'everything is an X'.

clircle 6y ago
Better yet, in R everything is a vector.

Cloudef 6y ago

Plan9, everything is a file(system). There is no concept of client or server.

janpot 6y ago

Something I noticed digging through the webpack source code: everything is a Webpack plugin.

mfDjB 6y ago

I've noticed when your organisation gets to a certain size, it is inevitable that you will end up working with someone who believes in an "everything is X" fervently enough to do damage. The most common one I've seen is backend developers trying to force frontend to adopt backend paradigms (such as everything is a service, even libraries, and must be communicated to over RPC calls). They tend not to care about the collateral damage or nuances, they want to order the software according to their beliefs. Once you recognise this you can see the same pattern repeating in a lot of other non-software disciplines. I guess it's just a personality type.

ktpsns 6y ago
This mentality is also called: "if you have a hammer, every problem is a nail". I guess this paradigm is much older then programming :-)
- scoot 6y ago
  Close. "If all you have is a hammer, everything looks like a nail." https://en.wiktionary.org/wiki/if_all_you_have_is_a_hammer,_everything_looks_like_a_nail https://en.wiktionary.org/wiki/if_all_you_have_is_a_hammer,_...
generalizations 6y ago
Seeems like the problem is when it's misapplied - I don't doubt what you're saying, but it has seemed like *nix got pretty far with the 'everything is a file' paradigm.
- whatshisface 6y ago
  'nix doesn't actually think everything is a file. At most that's a user/programmer interface thing. Imagine how bad it would be if some insane zealot demanded that everything inside the kernel had to be a file!
  - generalizations 6y ago

moron4hire 6y ago

Sounds like my parents when I was growing up: "everything is an argument". Or in my own house, now that I have kids of my own: "everything is sticky". But (a little) more back on topic, I find a lot of open source software to be: "everything is a missing dependency hunt". Some days, C++ feels like: "everything is undefined behavior".

donaldihunter 6y ago
Everything is bloat. Everything is being reinvented again.
whatshisface 6y ago
I am so thankful that Rust has Cargo. It is the only place where missing dependency hunts never happen. One day I hope every other language can have something that good.
- moron4hire 6y ago
  In the .NET world, we have NuGet, which was basically useless until Microsoft finally turned on "restore packages on build" by default about 10 years ago, and now it's absolutely wonderful.

armanboyaci 6y ago

Reminded me this tweet: https://twitter.com/dotMudge/status/739502344073797632 https://twitter.com/dotMudge/status/739502344073797632

asddubs 6y ago

php - everything is an array. and an array is also everything

cercatrova 6y ago

Any non-programming examples?

godelzilla 6y ago
Econ: Everything is a market.

realschnabler 6y ago

iOS/UIKit interface development: Everything is a UICollectionView. (UITableView back in the days)

jadar 6y ago

I thought this was going to talk about all the products that just stick "X" on the end of the name.

lambdatronics 6y ago

>If you extended the meta-powers, you could create/delete tables by inserting into/deleting from the “tables table”, and similarly add/remove/alter columns by manipulating the “columns table”. Your schema then becomes data, and your power has become a super-power. I don’t know if any DB has implemented this, or if the advantages would be really compelling over normal DDL. That's interesting. Has anyone heard of such a meta-relational database?

dragonwriter 6y ago
> Has anyone heard of such a meta-relational database? Schema-as-data is a central part of the relational model, and is implemented (usually with some limitations) in most RDBMSs. OTOH, DDL manipulation is usually more convenient than DML against catalog tables, even when you can in theory acheive the same result either way.