4 ms·

Aaaaaaaand ofc there is nothing about the tools thst are in the main runtime built around distributed systems. Their explanation on tracing show they have neve

by di4na 7y ago

Aaaaaaaand ofc there is nothing about the tools thst are in the main runtime built around distributed systems. Their explanation on tracing show they have never played with an erlang system. As someone that build distributed systems everyday, i would really like all of our industry to play more with production erlang systems. That would answer a lot of these explorations instead of badly reinventing the wheel over and over again.

tempotemporary 7y ago
Any good stuff to read on topic you can recommend?
- di4na 7y ago
  Hmm I would say try to read Erlang in Anger, test it in a toy app, that is not hard to get. https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/ This is also a good entry point https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d9c58547410 https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d...
mcqueenjordan 7y ago
Language/runtime is not the hard part of building distributed systems.
- dnautics 7y ago
  Have you ever used erlang as a distributed system in production?
- neutronicus 7y ago
  I always find myself holding this view ... Until someone bumps an MPI version and I can't fix my God damn build for a week
- di4na 7y ago
  I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible.
vardump 7y ago
The problem is legacy and how codebases have grown in the past. How would you deploy erlang given a system that has grown out of a legacy C++ codebase and needs to run on a mix of mobile devices, IoT embedded systems (some with less than 64 kB RAM), PCs and servers? Some distributed systems are heterogenous and will probably always be. Now, getting back to the main topic, any insights from erlang or otherwise how to troubleshoot said systems that are deployed to customers?
- AzzieElbab 7y ago
  No idea about deployment but I would ask couchdb people. https://en.wikipedia.org/wiki/Apache_CouchDB https://en.wikipedia.org/wiki/Apache_CouchDB There is an embedded version too
- di4na 7y ago
  I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible. Give your operators the ability to explore the system free form with these kind of tracing, dynamically, on production systems. It helps a lot. And yes that will need a lot of work in current runtimes.
h91wka 7y ago
As a systems Erlang designer who worked for 2 of the biggest Erlang companies in the market, I can assure you that Erlang/OTP is not some magic pixie dust that suddenly makes all the distributed systems problems go away. Playing is one thing, making a reliable, fault-tolerant system is another. OTP doesn't give you _state of art_ distributed consensus algorithms out of the box.
- ignoramous 7y ago
  AWS built simple-db with Erlang [0] but then seemingly haven't been using it much ever since... I wouldn't be surprised if simple-db has since been moved to aurora (like document-db [1], neptune, and timestream) or dynamo-db [2] behind the scenes? [0] https://en.wikipedia.org/wiki/Amazon_SimpleDB https://en.wikipedia.org/wiki/Amazon_SimpleDB [1] https://news.ycombinator.com/item?id=18869755 https://news.ycombinator.com/item?id=18869755 [2] https://news.ycombinator.com/item?id=18871854 https://news.ycombinator.com/item?id=18871854
  - rubiquity 7y ago
    Disclaimer: Former Erlanger, currently at AWS but these are disjoint properties. I’ve never worked on SimpleDB or even looked at it, but I think one lesson several database creators learned is that Erlang isn’t high throughout enough for certain classes of database systems. Especially with the last decade’s trend of making potential adoption of a given database as wide as possible and how important benchmarks are for marketing. I still think Erlang is wonderful for writing control plane type software, given the culture around using FSM-like structures.
    - ignoramous 7y ago
      > Erlang isn’t high throughout enough for certain classes of database systems. Interestingly, one of the early simple-db engineers, James Larson [0], later built Megastore [1] at Google, which seems a lot like a retake of simple-db, and a predecessor to Google F1 [2]. I wonder if Megastore uses Erlang. [0] https://www.linkedin.com/in/jilarson https://www.linkedin.com/in/jilarson [1] https://research.google/pubs/pub36971/ https://research.google/pubs/pub36971/ [2] https://research.google/pubs/pub41344/ https://research.google/pubs/pub41344/
namelosw 7y ago
From Erlang's perspective, debugging an average distributed application is just like being primitives hunting in the forest directly after graduating from college. I mean, there's even no parameter in the call stack, people can only guess instead of knowing things for sure. Of course, there are more important things like remote debugging in Erlang. To fix a production issue is like fixing a plane that is malfunctioned and crashing: For average language and runtime, the plan is to take a guess, then develop a theory, build a malfunctioned plane according to the theory. If it's not having same issue, repeat previous steps until people rebuild a plane having exact same issue as the crashing one. Then fix the plane having the same issue, and apply it to the crashing plane. For Erlang, the process would be, examine the malfunctioned plane to see where the problem is for sure, directly patch it in place to make it usable again. And then build a real fix, applying it without restart anything. Tools may not be the most useful things in the world but they definitely could be game-changers sometimes.

by di4na 7y ago

tempotemporary 7y ago

Any good stuff to read on topic you can recommend?

di4na 7y ago
Hmm I would say try to read Erlang in Anger, test it in a toy app, that is not hard to get. https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/ This is also a good entry point https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d9c58547410 https://medium.com/@mrjoelkemp/jvm-struggles-and-the-beam-4d...

mcqueenjordan 7y ago

Language/runtime is not the hard part of building distributed systems.

dnautics 7y ago
Have you ever used erlang as a distributed system in production?
neutronicus 7y ago
I always find myself holding this view ... Until someone bumps an MPI version and I can't fix my God damn build for a week
di4na 7y ago
I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible.

vardump 7y ago

The problem is legacy and how codebases have grown in the past. How would you deploy erlang given a system that has grown out of a legacy C++ codebase and needs to run on a mix of mobile devices, IoT embedded systems (some with less than 64 kB RAM), PCs and servers? Some distributed systems are heterogenous and will probably always be. Now, getting back to the main topic, any insights from erlang or otherwise how to troubleshoot said systems that are deployed to customers?

AzzieElbab 7y ago
No idea about deployment but I would ask couchdb people. https://en.wikipedia.org/wiki/Apache_CouchDB https://en.wikipedia.org/wiki/Apache_CouchDB There is an embedded version too
di4na 7y ago
I meant about the debugging tools that OTP ship with. There is nothing in this about messages or function tracing dynamically. Even less about the kind of introspection you get with it. I totally agree with your points, i am just pointing that it seems they do not seem to know that this kind of tooling are even possible. Give your operators the ability to explore the system free form with these kind of tracing, dynamically, on production systems. It helps a lot. And yes that will need a lot of work in current runtimes.

h91wka 7y ago

As a systems Erlang designer who worked for 2 of the biggest Erlang companies in the market, I can assure you that Erlang/OTP is not some magic pixie dust that suddenly makes all the distributed systems problems go away. Playing is one thing, making a reliable, fault-tolerant system is another. OTP doesn't give you _state of art_ distributed consensus algorithms out of the box.

ignoramous 7y ago
AWS built simple-db with Erlang [0] but then seemingly haven't been using it much ever since... I wouldn't be surprised if simple-db has since been moved to aurora (like document-db [1], neptune, and timestream) or dynamo-db [2] behind the scenes? [0] https://en.wikipedia.org/wiki/Amazon_SimpleDB https://en.wikipedia.org/wiki/Amazon_SimpleDB [1] https://news.ycombinator.com/item?id=18869755 https://news.ycombinator.com/item?id=18869755 [2] https://news.ycombinator.com/item?id=18871854 https://news.ycombinator.com/item?id=18871854
- rubiquity 7y ago
  Disclaimer: Former Erlanger, currently at AWS but these are disjoint properties. I’ve never worked on SimpleDB or even looked at it, but I think one lesson several database creators learned is that Erlang isn’t high throughout enough for certain classes of database systems. Especially with the last decade’s trend of making potential adoption of a given database as wide as possible and how important benchmarks are for marketing. I still think Erlang is wonderful for writing control plane type software, given the culture around using FSM-like structures.
  - ignoramous 7y ago
    > Erlang isn’t high throughout enough for certain classes of database systems. Interestingly, one of the early simple-db engineers, James Larson [0], later built Megastore [1] at Google, which seems a lot like a retake of simple-db, and a predecessor to Google F1 [2]. I wonder if Megastore uses Erlang. [0] https://www.linkedin.com/in/jilarson https://www.linkedin.com/in/jilarson [1] https://research.google/pubs/pub36971/ https://research.google/pubs/pub36971/ [2] https://research.google/pubs/pub41344/ https://research.google/pubs/pub41344/

namelosw 7y ago

From Erlang's perspective, debugging an average distributed application is just like being primitives hunting in the forest directly after graduating from college. I mean, there's even no parameter in the call stack, people can only guess instead of knowing things for sure. Of course, there are more important things like remote debugging in Erlang. To fix a production issue is like fixing a plane that is malfunctioned and crashing: For average language and runtime, the plan is to take a guess, then develop a theory, build a malfunctioned plane according to the theory. If it's not having same issue, repeat previous steps until people rebuild a plane having exact same issue as the crashing one. Then fix the plane having the same issue, and apply it to the crashing plane. For Erlang, the process would be, examine the malfunctioned plane to see where the problem is for sure, directly patch it in place to make it usable again. And then build a real fix, applying it without restart anything. Tools may not be the most useful things in the world but they definitely could be game-changers sometimes.