6 ms·

Diagram as Code for prototyping cloud system architectures

129 points by simplecto 7y ago

ivan_ah 7y ago
That's really cool to have the charts generated as code. Very concise way to specify things, and with all the power of graphviz behind the scenes. Another tool in that space is https://cloudcraft.co/ https://cloudcraft.co/ although only for AWS, and you have to manually do the diagram layout.
gugagore 7y ago
"as code" because it's version controlled? Is it also executable in any sense?
- simplecto 7y ago
  No, this just generates diagrams from code. I think it is a pretty neat idea for those who don't like drawo.io or Visio style charts.
  - jessaustin 7y ago
    The code is really just a pythonized version of the dot format. The real innovation is the fancy-looking icons. When using graphviz directly we have to futz around with images in the filesystem alongside the dot file, in order to avoid simple boxes.
    - elephantum 7y ago
      > The code is really just a pythonized version of the dot format. Not exactly. By utilizing Python as a base language you gain access to control structures as `for` loops, `if`-s and functions. It can make your diagram script much more powerful and/or more complicated.
- aliswe 7y ago
  I think because it's built imperatively?
makstaks 7y ago
This is a great idea. I'm not sure about the other major cloud providers, but AWS has the AWS CDK which allows you to use TypeScript, Python, Java, and .NET to create your cloud infrastructure via code. The ability to generate cloud diagrams from AWS CDK code would definitely would be awesome. Hopefully, your work gains transaction, then I can envision other devs writing adapter code for your project.
- andreineculau 7y ago
  I have a script that we call when generating cloudformation templates to generate a dot graph Cfn2dot source https://github.com/tobiipro/support-firecloud/blob/master/bin/aws-cfn2dot https://github.com/tobiipro/support-firecloud/blob/master/bi... Used like this https://github.com/tobiipro/support-firecloud/blob/master/repo/cfn/inc.mk https://github.com/tobiipro/support-firecloud/blob/master/re... One problem is that you need meta-data to give weights to different resources so that you highlight a lambda resource and not there associated cloudwatch loggroup resource, and its iam role, and iam policy, and its permission resource, etc The other problem is the graph itself shows how the cloudformation resources depend on one another, not how your data/execution pipeline flows. So you need to do it manually or maybe try to add lots of meta-data to the types of resources.
data_ders 7y ago
o man I was just looking for a way to not pay for LucidChart. I also refuse to use PowerPoint
- thedirt0115 7y ago
  You could try OpenOffice/LibreOffice Impress.
- wkrause 7y ago
  draw.io is a solid option as well
mr_tristan 7y ago
I've used asciidoctor with it's diagram plugin (https://asciidoctor.org/docs/asciidoctor-diagram/ https://asciidoctor.org/docs/asciidoctor-diagram/) internally on memos/draft documentation, which that gives us an ability to actually just check in the asciidoc with diagram source into git. Then we use merge requests for a real review process. Then, I just usually just export to pdf or word for broader sharing among the masses. (Amazing how this usually gets far more attention than Confluence, etc) Main benefit is that most of these kinds of CLI diagram tools have much longer lives than any GUI-based diagram software I've used. But even just having a separate "diagram source" file where you just check in an image is still nice. Then you don't have to bother with wondering who has access to the tools, etc.
spectramax 7y ago
The entire contemporary devops process is awfully convoluted. In fact, I've heard DevOps people call themselves "yaml engineers". Often in your docker compose there is some stupid error, there are 3 versions of Docker compose files, there are 7 ways to do something, Docker CLI manual can be printed in 1000 pages and we've created extremely bloated (despite of caching whatever) CI/CD. For what? Dependencies? Let's download 4.5 GB of images because my app needs to be reproduceable. I tried reading about Kubernetes. Holyshit what a monster we've created. I am sure there is going to be an orchestrator abstraction for multiple Kubernetes clusters. And then we have environment variables, secrets, access keys, a bazillion different parameters that need to trickle down from Gitlab CI settings all the way to the app through many layers. Good lord. All engineers should be ashamed of this mess. We've forgotten how to build simple, beautiful, lean and parsimonious software. Instead of writing logic, I am configurating things all day. Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers. But, let's shit on the F-35 program everytime there is a Lockheed article on HN. Because surely, software engineers are the smart bunch ya. I know, I am unreasonably obtuse here, but jeez, no one complains about today's deployment process,... its apparently accepted without a question.
- rraghur 7y ago
  > Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers I'm with you on the feeling that modern stack is bloated and heavy... However, this is exactly the scenario where the compiler bug is isolated quickly because you're not chasing dependency differences between boxes
  - spectramax 7y ago
    No! What if there is a bug in Kubernetes itself? or in the Docker engine? Or a compiler bug that affects Kubernetes in mysterious ways?
    - 7y ago
jcpst 7y ago
DOT notation is simple, and works great for generating diagrams.
wiradikusuma 7y ago
Finally related topic! I've been meaning to ask, is there any tool (or a suite of tools) that can be used to visualize network along with real-time health status of each nodes? Imagine you have a web server, API server and DB server, lined up left to right, each connected with a line. Each of them has green dot, and when they're down, the dot changes color to red. The diagram can be manually laid out, but the "health indicator" must be updated automatically.
- feisuzhu 7y ago
  `kiali` maybe the thing you want. But it should be used with Istio stuffs.
- ygra 7y ago
  We have customers doing this with yFiles. But that's a generic graph drawing and layout library, so getting from your network topology to a graph structure and getting health events is not something the library helps with. You'd get automatic layout, however, and colored dots are on the trivial side for a visualization.
alexott 7y ago
It’s interesting that PlantUML wasn’t mentioned... there are cloud icons for it as well
- ygra 7y ago
  The icons are Amazon's own ones. You can use them with any tool.
polskibus 7y ago
I wish it was an extension to plantuml instead.
gregwebs 7y ago
I started doing diagrams from code and don't want to go back to dragging things with a mouse again. VSCode plantuml gives live feedback. I started using this for c4 diagrams https://github.com/RicardoNiepel/C4-PlantUML https://github.com/RicardoNiepel/C4-PlantUML
koobz 7y ago
Another one if you like dot-esque notation and pretty output out the gate is https://mermaid-js.github.io/mermaid/#/ https://mermaid-js.github.io/mermaid/#/.

129 points by simplecto 7y ago

That's really cool to have the charts generated as code. Very concise way to specify things, and with all the power of graphviz behind the scenes. Another tool in that space is https://cloudcraft.co/ https://cloudcraft.co/ although only for AWS, and you have to manually do the diagram layout.

gugagore 7y ago

"as code" because it's version controlled? Is it also executable in any sense?

simplecto 7y ago
No, this just generates diagrams from code. I think it is a pretty neat idea for those who don't like drawo.io or Visio style charts.
- jessaustin 7y ago
  The code is really just a pythonized version of the dot format. The real innovation is the fancy-looking icons. When using graphviz directly we have to futz around with images in the filesystem alongside the dot file, in order to avoid simple boxes.
  - elephantum 7y ago
    > The code is really just a pythonized version of the dot format. Not exactly. By utilizing Python as a base language you gain access to control structures as `for` loops, `if`-s and functions. It can make your diagram script much more powerful and/or more complicated.
aliswe 7y ago
I think because it's built imperatively?

makstaks 7y ago

This is a great idea. I'm not sure about the other major cloud providers, but AWS has the AWS CDK which allows you to use TypeScript, Python, Java, and .NET to create your cloud infrastructure via code. The ability to generate cloud diagrams from AWS CDK code would definitely would be awesome. Hopefully, your work gains transaction, then I can envision other devs writing adapter code for your project.

andreineculau 7y ago
I have a script that we call when generating cloudformation templates to generate a dot graph Cfn2dot source https://github.com/tobiipro/support-firecloud/blob/master/bin/aws-cfn2dot https://github.com/tobiipro/support-firecloud/blob/master/bi... Used like this https://github.com/tobiipro/support-firecloud/blob/master/repo/cfn/inc.mk https://github.com/tobiipro/support-firecloud/blob/master/re... One problem is that you need meta-data to give weights to different resources so that you highlight a lambda resource and not there associated cloudwatch loggroup resource, and its iam role, and iam policy, and its permission resource, etc The other problem is the graph itself shows how the cloudformation resources depend on one another, not how your data/execution pipeline flows. So you need to do it manually or maybe try to add lots of meta-data to the types of resources.

data_ders 7y ago

o man I was just looking for a way to not pay for LucidChart. I also refuse to use PowerPoint

thedirt0115 7y ago
You could try OpenOffice/LibreOffice Impress.
wkrause 7y ago
draw.io is a solid option as well

mr_tristan 7y ago

I've used asciidoctor with it's diagram plugin (https://asciidoctor.org/docs/asciidoctor-diagram/ https://asciidoctor.org/docs/asciidoctor-diagram/) internally on memos/draft documentation, which that gives us an ability to actually just check in the asciidoc with diagram source into git. Then we use merge requests for a real review process. Then, I just usually just export to pdf or word for broader sharing among the masses. (Amazing how this usually gets far more attention than Confluence, etc) Main benefit is that most of these kinds of CLI diagram tools have much longer lives than any GUI-based diagram software I've used. But even just having a separate "diagram source" file where you just check in an image is still nice. Then you don't have to bother with wondering who has access to the tools, etc.

spectramax 7y ago

The entire contemporary devops process is awfully convoluted. In fact, I've heard DevOps people call themselves "yaml engineers". Often in your docker compose there is some stupid error, there are 3 versions of Docker compose files, there are 7 ways to do something, Docker CLI manual can be printed in 1000 pages and we've created extremely bloated (despite of caching whatever) CI/CD. For what? Dependencies? Let's download 4.5 GB of images because my app needs to be reproduceable. I tried reading about Kubernetes. Holyshit what a monster we've created. I am sure there is going to be an orchestrator abstraction for multiple Kubernetes clusters. And then we have environment variables, secrets, access keys, a bazillion different parameters that need to trickle down from Gitlab CI settings all the way to the app through many layers. Good lord. All engineers should be ashamed of this mess. We've forgotten how to build simple, beautiful, lean and parsimonious software. Instead of writing logic, I am configurating things all day. Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers. But, let's shit on the F-35 program everytime there is a Lockheed article on HN. Because surely, software engineers are the smart bunch ya. I know, I am unreasonably obtuse here, but jeez, no one complains about today's deployment process,... its apparently accepted without a question.

rraghur 7y ago
> Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers I'm with you on the feeling that modern stack is bloated and heavy... However, this is exactly the scenario where the compiler bug is isolated quickly because you're not chasing dependency differences between boxes
- spectramax 7y ago
  No! What if there is a bug in Kubernetes itself? or in the Docker engine? Or a compiler bug that affects Kubernetes in mysterious ways?
  - 7y ago

jcpst 7y ago

DOT notation is simple, and works great for generating diagrams.

wiradikusuma 7y ago

Finally related topic! I've been meaning to ask, is there any tool (or a suite of tools) that can be used to visualize network along with real-time health status of each nodes? Imagine you have a web server, API server and DB server, lined up left to right, each connected with a line. Each of them has green dot, and when they're down, the dot changes color to red. The diagram can be manually laid out, but the "health indicator" must be updated automatically.

feisuzhu 7y ago
`kiali` maybe the thing you want. But it should be used with Istio stuffs.
ygra 7y ago
We have customers doing this with yFiles. But that's a generic graph drawing and layout library, so getting from your network topology to a graph structure and getting health events is not something the library helps with. You'd get automatic layout, however, and colored dots are on the trivial side for a visualization.

alexott 7y ago

It’s interesting that PlantUML wasn’t mentioned... there are cloud icons for it as well

ygra 7y ago
The icons are Amazon's own ones. You can use them with any tool.

polskibus 7y ago

I wish it was an extension to plantuml instead.

gregwebs 7y ago

I started doing diagrams from code and don't want to go back to dragging things with a mouse again. VSCode plantuml gives live feedback. I started using this for c4 diagrams https://github.com/RicardoNiepel/C4-PlantUML https://github.com/RicardoNiepel/C4-PlantUML

koobz 7y ago

Another one if you like dot-esque notation and pretty output out the gate is https://mermaid-js.github.io/mermaid/#/ https://mermaid-js.github.io/mermaid/#/.