5 ms·

I use Graphviz dot syntax to configure entire cloud computing environments in the order they must be built. This is pipelines as code and infrastructure as cod

I use Graphviz dot syntax to configure entire cloud computing environments in the order they must be built. This is pipelines as code and infrastructure as code It's very powerful! Not every organisation can bring up an entire environment with one command of documentation their infrastructure with Graphviz. Here is a executable diagram of a build worker cloud nodes, Kubernetes, consul, vault, Debian package server, Java app, SSH bastion, Prometheus, grafana, DNS and security groups. https://github.com/samsquire/mazzle/blob/master/docs/architecture.png https://github.com/samsquire/mazzle/blob/master/docs/archite... The tool also parallelises the build based on the graph so for example packer builds can run in parallel. https://devops-pipeline.com/ https://devops-pipeline.com/ It also has a GUI but as a tool it's not ready for other people to use.

fsiefken 4y ago
o wow! i know that graphana can be used with mermaid diagrams, but this is a whole new (lower) level. it would be nice if the graphviz could be styled a bit nices (like with sketchviz), but that's icing on the cake.
jeffparsons 4y ago
Holy macaroni, this is one of the coolest things I've seen in a while. I wonder if a top-level post of this could generate some interesting discussions about other people's exotic approaches to "DevOps", too?
- samsquire 4y ago
  Unfortunately I tried promoting my things too much so most of my submissions are blocked by the self promotion filter. Yes I would love to hear other people's solutions to DevOps problems. I find it an interesting space. There's lots of good ideas but they haven't been brought together.
DanielBMarkham 4y ago
Very cool. I created an analysis compiler that takes intent information and makes it part of the Enterprise DevSecOps pipeline in order to share realtime analysis with the entire organization. As I continued that work, I've also started talking about the feasibility of creating a supercompiler. GraphViz is the shit, man. It was exactly what I needed to work on this material. You can create your own notation and then couple it with all kinds of tech by way of batch. Highly recommended.* *The hardest part, frankly, was unlearning all of the other diagramming paradigms I've used. YMMV, but I found GraphViz most useful when you're going completely greenfield.
- samsquire 4y ago
  I love these words "As I continued that work, I've also started talking about the feasibility of creating a supercompiler." and thoughts already. One of my ideas is a supercomputer compiler I wrote about it here https://github.com/samsquire/ideas4#112-compiler-as-a-service-supercomputer-compiler https://github.com/samsquire/ideas4#112-compiler-as-a-servic... Application performance monitoring, scheduling, control loops ala Kubernetes, parallelism, multithreading, data scheduling (shifting and sharing data to where it needs to be used), advanced caching and regeneration and container scheduling and binpacking and data layout scheduling could all be compiled by a supercompiler for extreme performance.
  - DanielBMarkham 4y ago
    We are very much on the same page. I wrote about it here https://danielbmarkham.com/outlines-of-a-supercompiler-in-f/ https://danielbmarkham.com/outlines-of-a-supercompiler-in-f/ Our industry tends to over-complicate things and get in its own way of making the world a better place. SuperCompilers are a worthwhile effort to fix that. We need more forces for good. Kudos.
- HyperSane 4y ago
  "an analysis compiler that takes intent information" What does this mean?
  - DanielBMarkham 4y ago
    1. Whenever anyone in the organization has an interesting conversation about the product they take notes. They're just plaintext notes using notepad or whatever. 2. There are about 20 keywords. They are represented as ALL CAPS 3. When you take notes, you tag them with the keywords 4. These notes are checked into git like everything else, and built as part of the dev/team/pmo/enterprise DevSecOps pipeline 5. Doing this creates a realtime and ever-changing analysis model This leads to better conversations, better questions, elimination of duplication and conflicting business ideas, entry of data once, tracking bugs and features in plain language all of the stakeholders can understand, automated testing frameworks, interactive diagrams showing what the business thinks the world looks like and how they're products will change it, and more. It was fun trying to design the simplest system possible able to map the almost endless complexity of human interactions around a given domain. tl;dr "Intent" is everything that happens between humans to organize their view of the world and the tech they're making to interact with it. Intent begins with brainstorming, perhaps BS sessions with coders, marketers, customers, and so forth and ends with feature tests, user manuals, and everything but the code itself.
bitexploder 4y ago
I am big on state machines. Even simple tooling around them makes life easier for a lot of tasks. We often poorly implement a state machine in our code. Formalizing it in some way often makes life easier for me.
jordanbeiber 4y ago
I’ve kind of considered something similar to this approach but ended up using a graphdb to keep dependencies instead. Using a db gives you a better way of modifying meta data in batches which might be required at scale. It also makes it possible for tooling to query stuff quickly and do operations such as setting consul intentions, register something in an api gateway or other surrounding config tasks. The pipelines we built checked out complete subgraphs from an api and generated json that referenced versioned TF modules and committed these to a service repo. Thought about open-sourcing for a long time but never could get the go from legal. :( This is very cool though!

fsiefken 4y ago

o wow! i know that graphana can be used with mermaid diagrams, but this is a whole new (lower) level. it would be nice if the graphviz could be styled a bit nices (like with sketchviz), but that's icing on the cake.

jeffparsons 4y ago

Holy macaroni, this is one of the coolest things I've seen in a while. I wonder if a top-level post of this could generate some interesting discussions about other people's exotic approaches to "DevOps", too?

samsquire 4y ago
Unfortunately I tried promoting my things too much so most of my submissions are blocked by the self promotion filter. Yes I would love to hear other people's solutions to DevOps problems. I find it an interesting space. There's lots of good ideas but they haven't been brought together.

DanielBMarkham 4y ago

Very cool. I created an analysis compiler that takes intent information and makes it part of the Enterprise DevSecOps pipeline in order to share realtime analysis with the entire organization. As I continued that work, I've also started talking about the feasibility of creating a supercompiler. GraphViz is the shit, man. It was exactly what I needed to work on this material. You can create your own notation and then couple it with all kinds of tech by way of batch. Highly recommended.* *The hardest part, frankly, was unlearning all of the other diagramming paradigms I've used. YMMV, but I found GraphViz most useful when you're going completely greenfield.

samsquire 4y ago
I love these words "As I continued that work, I've also started talking about the feasibility of creating a supercompiler." and thoughts already. One of my ideas is a supercomputer compiler I wrote about it here https://github.com/samsquire/ideas4#112-compiler-as-a-service-supercomputer-compiler https://github.com/samsquire/ideas4#112-compiler-as-a-servic... Application performance monitoring, scheduling, control loops ala Kubernetes, parallelism, multithreading, data scheduling (shifting and sharing data to where it needs to be used), advanced caching and regeneration and container scheduling and binpacking and data layout scheduling could all be compiled by a supercompiler for extreme performance.
- DanielBMarkham 4y ago
  We are very much on the same page. I wrote about it here https://danielbmarkham.com/outlines-of-a-supercompiler-in-f/ https://danielbmarkham.com/outlines-of-a-supercompiler-in-f/ Our industry tends to over-complicate things and get in its own way of making the world a better place. SuperCompilers are a worthwhile effort to fix that. We need more forces for good. Kudos.
HyperSane 4y ago
"an analysis compiler that takes intent information" What does this mean?
- DanielBMarkham 4y ago
  1. Whenever anyone in the organization has an interesting conversation about the product they take notes. They're just plaintext notes using notepad or whatever. 2. There are about 20 keywords. They are represented as ALL CAPS 3. When you take notes, you tag them with the keywords 4. These notes are checked into git like everything else, and built as part of the dev/team/pmo/enterprise DevSecOps pipeline 5. Doing this creates a realtime and ever-changing analysis model This leads to better conversations, better questions, elimination of duplication and conflicting business ideas, entry of data once, tracking bugs and features in plain language all of the stakeholders can understand, automated testing frameworks, interactive diagrams showing what the business thinks the world looks like and how they're products will change it, and more. It was fun trying to design the simplest system possible able to map the almost endless complexity of human interactions around a given domain. tl;dr "Intent" is everything that happens between humans to organize their view of the world and the tech they're making to interact with it. Intent begins with brainstorming, perhaps BS sessions with coders, marketers, customers, and so forth and ends with feature tests, user manuals, and everything but the code itself.

bitexploder 4y ago

I am big on state machines. Even simple tooling around them makes life easier for a lot of tasks. We often poorly implement a state machine in our code. Formalizing it in some way often makes life easier for me.

jordanbeiber 4y ago

I’ve kind of considered something similar to this approach but ended up using a graphdb to keep dependencies instead. Using a db gives you a better way of modifying meta data in batches which might be required at scale. It also makes it possible for tooling to query stuff quickly and do operations such as setting consul intentions, register something in an api gateway or other surrounding config tasks. The pipelines we built checked out complete subgraphs from an api and generated json that referenced versioned TF modules and committed these to a service repo. Thought about open-sourcing for a long time but never could get the go from legal. :( This is very cool though!