3 ms·

This reminds me of a certain architect at my last shop who invented a DSL on top of his Python superapp. He expected all projects to go through his superapp.

This reminds me of a certain architect at my last shop who invented a DSL on top of his Python superapp. He expected all projects to go through his superapp. The DSL was configured in YAML. The YAML was often so dense he recommended devs use Jinja to generate the YAML. This meant debug was hell, plus it wasn't always clear if what you were trying to do was even supported / if not why & what needed to be changed. This was because you were now 3 levels of abstraction away from the Python code that was actually executing. Every time a dev took on a new project they had to jump on a call with architect or right hand man to figure out if what they were trying to do was going to be possible. It escalated into the architect demanding to know a sprint in advance any task devs were trying to do, in a review session, so he could explain if it was possible or not and try to triage in his DSL..

goodoldneon 4y ago
I would quit within a month
- steveBK123 4y ago
  Some did :-D
singingfish 4y ago
I guess YAML has a place in that it would prevent that kind of thing happening in the first place. YAML is easy to debug (thanks to having comment syntax) because it just deserialises into code. Sometimes it deserialises into code that compiles on the fly mind you which is never a good idea. On the other hand one time I debugged a really nasty memory leak by dumping many megabytes of YAML then running git diff against the dumps. That was fun. Of course the client used the quick and bad hack rather than the demonstrably correct fix (thanks to the dumps) because they were frightened of their own code.
rollcat 4y ago
I've read about places that do this kind of stuff. Although it sounds like pure hell, I'm sure there's always a reasonable explanation, an intent. What kinds of problems was the org facing that led to the development of this?
- steveBK123 4y ago
  It wasn't really necessary. It also made the core superapp a blocker of essentially every user delivery. If it has to do everything, then you have to add a lot of features to it. So you have N devs doing jinja/yaml/dsl and N/4 doing core superapp underlying dev. For the first Z features/projects, you inevitably see new things that your superapp doesn't support yet, and becomes emergent blockers midway through implementation. Given the ratio of devs, new blockers were being generated faster than they could be cleared. Eventually business side pulled the fire alarm and grabbed most of the devs over to use a more common AWS-centric service directly and exit the superapp dev use completely. IE - if you are going to depend on something, would you rather it be an AWS service with 10s-100s of dev-years behind it, or some internal superapp spun up 3 months ago with 2 guys on it? Which is more likely to already support what you need?
formerly_proven 4y ago
At work someone is trying to introduce a system where a bunch of Jinja templates in a repository are used to generate XML which can then be used to generate another XML document which can then be "executed", resulting in an annotated XML document :)
- steveBK123 4y ago
  Wow that's like 2000s & 2020s mutant baby
  - formerly_proven 4y ago
    And the context variables for the jinja is in part read from a YAML file iirc
duxup 4y ago
That sounds like a layer of insanity that would make me consider jobs elsewhere. It sounds entirely unnecessary and burdensome , but was it unnecessary?
- steveBK123 4y ago
  It was unnecessary because he was being too clever & having a good time, versus ever having delivered real production systems in our industry. It also put devs on a dead end path which they realized pretty quickly. Do you want to work for years on this team becoming experts in jinja to yaml to in-house DSL you'll never use anywhere else? Or do you want to write some python? If you can't get "promoted" into the team writing the core python engine, then you are obviously a second rate.. why stay?
  - duxup 4y ago
    Even beyond that I would worry about the decision making at the top that… they thought that was a good idea and kept at it. At that point a lot of potentially absurd things could be in play.
    - steveBK123 4y ago
      Less technical management hires a hero who tells them everything they want to hear! "I'm going to deliver the superapp, everything will be super centralized & tidy.. small dev team, then all the specific implementations will be grunt work by cheap devs!" Throw in some buzzwords and they are sold. Same audience that always signs the checks for no code/low code stuff no one actually wants.
ilyt 4y ago
>The DSL was configured in YAML. The YAML was often so dense he recommended devs use Jinja to generate the YAML. Did he then went on to design Ansible ? It falls into same trap Only way you should be generating data format using language's templating system is <%= YAML.dump(@config) %> Also 9 times out of 10 I wished the app designer just used <app language> or <any common embeddable language> (like Lua) instead of making any kind of DSL (whether that's just data file pretending to be code or micro programming language)
- steveBK123 4y ago
  Yes. I think this is like the uncanny valley of development. It's not no-code UI driven stuff you can put in front of a business user. It's not real coding, which an engineer wants to do. It's config jockeying, which devs find boring, and is generally far more limited. So you end up building out more and more complex layers of config to work around the limits, including scripts to generate config.. etc. Seems like what you really want is modular apps that are easy to extend in the native programming language(s). Also maybe I'm just stupid, but 9 out of 10 times, a text file or csv/demitted file accomplishes most of what you need for pure config that really belongs in config.
  - gpderetta 4y ago
    Tell me! I'm currently attempting to resist an upcoming transition form .csv to yaml... ... which is used to generate .xml.
ljm 4y ago
This is basically how I feel about working with K8S and dredging through a repo full of templated YAML spaghetti. What am I looking at now? Helm, Keda, Flux, Argo, OperatorHub, GitHub Actions? oh actually this bit is in Terraform in another folder, whoops. You can’t actually deploy something unless you can mentally untangle it all, it just sits in front of your infra as a sort of DevOps Coming of Age ritual, where you look whistfully over your shoulder at the old Heroku or Vercel account you grew up with. Simpler times.

goodoldneon 4y ago

I would quit within a month

steveBK123 4y ago
Some did :-D

singingfish 4y ago

I guess YAML has a place in that it would prevent that kind of thing happening in the first place. YAML is easy to debug (thanks to having comment syntax) because it just deserialises into code. Sometimes it deserialises into code that compiles on the fly mind you which is never a good idea. On the other hand one time I debugged a really nasty memory leak by dumping many megabytes of YAML then running git diff against the dumps. That was fun. Of course the client used the quick and bad hack rather than the demonstrably correct fix (thanks to the dumps) because they were frightened of their own code.

rollcat 4y ago

I've read about places that do this kind of stuff. Although it sounds like pure hell, I'm sure there's always a reasonable explanation, an intent. What kinds of problems was the org facing that led to the development of this?

steveBK123 4y ago
It wasn't really necessary. It also made the core superapp a blocker of essentially every user delivery. If it has to do everything, then you have to add a lot of features to it. So you have N devs doing jinja/yaml/dsl and N/4 doing core superapp underlying dev. For the first Z features/projects, you inevitably see new things that your superapp doesn't support yet, and becomes emergent blockers midway through implementation. Given the ratio of devs, new blockers were being generated faster than they could be cleared. Eventually business side pulled the fire alarm and grabbed most of the devs over to use a more common AWS-centric service directly and exit the superapp dev use completely. IE - if you are going to depend on something, would you rather it be an AWS service with 10s-100s of dev-years behind it, or some internal superapp spun up 3 months ago with 2 guys on it? Which is more likely to already support what you need?

formerly_proven 4y ago

At work someone is trying to introduce a system where a bunch of Jinja templates in a repository are used to generate XML which can then be used to generate another XML document which can then be "executed", resulting in an annotated XML document :)

steveBK123 4y ago
Wow that's like 2000s & 2020s mutant baby
- formerly_proven 4y ago
  And the context variables for the jinja is in part read from a YAML file iirc

duxup 4y ago

That sounds like a layer of insanity that would make me consider jobs elsewhere. It sounds entirely unnecessary and burdensome , but was it unnecessary?

steveBK123 4y ago
It was unnecessary because he was being too clever & having a good time, versus ever having delivered real production systems in our industry. It also put devs on a dead end path which they realized pretty quickly. Do you want to work for years on this team becoming experts in jinja to yaml to in-house DSL you'll never use anywhere else? Or do you want to write some python? If you can't get "promoted" into the team writing the core python engine, then you are obviously a second rate.. why stay?
- duxup 4y ago
  Even beyond that I would worry about the decision making at the top that… they thought that was a good idea and kept at it. At that point a lot of potentially absurd things could be in play.
  - steveBK123 4y ago
    Less technical management hires a hero who tells them everything they want to hear! "I'm going to deliver the superapp, everything will be super centralized & tidy.. small dev team, then all the specific implementations will be grunt work by cheap devs!" Throw in some buzzwords and they are sold. Same audience that always signs the checks for no code/low code stuff no one actually wants.

ilyt 4y ago

>The DSL was configured in YAML. The YAML was often so dense he recommended devs use Jinja to generate the YAML. Did he then went on to design Ansible ? It falls into same trap Only way you should be generating data format using language's templating system is <%= YAML.dump(@config) %> Also 9 times out of 10 I wished the app designer just used <app language> or <any common embeddable language> (like Lua) instead of making any kind of DSL (whether that's just data file pretending to be code or micro programming language)

steveBK123 4y ago
Yes. I think this is like the uncanny valley of development. It's not no-code UI driven stuff you can put in front of a business user. It's not real coding, which an engineer wants to do. It's config jockeying, which devs find boring, and is generally far more limited. So you end up building out more and more complex layers of config to work around the limits, including scripts to generate config.. etc. Seems like what you really want is modular apps that are easy to extend in the native programming language(s). Also maybe I'm just stupid, but 9 out of 10 times, a text file or csv/demitted file accomplishes most of what you need for pure config that really belongs in config.
- gpderetta 4y ago
  Tell me! I'm currently attempting to resist an upcoming transition form .csv to yaml... ... which is used to generate .xml.

ljm 4y ago

This is basically how I feel about working with K8S and dredging through a repo full of templated YAML spaghetti. What am I looking at now? Helm, Keda, Flux, Argo, OperatorHub, GitHub Actions? oh actually this bit is in Terraform in another folder, whoops. You can’t actually deploy something unless you can mentally untangle it all, it just sits in front of your infra as a sort of DevOps Coming of Age ritual, where you look whistfully over your shoulder at the old Heroku or Vercel account you grew up with. Simpler times.