7 ms·

Taming OpenAPI using Racket to create a DSL

74 points by spdegabrielle 7y ago

nudpiedo 7y ago
Little good nice story of how to introduce a new tool for work without getting into a crusade. I guess however the manager/architect didn't care much of the additional dependency added to the project thought.
sametmax 7y ago
Why didnt they use the language they already coded the API in then ?
- king_magic 7y ago
  This is where I’m really struggling with this story. Now, they have two places documentation can reside: in the code for the API itself, and now in a separate file encoded with a custom LISP-like language? I feel like the author is forgetting how software gets developed in the real world... or just human nature in general. Documentation created with this approach will instantly go stale, and will likely get out of sync with the true API very quickly. How is this a good solution at all? How is “tricking” people into this unsustainable situation a good thing?
  - amelius 7y ago
    I'm guessing the real plan is to trick the co-workers into rewriting the entire application in Racket.
    - yowlingcat 7y ago
      It really irks me how some treat the practicalities and existing choices of real world software development as an annoying impediment to getting to use a language they're interested in. Whether you're interested in a language and whether you should use it in production can be two separate things. If you've got a compelling reason to use a given language in production, you probably don't have to trick your co-workers into doing anything, you just have a conversation with them! And if you "trick" them into using it for something small, aren't you going to sabotage your long term relationship and trust with them for minimal gain?
      - detaro 7y ago
        > If you've got a compelling reason to use a given language in production, you probably don't have to trick your co-workers into doing anything, you just have a conversation with them! Nothing in the article suggests that conversation didn't happen. The only mention of "tricked" is the headline the submitter gave it on HN.
  - lukeschlather 7y ago
    It all depends on how concise and easy to use this is. My experience with yaml/json API generators is that they are extremely buggy due to myriad edge cases. If it goes stale, it's because it takes an hour to figure out how to represent what you want to say in the language, vs. just writing a json blob with some non-machine-readable type annotations. This is self-contained enough that I could see the language not mattering too much. And while I don't use Racket at all it sounds like the ideal language for this task. I'm not saying it confidently, but I wouldn't be surprised if the Racket implementation is less buggy than any non-lisp. Generating YAML/JSON is a task which lisp might be uniquely suited for.
yowlingcat 7y ago
The idea of using a DSL to get around constraints in an upstream language you don't control is indeed a nice one, and one I've used before to a good degree. I'm sure I'm going to get downvoted for this, but the use of Racket is a smell here. Why not use whatever language you were using before to turn the input DSL into the output OpenAPI spec? Sooner or later, another engineer is going to have to maintain that. You already have another language you're using for other things, and you're introducing an entirely new language and ecosystem to do one small thing. Is this something that couldn't be done in your existing stack?
- deleted 7y ago
  [deleted]
Smithalicious 7y ago
Please don't try to nest quotes of the same kind like this. It reads as 3 parts, quoted - unquoted - quoted, and makes no sense like that. Yes this is a very pedantic thing to complain about and I do understand what is meant. But it does bother me. Yes, I do realize it's using slanted quotes and not just " - still doesn't read right for me.
JoelMcCracken 7y ago
Generally, this technique can work for adding any kind of pet tech to your work stack: - identify an isolated, low-risk need. - develop a solution in your tech (usually on your own time). - introduce to your boss/company as already developed. The biggest risk for the business is needing to replace the component. In these cases usually an isolated, irregular solution is acceptable
- yowlingcat 7y ago
  Why the phrase "pet tech"? This is a good way to "ask for forgiveness, not permission" which I approve of. But is the goal of adding "pet tech" necessarily a good one? If technology is actually going to move your firm forward long term and is a good long term decision, it's not pet technology. It's a good decision, and it's something that hopefully your team and firm is behind.
  - marcinzm 7y ago
    If it's a good decision then you shouldn't have to ask for forgiveness rather than permission. It's also, if it's not just a pet technology, a strategic decision that impacts everyone on your team and it's pure ego to think you should make that decision alone.
    - yowlingcat 7y ago
      I agree with both points. I do think there are orgs where, due to shall we say "personality constraints" you can have majority consensus that a decision is good and should be made but it can still get roadblocked. Of course, such situations are an indicator of a dysfunctional org; in a functional one, you have these kinds of open discussions and like you said, you don't need to ask for forgiveness rather than permission, and you can make these strategic decisions together as a team.
sergiotapia 7y ago
Now you are the single point of failure for this service, and worse, you forced your team to learn something they may have had zero interest in looking into. Never do this in prod without a heads up and approval from most of your team. I worked with a guy who after completing his tickets instead of looking for more tickets or helping his team, spent his time rewriting the entire clientside application in typescript. He was let go the following month.
- deleted 7y ago
  [deleted]
- detaro 7y ago
  Am I missing something? The only thing pointing in that direction I see is the submission title here on HN, which is in no way reflected in the article?
  - sergiotapia 7y ago
    No mention in the article but the title is clearly implying that.
    - detaro 7y ago
      Which is the title on HN, not the title of the article.
deleted 7y ago
[deleted]
marcinzm 7y ago
edit: Got confused by the title not actually matching the article title and having click-bait information added to it that didn't appear in the original.
- detaro 7y ago
  Which part of the article(!) suggests to you they didn't have that discussion?
  - marcinzm 7y ago
    The word tricked.
    - detaro 7y ago
      Where in the article does it appear? From what I see, the only claim in that direction is the random(?) headline the submitter added here. And even then, it doesn't have to be interpreted that way - it could also apply to "they'd normally not consider a language like this, but for this specific case it seemed to make sense".
      - marcinzm 7y ago
        Good point, didn't realize they changed the original title since that's almost almost frowned upon on HN.
- ehnto 7y ago
  It's definitely something that should be justified in a PR. Introducing a new tool, it's dependencies, maintenance, knowledge and training requirements is something that should be agreed upon by the team. For agencies especially, tooling is a strategic business decision.
seandhi 7y ago
I don’t see that quote in the article at all, and it doesn’t appear that the author claims any kind of trickery or sneakiness to trick his team into using Racket. It seems like the team built the tool together.
- neilv 7y ago
  Yes, I don't see any mention of trickery in the article. For the record, capital-R Racketeers do not condone trickery.
  - zests 7y ago
    I interpret the "double "scare quotes"" in the title as suggesting cheeky sarcasm.
    - neilv 7y ago
      (In case future readers are confused, the original HN title had odd nested double-quotes, and the innermost quoted part was something like "how I tricked my employer into using Racket". Which seemed like terrible PR, giving the impression that this is something for irresponsible fringe fanatics.)
  - ken 7y ago
    That's a surprise! It sounds like the first in its line. As Wikipedia says: > Scheme was originally called "Schemer", in the tradition of other Lisp-derived languages such as Planner or Conniver. If there's any programming language family which condones trickery, I would think this would be it.
    - neilv 7y ago
      Whatever the goals of all the Scheming, it doesn't seem to have involved monetary wealth. :)
- guessmyname 7y ago
  Yes, I agree with you. For context, the original title was: > “This is “how I tricked my co-workers” into using Racket.” But has now been changed to: > Taming OpenAPI using Racket to create a DSL Which is the original sub-title of the article. The real title is actually: > Making OpenAPI / Swagger Bearable With Your Own DSL
BerislavLopac 7y ago
> Problem: while OpenAPI is great for describing APIs, generating client SDK code and defining your API contracts, it's definitely not a great experience to write OpenAPI documents from scratch. Personally, my main problem with OpenAPI is that it is commonly used for documenting contracts, but very rarely for actually driving the implementation. In practice, I have seen a few implementations like OpenAPI Generator [0] but a) with dynamic languages like Python code generation is an anti-pattern, and b) even disregarding the former, the generated result is (in my experience) very incomplete and still needs a lot of manual coding. I would like to see more frameworks that utilise OpenAPI for routing and request/response validation. [0] https://openapi-generator.tech/ https://openapi-generator.tech/
- jbmsf 7y ago
  It's always a bit risky to document contracts with the left hand and implement them with the right. The best case to me is a framework that serves both functions at once. I built something like this at my last job by having an opinionated layer on top of Flask in Python. The idea was to have a small set of standard operation types (think:create, retrieve, update) and choices as to input/output types (via the marshmallow library ) and the function to process the route. This turns out to be enough to generate both the api contract and the boilerplate for routing, (un)marshalling, etc. You can find the implementation in this repo: https://github.com/globality-corp/microcosm-flask https://github.com/globality-corp/microcosm-flask
  - BerislavLopac 7y ago
    Yes, FastAPI [0] takes a similar approach. The problem here is that the single-source-of-truth for your contract is not OpenAPI, but rather your Flask implementation, with the OpenAPI being a documentation artefact; this is problematic for two reasons: 1. Just like any other documentation, it is difficult to verify that the documentation is actually correct and does not diverge from he underlying implementation. This is a lesser issue, as it can be mitigated with good integration testing practices. 2. The bigger issue, for me, is that any clients depends on the service implementation. In practice, it is very common to build clients -- especially complex ones like React or mobile UI -- alongside the service development, and if the spec is defined first both can be done in parallel. With the above approach you either need to a) wait for the service implementation to be complete before you can start implementing the client, or b) base the client on an OpenAPI spec which might potentially differ from the one your framework will generate based on the implementation. I have recently worked on a project which had that same issue, and our initial solution was to build tests which would compare the generated OpenAPI with the design specification, but that turned to be extremely complex when we started running into all of the edge cases. The alternative was to treat OpenAPI as the single-source-of-truth, using it to generate routes and execute validation over requests and responses. The first attempt used Connexion [1], which proved to be a bit too incomplete for our needs, so we implemented an alternative framework [2] (which includes a basic client-side support as well). [0] https://fastapi.tiangolo.com/ https://fastapi.tiangolo.com/ [1] https://connexion.readthedocs.io https://connexion.readthedocs.io [2] https://github.com/berislavlopac/pyotr https://github.com/berislavlopac/pyotr (still under heavy construction)
tjalfi 7y ago
It isn’t clear whether they already use Racket. If Racket was added for this purpose then here are a couple boring[0] solutions that could have lower operational costs. Generate OpenAPI specifications from your service code. Use a YAML preprocessor/template engine such as [1]. I have used a similar technique to generate SQL queries. [0] https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology [1] https://github.com/con2/emrichen https://github.com/con2/emrichen
WhitneyLand 7y ago
Seems like it could be convenient to use typescript in some way. Not to code the DSL, but as the DSL itself. It's language specific, but if a variation with minor attributing extensions existed that seamlessly round-tripped to yaml, the verbosity would just completely collapse. Readability and creating from scratch could be an order of magnitude more efficient for anyone familiar with JS (reading a handful of interface definitions vs. human parsing back and forth between the docs and YAML).
sulami 7y ago
I’ve recently built a thing at work to generate an OpenAPI spec from Clojure specs and some annotations, which worked out surprisingly well. We already have a collection of endpoints through compojure, and we just spec the handler functions and add some annotations for docstrings, and that’s basically it. As an added bonus we have the test suite validate all api calls against the spec, so in theory the OpenAPI spec should always be accurate. For the curious, we took and hacked https://github.com/metosin/spec-tools https://github.com/metosin/spec-tools to output OpenAPI 3, and added a very thin shim to hook that up to our routes.

74 points by spdegabrielle 7y ago

Little good nice story of how to introduce a new tool for work without getting into a crusade. I guess however the manager/architect didn't care much of the additional dependency added to the project thought.

sametmax 7y ago

Why didnt they use the language they already coded the API in then ?

king_magic 7y ago
This is where I’m really struggling with this story. Now, they have two places documentation can reside: in the code for the API itself, and now in a separate file encoded with a custom LISP-like language? I feel like the author is forgetting how software gets developed in the real world... or just human nature in general. Documentation created with this approach will instantly go stale, and will likely get out of sync with the true API very quickly. How is this a good solution at all? How is “tricking” people into this unsustainable situation a good thing?
- amelius 7y ago
  I'm guessing the real plan is to trick the co-workers into rewriting the entire application in Racket.
  - yowlingcat 7y ago
    It really irks me how some treat the practicalities and existing choices of real world software development as an annoying impediment to getting to use a language they're interested in. Whether you're interested in a language and whether you should use it in production can be two separate things. If you've got a compelling reason to use a given language in production, you probably don't have to trick your co-workers into doing anything, you just have a conversation with them! And if you "trick" them into using it for something small, aren't you going to sabotage your long term relationship and trust with them for minimal gain?
    - detaro 7y ago
      > If you've got a compelling reason to use a given language in production, you probably don't have to trick your co-workers into doing anything, you just have a conversation with them! Nothing in the article suggests that conversation didn't happen. The only mention of "tricked" is the headline the submitter gave it on HN.
- lukeschlather 7y ago
  It all depends on how concise and easy to use this is. My experience with yaml/json API generators is that they are extremely buggy due to myriad edge cases. If it goes stale, it's because it takes an hour to figure out how to represent what you want to say in the language, vs. just writing a json blob with some non-machine-readable type annotations. This is self-contained enough that I could see the language not mattering too much. And while I don't use Racket at all it sounds like the ideal language for this task. I'm not saying it confidently, but I wouldn't be surprised if the Racket implementation is less buggy than any non-lisp. Generating YAML/JSON is a task which lisp might be uniquely suited for.

yowlingcat 7y ago

The idea of using a DSL to get around constraints in an upstream language you don't control is indeed a nice one, and one I've used before to a good degree. I'm sure I'm going to get downvoted for this, but the use of Racket is a smell here. Why not use whatever language you were using before to turn the input DSL into the output OpenAPI spec? Sooner or later, another engineer is going to have to maintain that. You already have another language you're using for other things, and you're introducing an entirely new language and ecosystem to do one small thing. Is this something that couldn't be done in your existing stack?

deleted 7y ago
[deleted]

Smithalicious 7y ago

Please don't try to nest quotes of the same kind like this. It reads as 3 parts, quoted - unquoted - quoted, and makes no sense like that. Yes this is a very pedantic thing to complain about and I do understand what is meant. But it does bother me. Yes, I do realize it's using slanted quotes and not just " - still doesn't read right for me.

JoelMcCracken 7y ago

Generally, this technique can work for adding any kind of pet tech to your work stack: - identify an isolated, low-risk need. - develop a solution in your tech (usually on your own time). - introduce to your boss/company as already developed. The biggest risk for the business is needing to replace the component. In these cases usually an isolated, irregular solution is acceptable

yowlingcat 7y ago
Why the phrase "pet tech"? This is a good way to "ask for forgiveness, not permission" which I approve of. But is the goal of adding "pet tech" necessarily a good one? If technology is actually going to move your firm forward long term and is a good long term decision, it's not pet technology. It's a good decision, and it's something that hopefully your team and firm is behind.
- marcinzm 7y ago
  If it's a good decision then you shouldn't have to ask for forgiveness rather than permission. It's also, if it's not just a pet technology, a strategic decision that impacts everyone on your team and it's pure ego to think you should make that decision alone.
  - yowlingcat 7y ago
    I agree with both points. I do think there are orgs where, due to shall we say "personality constraints" you can have majority consensus that a decision is good and should be made but it can still get roadblocked. Of course, such situations are an indicator of a dysfunctional org; in a functional one, you have these kinds of open discussions and like you said, you don't need to ask for forgiveness rather than permission, and you can make these strategic decisions together as a team.

sergiotapia 7y ago

Now you are the single point of failure for this service, and worse, you forced your team to learn something they may have had zero interest in looking into. Never do this in prod without a heads up and approval from most of your team. I worked with a guy who after completing his tickets instead of looking for more tickets or helping his team, spent his time rewriting the entire clientside application in typescript. He was let go the following month.

deleted 7y ago
[deleted]
detaro 7y ago
Am I missing something? The only thing pointing in that direction I see is the submission title here on HN, which is in no way reflected in the article?
- sergiotapia 7y ago
  No mention in the article but the title is clearly implying that.
  - detaro 7y ago
    Which is the title on HN, not the title of the article.

deleted 7y ago

[deleted]

marcinzm 7y ago

edit: Got confused by the title not actually matching the article title and having click-bait information added to it that didn't appear in the original.

detaro 7y ago
Which part of the article(!) suggests to you they didn't have that discussion?
- marcinzm 7y ago
  The word tricked.
  - detaro 7y ago
    Where in the article does it appear? From what I see, the only claim in that direction is the random(?) headline the submitter added here. And even then, it doesn't have to be interpreted that way - it could also apply to "they'd normally not consider a language like this, but for this specific case it seemed to make sense".
    - marcinzm 7y ago
      Good point, didn't realize they changed the original title since that's almost almost frowned upon on HN.
ehnto 7y ago
It's definitely something that should be justified in a PR. Introducing a new tool, it's dependencies, maintenance, knowledge and training requirements is something that should be agreed upon by the team. For agencies especially, tooling is a strategic business decision.

seandhi 7y ago

I don’t see that quote in the article at all, and it doesn’t appear that the author claims any kind of trickery or sneakiness to trick his team into using Racket. It seems like the team built the tool together.

neilv 7y ago
Yes, I don't see any mention of trickery in the article. For the record, capital-R Racketeers do not condone trickery.
- zests 7y ago
  I interpret the "double "scare quotes"" in the title as suggesting cheeky sarcasm.
  - neilv 7y ago
    (In case future readers are confused, the original HN title had odd nested double-quotes, and the innermost quoted part was something like "how I tricked my employer into using Racket". Which seemed like terrible PR, giving the impression that this is something for irresponsible fringe fanatics.)
- ken 7y ago
  That's a surprise! It sounds like the first in its line. As Wikipedia says: > Scheme was originally called "Schemer", in the tradition of other Lisp-derived languages such as Planner or Conniver. If there's any programming language family which condones trickery, I would think this would be it.
  - neilv 7y ago
    Whatever the goals of all the Scheming, it doesn't seem to have involved monetary wealth. :)
guessmyname 7y ago
Yes, I agree with you. For context, the original title was: > “This is “how I tricked my co-workers” into using Racket.” But has now been changed to: > Taming OpenAPI using Racket to create a DSL Which is the original sub-title of the article. The real title is actually: > Making OpenAPI / Swagger Bearable With Your Own DSL

BerislavLopac 7y ago

> Problem: while OpenAPI is great for describing APIs, generating client SDK code and defining your API contracts, it's definitely not a great experience to write OpenAPI documents from scratch. Personally, my main problem with OpenAPI is that it is commonly used for documenting contracts, but very rarely for actually driving the implementation. In practice, I have seen a few implementations like OpenAPI Generator [0] but a) with dynamic languages like Python code generation is an anti-pattern, and b) even disregarding the former, the generated result is (in my experience) very incomplete and still needs a lot of manual coding. I would like to see more frameworks that utilise OpenAPI for routing and request/response validation. [0] https://openapi-generator.tech/ https://openapi-generator.tech/

jbmsf 7y ago
It's always a bit risky to document contracts with the left hand and implement them with the right. The best case to me is a framework that serves both functions at once. I built something like this at my last job by having an opinionated layer on top of Flask in Python. The idea was to have a small set of standard operation types (think:create, retrieve, update) and choices as to input/output types (via the marshmallow library ) and the function to process the route. This turns out to be enough to generate both the api contract and the boilerplate for routing, (un)marshalling, etc. You can find the implementation in this repo: https://github.com/globality-corp/microcosm-flask https://github.com/globality-corp/microcosm-flask
- BerislavLopac 7y ago
  Yes, FastAPI [0] takes a similar approach. The problem here is that the single-source-of-truth for your contract is not OpenAPI, but rather your Flask implementation, with the OpenAPI being a documentation artefact; this is problematic for two reasons: 1. Just like any other documentation, it is difficult to verify that the documentation is actually correct and does not diverge from he underlying implementation. This is a lesser issue, as it can be mitigated with good integration testing practices. 2. The bigger issue, for me, is that any clients depends on the service implementation. In practice, it is very common to build clients -- especially complex ones like React or mobile UI -- alongside the service development, and if the spec is defined first both can be done in parallel. With the above approach you either need to a) wait for the service implementation to be complete before you can start implementing the client, or b) base the client on an OpenAPI spec which might potentially differ from the one your framework will generate based on the implementation. I have recently worked on a project which had that same issue, and our initial solution was to build tests which would compare the generated OpenAPI with the design specification, but that turned to be extremely complex when we started running into all of the edge cases. The alternative was to treat OpenAPI as the single-source-of-truth, using it to generate routes and execute validation over requests and responses. The first attempt used Connexion [1], which proved to be a bit too incomplete for our needs, so we implemented an alternative framework [2] (which includes a basic client-side support as well). [0] https://fastapi.tiangolo.com/ https://fastapi.tiangolo.com/ [1] https://connexion.readthedocs.io https://connexion.readthedocs.io [2] https://github.com/berislavlopac/pyotr https://github.com/berislavlopac/pyotr (still under heavy construction)

tjalfi 7y ago

It isn’t clear whether they already use Racket. If Racket was added for this purpose then here are a couple boring[0] solutions that could have lower operational costs. Generate OpenAPI specifications from your service code. Use a YAML preprocessor/template engine such as [1]. I have used a similar technique to generate SQL queries. [0] https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology [1] https://github.com/con2/emrichen https://github.com/con2/emrichen

WhitneyLand 7y ago

Seems like it could be convenient to use typescript in some way. Not to code the DSL, but as the DSL itself. It's language specific, but if a variation with minor attributing extensions existed that seamlessly round-tripped to yaml, the verbosity would just completely collapse. Readability and creating from scratch could be an order of magnitude more efficient for anyone familiar with JS (reading a handful of interface definitions vs. human parsing back and forth between the docs and YAML).

sulami 7y ago

I’ve recently built a thing at work to generate an OpenAPI spec from Clojure specs and some annotations, which worked out surprisingly well. We already have a collection of endpoints through compojure, and we just spec the handler functions and add some annotations for docstrings, and that’s basically it. As an added bonus we have the test suite validate all api calls against the spec, so in theory the OpenAPI spec should always be accurate. For the curious, we took and hacked https://github.com/metosin/spec-tools https://github.com/metosin/spec-tools to output OpenAPI 3, and added a very thin shim to hook that up to our routes.