6 ms·

Show HN: We built a tool that automatically generates API tests

108 points by Wissmueller 4y ago

Hi, this is Mish and Sebastian. We are working on Step CI - a fully automated API testing platform for developers. Step CI works programming-language independent and for different API paradigms (REST, GraphQL, XML). Our CLI and test runner are available on GitHub (https://github.com/stepci https://github.com/stepci) under the MPLv2 license. Since our last launch, Step CI is now able to generate automated tests for your API based on your OpenAPI (Swagger) spec. This saves you a lot of time as you never have to write and maintain your tests again! We would like to invite you to try our tool and give us feedback! Please star us on GitHub, if you like what we are working on! We are very thankful for your attention and any feedback or suggestions we receive from you :) Mish and Sebastian from Germany

fernandorojo 4y ago
Cool idea. I’ll definitely check it out. We’ve been thinking of making something similar internally.
- ushakov 4y ago
  Hello Fernando, this is Mish from Step CI We're glad you liked the idea! I had this idea in my head for about 2 years until I met Sebastian and we decided to give it a try! We already support sending GraphQL queries and validating the responses Soon we will also be able to generate automated API tests based on your GraphQL Schema. You can watch our main repository (https://github.com/stepci/stepci https://github.com/stepci/stepci) or subscribe to our newsletter to follow our progress! Thanks in advance for checking out our tool!
inglor 4y ago
How do you deal with state between API calls? (For example, a user filling a shopping cart, getting discounts based on their geo/products and then checking out)
- ushakov 4y ago
  Hello, this is Mish from Step CI Thanks for your question! We allow you to capture content like headers, response and cookies between steps using captures, check out the "Captures" example in our Demo Here's a documentation with examples for all capture types: https://github.com/stepci/stepci/blob/main/docs/workflow-syntax.md#teststeststepsstepcaptures https://github.com/stepci/stepci/blob/main/docs/workflow-syn... also: we manage cookies for you automatically
  - fernandopj 4y ago
    Does capture implement what I'm trying to discuss here? https://github.com/stepci/stepci/discussions/17 https://github.com/stepci/stepci/discussions/17
    - ushakov 4y ago
      I answered you on GitHub!
manishrana 4y ago
This is really a useful tool
LeonidBugaev 4y ago
If you are looking for a more general tool, which can do not only API tests, but also interact with Databases and etc, but still use declrative syntax, try Venom https://github.com/ovh/venom https://github.com/ovh/venom
- airocker 4y ago
  Why use venom? Why not just pytest ?
- Wissmueller 4y ago
  Hi, this is Sebastian from Step CI Thanks for sharing Venom! Certainly, there are lots of tools in the space, both SaaS and open-source. Mish and I can name you a few! However, there’s one thing these tools seem to be missing - a greater goal. Our vision is to give you a tool that will free you from writing and maintaining API tests ever again - in any shape or form. What you currently see from us is the first step!
  - itake 4y ago
    > Venom is a CLI (Command Line Interface) that aim to create, manage and run your integration tests with efficiency. The linked repo seems to have a greater goal.
    - intelVISA 4y ago
      I'm torn, half of me wants to support them for the hustle but I can't shake the strong "accidentally swallowed the same BS we fed the VCs" vibes. But also in the same vein nothing really is that unique, if they can make some coins this way: who are we to judge?
- sly010 4y ago
  or python https://www.python.org/ https://www.python.org/
mrwnmonm 4y ago
love the design
airocker 4y ago
Hi , I tried to build cli from swagger, it was a disaster: very verbose and did not really work. I am wondering what’s the verbosity of your tool. How do you deal with imperfect specs?
- Wissmueller 4y ago
  We’re sorry to hear that! Would you mind sharing some context with us? Our OpenAPI (Swagger) integration is currently in Beta and only available through our website. We acknowledge, that it is not ready for prime-time just yet Everything we do is open-source. You can see the list of issues in the repository and contribute if possible: https://github.com/stepci/plugin-openapi/issues https://github.com/stepci/plugin-openapi/issues Disclaimer: We have tested our OpenAPI integration with the Petstore example and some other API definitions we had available. In both cases the generated tests would run after only some little adjustments
  - airocker 4y ago
    Swagger has the ability to turn api definition to libraries in any language. It is too verbose
    - raydiatian 4y ago
      openapi is insanely verbose. I personally treat openapi as an interchange format: I don’t write it by hand. The best workflow I’ve found thus far is: FastAPI (Python) or NestJS to produce self documented APIs via route annotations, handled by framework, and then from that, export an openapi file that you use to create the client in your necessary language. In so doing, every service I make can be commanded with ease by whoever wants to use it, at minimal overhead for me as the service maintainer. It was a decent amount of effort to work out the bugs, enough so that I think better documentation around getting setup would be valuable, but of course the documentation is so cross-cutting that it’s difficult. But the work was well worth it—I’m much happier with this workflow where I don’t have to roll a service layer lib for my front end, nor use vanilla fetch libraries.
jicea 4y ago
Really cool, with already a lot of features! Given the raise of CI/CD, I really think these kind of tools (CLI tests on HTTP requests), based on a simple format, will be really important. We've build Hurl (https://github.com/Orange-OpenSource/hurl https://github.com/Orange-OpenSource/hurl), that shares a lot of similarities with Step CI (plain text instead of yaml, captures, jsonpath, xpath etc...). I will shamelessly take inspiration for some new features (like GraphQL for instance)!
- Sebastian_Wis 4y ago
  Oh, nice to meet you! Our Matcher functionality was actually inspired by Hurl (you call them predicates!)! We would take your inspiration from us as a huge compliment :) The main difference between Hurl and Step CI is from our persepective an ideological one. What do you think? Hurl is a CLI tool, while Step CI is a library. This means that you can include our Test Runner (https://github.com/stepci/runner https://github.com/stepci/runner) and use it with other tools. For example, you can combine it with testing frameworks like Mocha, Jest, and Ava (https://github.com/stepci/stepci/blob/main/docs/using-test-tools.md https://github.com/stepci/stepci/blob/main/docs/using-test-t...) or build your own platform on top of it. That's what's so nice about open source.
  - jicea 4y ago
    Yes, I can see that! We focus on the CLI and I see how Step CI can shine. Happy to be a (small) source of inspiration. Good luck with Step CI!
reagan83 4y ago
This is a great idea & brilliantly executed. I love to see the innovation in this space. Nice work Mish and Sebastian!
mkl95 4y ago
OpenAPI Generator + This tool + some good UI would be a killer combination
yuvalsteuer 4y ago
Love it! > "never have to write and maintain your tests again!" this feels a bit misleading though. What are you doing step ci? :D
- Sebastian_Wis 4y ago
  We have been watching you and we know testing APIs is your hobby and you spend a lot of time on it. We'll have to take it away from you! ;) Just give us a little more time and hopefully your support. :)
  - yuvalsteuer 4y ago
    Haha! love the spirit! :)
yuvalsteuer 4y ago
BTW I appreciate good UI, your site is incredible, how did you build it?
- ushakov 4y ago
  Mish from Step CI here We too like good UI! Our website is built with Nuxt and the Demo with CodeMirror 6
barefeg 4y ago
I was looking in the site to understand the automatic generation of tests from OpenAPI but I couldn’t find it. Maybe it’s very obvious but I missed it. Does it mean that it produces test cases from the examples provided? Or how does it figure out the request response pairs?
- Wissmueller 4y ago
  Thank you for the question!:) You can generate tests from your OpenAPI definition using the “import from OpenAPI” button on our website. The tests are generated from your request/response examples. If there are no examples given/available, we will generate examples for you based on the schema provided.
  - barefeg 4y ago
    Thanks for the quick answer! Are the examples generated based on the types? As in if a field is set as an integer, the tests will pass arbitrary integers? But still how does it know how to produce positive examples? Do you maybe have some documentation on this functionality?
    - ushakov 4y ago
      Yes! We generate placeholder values that are based on the types defined in your schema We don't have documentation for it at the moment, but the code for our OpenAPI integration is open-source on GitHub: https://github.com/stepci/plugin-openapi https://github.com/stepci/plugin-openapi
fernandopj 4y ago
Fantastic project. I worked on something similar that had the same approach, but the main goal was to test availability in different environments (dev, stage, production) against the same tests. We also had the feature of using the result of one test as input to another, so a complete user case from an API could be tested (auth, POST data, GET results)
jzig 4y ago
Yo, I tried uploading my openapi.json file and all I got was a big error on the right hand side.
- jzig 4y ago
  In the console: ``` Uncaught YAMLParseError: The : indicator must be at most 1024 chars after the start of an implicit block mapping key at line 5551, column 3: # $ref: '#/components/schemas/String' schemas: ^^^^^^^ ``` Edit: Apparently I have some validation errors when pasting into the Swagger Editor. That must be why; will have to fix those up and try again.
  - ushakov 4y ago
    Hello, did you try again? Please let us know how it went!
raydiatian 4y ago
Way to go guys.
BerislavLopac 4y ago
How does this compare to Schemathesis [0]? [0] https://schemathesis.readthedocs.io https://schemathesis.readthedocs.io
- ushakov 4y ago
  Mish from Step CI here Great question. We are aware of Schemathesis I believe the main difference is that our solution is very configurable, unopinionated and no-code. Schemathesis doesn't offer you an easy way to configure tests, relies on OpenAPI instead and for anything more complicated requires you to use Python Maybe you could add something else?
  - dmitry_dygalo 4y ago
    Hi, Schemathesis author here I'd add that Schemathesis is essentially a fuzzer, where from the first glance Step CI is not (correct me if I am wrong).
APhoenixRises 4y ago
How does Step CI handle testing business logic? I don't see how anything other than input/boundary testing could be generated automatically.
- devoutsalsa 4y ago
  There’s no way they could automate any non-trivial tests. That’s why we still have jobs.
- ushakov 4y ago
  Currently we are doing what is called "Synthetic monitoring" https://en.wikipedia.org/wiki/Synthetic_monitoring https://en.wikipedia.org/wiki/Synthetic_monitoring You can subscribe to our newsletter if you want to get updates once we add new features or make new tools!
SeriousM 4y ago
Some years ago I setup postman tests and run it with cli during deployment tests. The whole hassle with postman ui just makes it really not fun work with it even for simple tasks depending on each other. I wanted to use restclient from vscode where you just define requests in a raw format but there was no usable output to process for azure devops. I will give this project a chance to get things done in a convenient way. Thanks!
- ushakov 4y ago
  Thank you! We'd be happy to hear about your experience with Step CI If you have any questions, feel free to post to our GitHub Discussions: https://github.com/stepci/stepci/discussions https://github.com/stepci/stepci/discussions
thiht 4y ago
From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence. The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom https://github.com/ovh/venom Here's an example test file for the HTTP executor of Venom: https://github.com/ovh/venom/blob/master/tests/http.yml https://github.com/ovh/venom/blob/master/tests/http.yml it's very close to Step CI format. I'd still use Venom because it's way more powerful (you have DB executors for example, so after executing a POST request you can actually check in DB that you have what you expect) and I prefer focusing on actually writing integration tests instead of generating them. Maybe this post sounds harsh (I feel it as I write it because I have strong feelings against test generation) but I think your approach is a good one for actually writing automated tests. Testing APIs declaratively like this has a great benefit: your tests work on an interface. You can migrate your API to a whole new stack and your tests remain the same. I did it multiple time at OVHcloud: one time migrating a huge API from a Go router to another (Gin->Echo), and another time migrating public APIs from a legacy, in-house Perl engine to a Go server.
- ushakov 4y ago
  Hello, Thibaut. We appreciate the honesty! > I prefer working with no tests than automated tests We hear this often from our user-interviews and this is something we would like to change I have just learned about Venom from a fellow hacker yesterday. We're more focused on Web APIs and offer more features there. The key difference is that we're not just a CLI tool, but also a library, that you can use in other applications. Our end goal is to free you from writing tests
- Wissmueller 4y ago
  > I have strong feelings against test generation Thanks for your critical feedback. Could you please tell us more about your strong opinion against test generation?
  - thiht 4y ago

108 points by Wissmueller 4y ago

fernandorojo 4y ago

Cool idea. I’ll definitely check it out. We’ve been thinking of making something similar internally.

ushakov 4y ago
Hello Fernando, this is Mish from Step CI We're glad you liked the idea! I had this idea in my head for about 2 years until I met Sebastian and we decided to give it a try! We already support sending GraphQL queries and validating the responses Soon we will also be able to generate automated API tests based on your GraphQL Schema. You can watch our main repository (https://github.com/stepci/stepci https://github.com/stepci/stepci) or subscribe to our newsletter to follow our progress! Thanks in advance for checking out our tool!

inglor 4y ago

How do you deal with state between API calls? (For example, a user filling a shopping cart, getting discounts based on their geo/products and then checking out)

ushakov 4y ago
Hello, this is Mish from Step CI Thanks for your question! We allow you to capture content like headers, response and cookies between steps using captures, check out the "Captures" example in our Demo Here's a documentation with examples for all capture types: https://github.com/stepci/stepci/blob/main/docs/workflow-syntax.md#teststeststepsstepcaptures https://github.com/stepci/stepci/blob/main/docs/workflow-syn... also: we manage cookies for you automatically
- fernandopj 4y ago
  Does capture implement what I'm trying to discuss here? https://github.com/stepci/stepci/discussions/17 https://github.com/stepci/stepci/discussions/17
  - ushakov 4y ago
    I answered you on GitHub!

manishrana 4y ago

This is really a useful tool

LeonidBugaev 4y ago

If you are looking for a more general tool, which can do not only API tests, but also interact with Databases and etc, but still use declrative syntax, try Venom https://github.com/ovh/venom https://github.com/ovh/venom

airocker 4y ago
Why use venom? Why not just pytest ?
Wissmueller 4y ago
Hi, this is Sebastian from Step CI Thanks for sharing Venom! Certainly, there are lots of tools in the space, both SaaS and open-source. Mish and I can name you a few! However, there’s one thing these tools seem to be missing - a greater goal. Our vision is to give you a tool that will free you from writing and maintaining API tests ever again - in any shape or form. What you currently see from us is the first step!
- itake 4y ago
  > Venom is a CLI (Command Line Interface) that aim to create, manage and run your integration tests with efficiency. The linked repo seems to have a greater goal.
  - intelVISA 4y ago
    I'm torn, half of me wants to support them for the hustle but I can't shake the strong "accidentally swallowed the same BS we fed the VCs" vibes. But also in the same vein nothing really is that unique, if they can make some coins this way: who are we to judge?
sly010 4y ago
or python https://www.python.org/ https://www.python.org/

mrwnmonm 4y ago

love the design

airocker 4y ago

Hi , I tried to build cli from swagger, it was a disaster: very verbose and did not really work. I am wondering what’s the verbosity of your tool. How do you deal with imperfect specs?

Wissmueller 4y ago
We’re sorry to hear that! Would you mind sharing some context with us? Our OpenAPI (Swagger) integration is currently in Beta and only available through our website. We acknowledge, that it is not ready for prime-time just yet Everything we do is open-source. You can see the list of issues in the repository and contribute if possible: https://github.com/stepci/plugin-openapi/issues https://github.com/stepci/plugin-openapi/issues Disclaimer: We have tested our OpenAPI integration with the Petstore example and some other API definitions we had available. In both cases the generated tests would run after only some little adjustments
- airocker 4y ago
  Swagger has the ability to turn api definition to libraries in any language. It is too verbose
  - raydiatian 4y ago
    openapi is insanely verbose. I personally treat openapi as an interchange format: I don’t write it by hand. The best workflow I’ve found thus far is: FastAPI (Python) or NestJS to produce self documented APIs via route annotations, handled by framework, and then from that, export an openapi file that you use to create the client in your necessary language. In so doing, every service I make can be commanded with ease by whoever wants to use it, at minimal overhead for me as the service maintainer. It was a decent amount of effort to work out the bugs, enough so that I think better documentation around getting setup would be valuable, but of course the documentation is so cross-cutting that it’s difficult. But the work was well worth it—I’m much happier with this workflow where I don’t have to roll a service layer lib for my front end, nor use vanilla fetch libraries.

jicea 4y ago

Really cool, with already a lot of features! Given the raise of CI/CD, I really think these kind of tools (CLI tests on HTTP requests), based on a simple format, will be really important. We've build Hurl (https://github.com/Orange-OpenSource/hurl https://github.com/Orange-OpenSource/hurl), that shares a lot of similarities with Step CI (plain text instead of yaml, captures, jsonpath, xpath etc...). I will shamelessly take inspiration for some new features (like GraphQL for instance)!

Sebastian_Wis 4y ago
Oh, nice to meet you! Our Matcher functionality was actually inspired by Hurl (you call them predicates!)! We would take your inspiration from us as a huge compliment :) The main difference between Hurl and Step CI is from our persepective an ideological one. What do you think? Hurl is a CLI tool, while Step CI is a library. This means that you can include our Test Runner (https://github.com/stepci/runner https://github.com/stepci/runner) and use it with other tools. For example, you can combine it with testing frameworks like Mocha, Jest, and Ava (https://github.com/stepci/stepci/blob/main/docs/using-test-tools.md https://github.com/stepci/stepci/blob/main/docs/using-test-t...) or build your own platform on top of it. That's what's so nice about open source.
- jicea 4y ago
  Yes, I can see that! We focus on the CLI and I see how Step CI can shine. Happy to be a (small) source of inspiration. Good luck with Step CI!

reagan83 4y ago

This is a great idea & brilliantly executed. I love to see the innovation in this space. Nice work Mish and Sebastian!

mkl95 4y ago

OpenAPI Generator + This tool + some good UI would be a killer combination

yuvalsteuer 4y ago

Love it! > "never have to write and maintain your tests again!" this feels a bit misleading though. What are you doing step ci? :D

Sebastian_Wis 4y ago
We have been watching you and we know testing APIs is your hobby and you spend a lot of time on it. We'll have to take it away from you! ;) Just give us a little more time and hopefully your support. :)
- yuvalsteuer 4y ago
  Haha! love the spirit! :)

yuvalsteuer 4y ago

BTW I appreciate good UI, your site is incredible, how did you build it?

ushakov 4y ago
Mish from Step CI here We too like good UI! Our website is built with Nuxt and the Demo with CodeMirror 6

barefeg 4y ago

I was looking in the site to understand the automatic generation of tests from OpenAPI but I couldn’t find it. Maybe it’s very obvious but I missed it. Does it mean that it produces test cases from the examples provided? Or how does it figure out the request response pairs?

Wissmueller 4y ago
Thank you for the question!:) You can generate tests from your OpenAPI definition using the “import from OpenAPI” button on our website. The tests are generated from your request/response examples. If there are no examples given/available, we will generate examples for you based on the schema provided.
- barefeg 4y ago
  Thanks for the quick answer! Are the examples generated based on the types? As in if a field is set as an integer, the tests will pass arbitrary integers? But still how does it know how to produce positive examples? Do you maybe have some documentation on this functionality?
  - ushakov 4y ago
    Yes! We generate placeholder values that are based on the types defined in your schema We don't have documentation for it at the moment, but the code for our OpenAPI integration is open-source on GitHub: https://github.com/stepci/plugin-openapi https://github.com/stepci/plugin-openapi

fernandopj 4y ago

Fantastic project. I worked on something similar that had the same approach, but the main goal was to test availability in different environments (dev, stage, production) against the same tests. We also had the feature of using the result of one test as input to another, so a complete user case from an API could be tested (auth, POST data, GET results)

jzig 4y ago

Yo, I tried uploading my openapi.json file and all I got was a big error on the right hand side.

jzig 4y ago
In the console: ``` Uncaught YAMLParseError: The : indicator must be at most 1024 chars after the start of an implicit block mapping key at line 5551, column 3: # $ref: '#/components/schemas/String' schemas: ^^^^^^^ ``` Edit: Apparently I have some validation errors when pasting into the Swagger Editor. That must be why; will have to fix those up and try again.
- ushakov 4y ago
  Hello, did you try again? Please let us know how it went!

raydiatian 4y ago

Way to go guys.

BerislavLopac 4y ago

How does this compare to Schemathesis [0]? [0] https://schemathesis.readthedocs.io https://schemathesis.readthedocs.io

ushakov 4y ago
Mish from Step CI here Great question. We are aware of Schemathesis I believe the main difference is that our solution is very configurable, unopinionated and no-code. Schemathesis doesn't offer you an easy way to configure tests, relies on OpenAPI instead and for anything more complicated requires you to use Python Maybe you could add something else?
- dmitry_dygalo 4y ago
  Hi, Schemathesis author here I'd add that Schemathesis is essentially a fuzzer, where from the first glance Step CI is not (correct me if I am wrong).

APhoenixRises 4y ago

How does Step CI handle testing business logic? I don't see how anything other than input/boundary testing could be generated automatically.

devoutsalsa 4y ago
There’s no way they could automate any non-trivial tests. That’s why we still have jobs.
ushakov 4y ago
Currently we are doing what is called "Synthetic monitoring" https://en.wikipedia.org/wiki/Synthetic_monitoring https://en.wikipedia.org/wiki/Synthetic_monitoring You can subscribe to our newsletter if you want to get updates once we add new features or make new tools!

SeriousM 4y ago

Some years ago I setup postman tests and run it with cli during deployment tests. The whole hassle with postman ui just makes it really not fun work with it even for simple tasks depending on each other. I wanted to use restclient from vscode where you just define requests in a raw format but there was no usable output to process for azure devops. I will give this project a chance to get things done in a convenient way. Thanks!

ushakov 4y ago
Thank you! We'd be happy to hear about your experience with Step CI If you have any questions, feel free to post to our GitHub Discussions: https://github.com/stepci/stepci/discussions https://github.com/stepci/stepci/discussions

thiht 4y ago

From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence. The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom https://github.com/ovh/venom Here's an example test file for the HTTP executor of Venom: https://github.com/ovh/venom/blob/master/tests/http.yml https://github.com/ovh/venom/blob/master/tests/http.yml it's very close to Step CI format. I'd still use Venom because it's way more powerful (you have DB executors for example, so after executing a POST request you can actually check in DB that you have what you expect) and I prefer focusing on actually writing integration tests instead of generating them. Maybe this post sounds harsh (I feel it as I write it because I have strong feelings against test generation) but I think your approach is a good one for actually writing automated tests. Testing APIs declaratively like this has a great benefit: your tests work on an interface. You can migrate your API to a whole new stack and your tests remain the same. I did it multiple time at OVHcloud: one time migrating a huge API from a Go router to another (Gin->Echo), and another time migrating public APIs from a legacy, in-house Perl engine to a Go server.

ushakov 4y ago
Hello, Thibaut. We appreciate the honesty! > I prefer working with no tests than automated tests We hear this often from our user-interviews and this is something we would like to change I have just learned about Venom from a fellow hacker yesterday. We're more focused on Web APIs and offer more features there. The key difference is that we're not just a CLI tool, but also a library, that you can use in other applications. Our end goal is to free you from writing tests
Wissmueller 4y ago
> I have strong feelings against test generation Thanks for your critical feedback. Could you please tell us more about your strong opinion against test generation?
- thiht 4y ago