6 ms·

Please make this the industry standard for testing APIs. I'm tired of having to look at Postman screenshots sent from QA. I'm tired of having to wait for them

Please make this the industry standard for testing APIs. I'm tired of having to look at Postman screenshots sent from QA. I'm tired of having to wait for them to press Send once I've implemented a fix. I know they're tired of waiting for me to do that, too. Hurl is something both the devs and QA can speak and write. It can be automated and a part of CI. It makes communicating expectations straightforward. It can be chucked along with PRs as a starting point for QA. I don't see a reason not to use it wherever possible.

digikata 3y ago
I have been using it for API testing manual and in CI of one of our services and it's been very nice. You can basically put a series of http exchanges for a workflow per hurl file and get a nice test suite that also checks directly into git.
vidyesh 3y ago
For a team using VSCode you can try the vscode-restclient[1] But really Hurl looks really interesting, being editor agnostic is the best solution for your problem, I agree. [1] https://github.com/Huachao/vscode-restclient https://github.com/Huachao/vscode-restclient
- kxrm 3y ago
  I use the vscode-restclient and my primary reason is the conversational flows you can build against an API. Does Hurl support this? If so I would absolutely switch. All I would need to complete the experience is a plugin to do highlighting and integration with the Hurl files.
  - steve_adams_86 3y ago
    digikata’s comment above suggests it is possible. It would be great to have something that isn’t attached to the editor of choice.
    - kxrm 3y ago
      Indeed it does with https://hurl.dev/docs/capturing-response.html https://hurl.dev/docs/capturing-response.html Also for VS Code integration specifically there is https://marketplace.visualstudio.com/items?itemName=JacobPfeifer.pfeifer-hurl https://marketplace.visualstudio.com/items?itemName=JacobPfe...
ushakov 3y ago
There’s also Step CI: https://stepci.com https://stepci.com (I’m one of the authors) Hurl is brilliant though
- randomsofr 3y ago
  I was just looking at this. We might use it at my company, but i was wondering, is this funded? Or is this just an open source side project for you guys.
  - ushakov 3y ago
    Self-funded atm.
jicea 3y ago
Hi maintainer here! Thanks a lot for the kind words!
- convalescindrey 3y ago
  Great project! A comment I'd have is that it's quite hard to find out who authored this fine piece of software.
kbenson 3y ago
Not that I necessarily think it's best to stay with Postman, but have you looked at newman, which is the CLI runner for postman configs? We had postman as a test suite for something (which is more an API than an app), and I got tired of having to deal with setting up extra steps to test and of exporting the postman config to save in the repo, so I put newman on the test system and just run against the config directly in the test environment and check the output. I don't necessarily recommend editing the postman config json directly to set up new tests as it's a PITA, but it's generally what I do so I don't need to keep importing and exporting it with Postman. A tool designed for working with on the shell is likely better than what I'm doing with newman (since the config is not the most accessible), but it also meant I didn't need to rewrite a bunch of existing tests and verify they actually did the same thing.
- jug6ernaut 3y ago
  The problem Newman/postman have is the same for every GUI based testing application. They almost always produce non human readable config files. Making any kind of code review of such changes at best extremely painful and at worst impossible. IMO any testing tool that does not save it's test classes in a human readable format is DoA.
cmgriffing 3y ago
I personally like the approach of defining things via an OpenAPI and then using Dredd to validate the spec against itself. Even for tools that generate the spec from source code, it is usually still possible for user error to define the metadata for an endpoint incorrectly. Dredd catches that.
- TheBigRoomXXL 3y ago
  I also think that validating an API against it's OpenAPI schema is a great methodology. You should checkout schemathesis, it's fantastic for doing that. https://github.com/schemathesis/schemathesis https://github.com/schemathesis/schemathesis
recroad 3y ago
The bigger problem here is that you have QA, not the lack of tooling.
- dijit 3y ago
  The bigger problem here is that you have devs, devs make bugs. if you don't have devs there are no bugs, problem solved.
  - convalescindrey 3y ago
    If devs have to do QA themselves, many issues magically disappear.
    - dijit 3y ago
      Honestly that sounds like making pilots build aircraft engines. these are different disciplines that deserve to be done well. Maybe I am biased because I spent the last 10 years in gamedev, or maybe this is another push to make devs do basically everything tech related: but if a developer tells me a feature is done I always look to QA for a nod. That nod rarely comes, the feature is not done, the developer merely got it to work on their machine.
      - convalescindrey 3y ago
        Then your developer should learn that "done" means more than "works on their machine". The difference between devs and dedicated QA people is that devs know the dark corners of the implementation. They know the edge cases and scenarios that they struggled with getting right. That's where most testing focus has to be. QA doesn't know any of that. They can play through some scenarios that are the expected ones from the spec, perhaps have some hunch of what could be the tricky cases, but they don't actually know. I love seeing all those downvotes for my GP comment. That's all the people working at companies where stuff that devs feel is below them is dumped on QA people, which is the main reason my company is well-known for shipping top quality (without having a QA dept) while our competitors struggle with quality (despite having a QA dept). That's all the evidence I need to back up my claim.
imiric 3y ago
Have you tried https://k6.io/ https://k6.io/ ? (Full disclosure: I'm one of the maintainers.) It allows you to write load/performance tests in JS, commit them to your repo, easily automate them in CI, send metrics to several backends, use protocols besides HTTP, with a modern CLI, and many more features. There's also a Postman-to-k6 converter[1]. The conversion might not be perfect, but it will give you a head start. Note that the k6 philosophy is for developers to write these tests, similarly to how you write unit/integration tests, and to break the classic QA-dev cycle. I don't want to steal Hurl's thunder, it does look great, but it's limited in features compared to existing peformance testing tools, and I'd personally rather write tests in a programming language, than in a bespoke text format. [1]: https://github.com/apideck-libraries/postman-to-k6 https://github.com/apideck-libraries/postman-to-k6
- gavinray 3y ago
  If you maintain k6, can I log a complaint? I feel like I used to be able to run the CLI with just an URL and it would issue GET requests to it -- without needing a .js script to run Am I crazy, did this never exist? Please bring back/add this feature
  - imiric 3y ago
    I think you're mistaking k6 for another tool. k6 has only supported writing tests in JS. There are several other tools that are better suited for this single URL use case: ab, wrk, even Hurl. I doubt this will ever be supported in k6, since the focus is on scripting test logic, not firing off requests to a single URL (though this is certainly possible with a simple script).
- FridgeSeal 3y ago
  > but it's limited in features compared to existing peformance testing tools, and I'd personally rather write tests in a programming language, than in a bespoke text format. I’d hardly call hurl’s config bespoke, it basically amounts to HTTP codes and some header stuff, which basically everyone is familiar with. From personal experience a couple of weeks ago I needed some http tests. I got a CI-ready Hurl config working on some endpoints for work sorted within like, 5 minutes of downloading it. In comparison I opened the page for K6, saw the “write tests using JS” and basically noped out immediately,

digikata 3y ago

I have been using it for API testing manual and in CI of one of our services and it's been very nice. You can basically put a series of http exchanges for a workflow per hurl file and get a nice test suite that also checks directly into git.

vidyesh 3y ago

For a team using VSCode you can try the vscode-restclient[1] But really Hurl looks really interesting, being editor agnostic is the best solution for your problem, I agree. [1] https://github.com/Huachao/vscode-restclient https://github.com/Huachao/vscode-restclient

kxrm 3y ago
I use the vscode-restclient and my primary reason is the conversational flows you can build against an API. Does Hurl support this? If so I would absolutely switch. All I would need to complete the experience is a plugin to do highlighting and integration with the Hurl files.
- steve_adams_86 3y ago
  digikata’s comment above suggests it is possible. It would be great to have something that isn’t attached to the editor of choice.
  - kxrm 3y ago
    Indeed it does with https://hurl.dev/docs/capturing-response.html https://hurl.dev/docs/capturing-response.html Also for VS Code integration specifically there is https://marketplace.visualstudio.com/items?itemName=JacobPfeifer.pfeifer-hurl https://marketplace.visualstudio.com/items?itemName=JacobPfe...

ushakov 3y ago

There’s also Step CI: https://stepci.com https://stepci.com (I’m one of the authors) Hurl is brilliant though

randomsofr 3y ago
I was just looking at this. We might use it at my company, but i was wondering, is this funded? Or is this just an open source side project for you guys.
- ushakov 3y ago
  Self-funded atm.

jicea 3y ago

Hi maintainer here! Thanks a lot for the kind words!

convalescindrey 3y ago
Great project! A comment I'd have is that it's quite hard to find out who authored this fine piece of software.

kbenson 3y ago

Not that I necessarily think it's best to stay with Postman, but have you looked at newman, which is the CLI runner for postman configs? We had postman as a test suite for something (which is more an API than an app), and I got tired of having to deal with setting up extra steps to test and of exporting the postman config to save in the repo, so I put newman on the test system and just run against the config directly in the test environment and check the output. I don't necessarily recommend editing the postman config json directly to set up new tests as it's a PITA, but it's generally what I do so I don't need to keep importing and exporting it with Postman. A tool designed for working with on the shell is likely better than what I'm doing with newman (since the config is not the most accessible), but it also meant I didn't need to rewrite a bunch of existing tests and verify they actually did the same thing.

jug6ernaut 3y ago
The problem Newman/postman have is the same for every GUI based testing application. They almost always produce non human readable config files. Making any kind of code review of such changes at best extremely painful and at worst impossible. IMO any testing tool that does not save it's test classes in a human readable format is DoA.

cmgriffing 3y ago

I personally like the approach of defining things via an OpenAPI and then using Dredd to validate the spec against itself. Even for tools that generate the spec from source code, it is usually still possible for user error to define the metadata for an endpoint incorrectly. Dredd catches that.

TheBigRoomXXL 3y ago
I also think that validating an API against it's OpenAPI schema is a great methodology. You should checkout schemathesis, it's fantastic for doing that. https://github.com/schemathesis/schemathesis https://github.com/schemathesis/schemathesis

recroad 3y ago

The bigger problem here is that you have QA, not the lack of tooling.

dijit 3y ago
The bigger problem here is that you have devs, devs make bugs. if you don't have devs there are no bugs, problem solved.
- convalescindrey 3y ago
  If devs have to do QA themselves, many issues magically disappear.
  - dijit 3y ago
    Honestly that sounds like making pilots build aircraft engines. these are different disciplines that deserve to be done well. Maybe I am biased because I spent the last 10 years in gamedev, or maybe this is another push to make devs do basically everything tech related: but if a developer tells me a feature is done I always look to QA for a nod. That nod rarely comes, the feature is not done, the developer merely got it to work on their machine.
    - convalescindrey 3y ago
      Then your developer should learn that "done" means more than "works on their machine". The difference between devs and dedicated QA people is that devs know the dark corners of the implementation. They know the edge cases and scenarios that they struggled with getting right. That's where most testing focus has to be. QA doesn't know any of that. They can play through some scenarios that are the expected ones from the spec, perhaps have some hunch of what could be the tricky cases, but they don't actually know. I love seeing all those downvotes for my GP comment. That's all the people working at companies where stuff that devs feel is below them is dumped on QA people, which is the main reason my company is well-known for shipping top quality (without having a QA dept) while our competitors struggle with quality (despite having a QA dept). That's all the evidence I need to back up my claim.

imiric 3y ago

Have you tried https://k6.io/ https://k6.io/ ? (Full disclosure: I'm one of the maintainers.) It allows you to write load/performance tests in JS, commit them to your repo, easily automate them in CI, send metrics to several backends, use protocols besides HTTP, with a modern CLI, and many more features. There's also a Postman-to-k6 converter[1]. The conversion might not be perfect, but it will give you a head start. Note that the k6 philosophy is for developers to write these tests, similarly to how you write unit/integration tests, and to break the classic QA-dev cycle. I don't want to steal Hurl's thunder, it does look great, but it's limited in features compared to existing peformance testing tools, and I'd personally rather write tests in a programming language, than in a bespoke text format. [1]: https://github.com/apideck-libraries/postman-to-k6 https://github.com/apideck-libraries/postman-to-k6

gavinray 3y ago
If you maintain k6, can I log a complaint? I feel like I used to be able to run the CLI with just an URL and it would issue GET requests to it -- without needing a .js script to run Am I crazy, did this never exist? Please bring back/add this feature
- imiric 3y ago
  I think you're mistaking k6 for another tool. k6 has only supported writing tests in JS. There are several other tools that are better suited for this single URL use case: ab, wrk, even Hurl. I doubt this will ever be supported in k6, since the focus is on scripting test logic, not firing off requests to a single URL (though this is certainly possible with a simple script).
FridgeSeal 3y ago
> but it's limited in features compared to existing peformance testing tools, and I'd personally rather write tests in a programming language, than in a bespoke text format. I’d hardly call hurl’s config bespoke, it basically amounts to HTTP codes and some header stuff, which basically everyone is familiar with. From personal experience a couple of weeks ago I needed some http tests. I got a CI-ready Hurl config working on some endpoints for work sorted within like, 5 minutes of downloading it. In comparison I opened the page for K6, saw the “write tests using JS” and basically noped out immediately,