10 ms·

Robot Framework: generic open source automation framework

163 points by wiradikusuma 4y ago

dang 4y ago
Related: Robot Framework – test automation in Python - https://news.ycombinator.com/item?id=10631074 https://news.ycombinator.com/item?id=10631074 - Nov 2015 (5 comments)
ithinkso 4y ago
Now that's a name I haven't heard in a long time... Burn. It. With. Fire. I was at Nokia and we used robot, what an absolute pain it was
- keewee7 4y ago
  The hardware people at my job love it.
  - ithinkso 4y ago
    Maybe we used it incorrectly but for testing SW on target it's the worst thing I have ever experienced. The 'human readable' keywords are a hidden evil no one should meet. And the sheer amount of them, matched with random Setups and Teardowns, wierd syntax workarounds to do simple stuff, performance...
    - z33k 4y ago
      What would you rather use for end-to-end acceptance testing? Pabot runs multiple robot tests in parallel, meaning dozens of tests can run in a few minutes. It seems acceptable if you start with the assumption that acceptance testing part of the pipeline doesn't need to be run on every commit.
    - cvak 4y ago
      Yeah this is my experience as well, I understand the appeal, hell I was even the one to frontrun/poc it 6 years ago. But in the end it's just worse dev experience, no sane Keywords autocomplete/lookup in any IDE out there(by that time). In the end you end up programming inside english like DSL with weird quirks, that's somewhere connected to python, with different variable syntax, and not really good debugging. I guess everyone has to pass through that stage at least once, but I will never go back to it.
      - fabioz 4y ago
        Well, nowadays Robot Framework Language Server (which I develop) does have autocomplete/semantic analysis/interactive console/debugger (which can debug Python and Robot Framework at the same time) and a bunch of other things, so, I guess things did improve a bit in the meanwhile...
        cvak 4y ago
        I suspected LSP would be awesome for it, but nothing existed like 3 years ago - kudos to you sir, I think it would make the experience 10x better.
- pjmlp 4y ago
  Same here, hello from ex-Nokia Networks GmbH and NectAct migration into Java project. :) I don't miss Robot at all.
- deleted 4y ago
  [deleted]
- fabioz 4y ago
  Well, I don't know how things were back then, but Robot Framework had many changes in the latest times, so, it's possible things improved from there (thus your past experiences may not match the current state of affairs). I'd say that it does have some quirks, but if well used it can be a good tool to have in your toolbox. Disclaimer: I'm currently working on the Robot Framework Language Server extension for VSCode (by Robocorp, which uses Robot Framework to work with RPA -- albeit Robocorp supports well both Python as well as Robot Framework in the platform, there's a high traction on the Robot Framework side and many users enjoy it vs using just Python).
  - pjmlp 4y ago
    Back in the early Robot days you would write the tests on a tabular form, HTML based. The Python code would parse those HTML tables to execute the code, lovely.
    - p_l 4y ago
      Most common was afaik Excel with export to CSV or TSV, though some people started with Robot IDE (circa 2015-2016 was my last contact)
      - pjmlp 4y ago
        My last contact with Robot was in 2007.
- rasjani 4y ago
  I was also in nokia and we where pitched to use RF - at the time, we (well, i) made a decision of not to use it - mainly because our actual testing tool was implemented in ruby and using remote library wasnt really possible for our scenario and also i didnt like the test data format then (we where pitched to use a frigin EXCEL or html tables) .. Things have changed a lot from those times though and I am somewhat happy camper with robot in its current state. Its not without its warts but definitely has come a long way since "nokia days".
- djvdq 4y ago
  Also ex-Nokia employee (in PL) here. I see all of us share the same "love" to RF
frozencell 4y ago
- toxicFork 4y ago
  I think Russia may have invaded Ukraine sometime recently
  - frozencell 4y ago
    Recently, it's at least 2014. And they are about 15 countries in ongoing war now (https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflicts https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflict...).
mkl95 4y ago
A QA consultancy used this tool at my previous gig. They were extremely unproductive, as in several months to deliver a few tests. Honestly if you know your Python you are better off with Selenium, which is probably what Robot uses under the hood anyway.
- hpaavola 4y ago
  RF is not in any way only web automation tool. Sure, you can use Selenium with it to automate web ui testing. Or you can use playwright. Or you can just not do web ui testing and do mobile ui testing, or REST, or SAP, or pretty much any other automation you wish. What RF provides is by far the best reporting, test tagging, test filtering, and test data injection compared to any other test automation framework. And you can automate pretty much anything using it.
  - Ekaros 4y ago
    I agree whole keyword driven thing is quite messy, but the reporting is the part I love about it. Actually being able to reason why things failed and where, if test case are semi-reasonably written.
- jtnag 4y ago
  Depending what RF is being used for, Selenium may be under the hood wrapped in multi-layers of keywords. In other cases, where testing is not web/REST based (protocol testing, HW testing), what is being used for executing logic is up to devs/testers.
- captnswing 4y ago
  Agree, except: friends, don't let friends use Selenium. use PlayWright!!
  - spikej 4y ago
    or cypess
    - 4y ago
jtnag 4y ago
Blast from the past, I was involved in development of IDE for RF while in Nokia.
hpaavola 4y ago
From QA perspective Robot Framework is by far the best tool there is. First of all, it's not web testing tool, it's not mobile testing tool, it's not REST API testing tool. Or any other specific testing tool. You can automate pretty much any testing activity using RF, including web, mobile and REST, but not in any way limited to those. The killer feature from quality assurance point of view is test tagging combined with really powerful way of selecting what to include in test run and getting good reports which are supported by many many tools. Another killer feature is dead simple test instrumentation. Regarding tagging. Let's take this example: *** Settings *** Force Tags feature-xyz Suite Setup Initialize Tests *** Variables *** ${test_environment} dev *** Test Cases *** Foobar [Tags] jira-id-001 jira-test-id-001 smoke No Operation Lorem [Tags] jira-id-002 No Operation Ipsum [Tags] jira-id-003 bug-in-jira-001 No Operation *** Keywords *** Initialize Tests Connect To Environment ${test_environment} I can select any combination of those test to be included in given test run, specify which environment to connect and send results automatically to Jira. This allows me to run only "smoke" tests against "Pull Request" environment when ever PR is opened. This also allows me to automatically run all tests every hour against "Dev" environment and submit results to Jira. Like this: robot --include smoke --variable test_environment:pr That would only run the one test tagged smoke and Connect To Environment would get the value "pr". robot -i feature-xyz . Would run all tests with tag feature-xyz (and in the example file that would be all tests) against dev environment. And then I could just `curl` the XML result file from the run to Jira (given it has XRAY installed) and Jira would automatically update all the Jira tickets in mentioned in the tags with the test results. If there is no Jira Test tagged in RF test, Jira would automatically create new Jira Test for me. And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc. That way, when you need to know what is you test coverage, just open Jira and see it yourself.
- rasjani 4y ago
  > And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc. Sadly, that same thing can be also a negative side.. There is really no proper way to expose the results if you are not using Jenkins as your main CI. Ofcourse one can use junit reports or whip up a own listener that will generate some sort of test reports for the given ci platform...
dcanelhas 4y ago
For other roboticists who might stumble upon this: Robot Framework seems to be an automated testing framework (for testing software)
chungus 4y ago
Fun to see this is still around! Used this in 2016, at a big bank. I did not enjoy it, but that can largely be attributed to the way management wanted us to use it. The tests had to be written in our native language (not English) which caused a lot of weird mixed language use behind the scenes. Then a manager would sign off on the test report, which we would deliver to him on paper.
- rasjani 4y ago
  The latest 6.0 release finaly adds ability to translate most buildin syntax and afaik, keywords.
Cupprum 4y ago
I wrote my bachelor thesis about Robot Framework. The biggest issue i had with Robot Framework was the lack of support in IDEs. Initially you were suppose to work on Robot Framework in RIDE (dedicated IDE build on top of Eclipse) or use a Eclipse extension. I believe both of these are not supported anymore. I like Vim, so i wrote a package, which helps users when developing with Robot Framework. I also created some pull requests for Vim ALE to also support rflint. We also used Robot Framework at work for automated testing, worked like charm.
- wraptile 4y ago
  There's a quite a popular extension called "Robot Code" for vscode: > An extension which brings support for RobotFramework to Visual Studio Code, including features like code completion, debugging, test explorer, refactoring and more! Have you tried it?
- hpaavola 4y ago
  RIDE is a totally independent RF editor, not written on top of any other IDE. RED is RF editor built on Eclipse. Don't use either of them. https://marketplace.visualstudio.com/items?itemName=robocorp.robotframework-lsp https://marketplace.visualstudio.com/items?itemName=robocorp... and https://plugins.jetbrains.com/plugin/16086-robot-framework-language-server https://plugins.jetbrains.com/plugin/16086-robot-framework-l... are the way to go. Personally I use SublimeText with the RF LSP server with this https://github.com/Harri/RobotFrameworkSyntax https://github.com/Harri/RobotFrameworkSyntax but that's not "batteries included" like the two other plugins.
- rasjani 4y ago
  @fabioz who also posted to this discussion has been building "language server protocol" support for robot framework for a while now and is being sponsored to do so by a company called Robocorp. RIDE is not at least officially "not maintained" but my POV is that the current maintainer might not be up to the task -- or at least doesn't have time to dedicate to its maintenance.
Cupprum 4y ago
The developer experience is fairly nice when it comes to Robot Framework. Nice thing is, that Robot Framework was originally designed as automation tool, so also people who do not know a lot about coding can get up to speed fairly easily.
Cupprum 4y ago
If you use Robot Framework and have a lot of tests, which can be executed in parallel. Take a look at pabot [1]. It is a parallel executor for Robot Framework. [1] = https://pabot.org https://pabot.org
lvncelot 4y ago
I am less than lukewarm on those sort of "Natural-Language-Like" DSLs that seem to mostly come out of the QA space. When I worked together with a QA team for automated testing, they used Cucumber, which uses Gherkin as an english-but-not-really DSL. Now the idea is obviously, that someone who doesn't know how to code can still describe and read test scenarios. But what ended up happening eventually is that I spent a lot of time writing code to support different Gherkin expressions, but then had to take over writing the Gherkin expressions as well, because even though Gherkin -looks- like english, it still behaves like a programming language. In the end, it would've been much easier for me to just write the tests in a programming language from the start, instead of using an abstraction layer that turned out to serve no purpose. What I took away from that frustrating experience is that a lot of people think that learning to code is learning a syntax of a programming language, and if you could just do away with that pesky syntax, everybody could code. But, obviously, the syntax ist just the initial barrier, and actually coding is crystallizing and formulating intent - something which programming languages are much better at than natural languages. But for some reason, this low/no-code idea never seems to die down, especially in some spaces, and I have the feeling I'm just missing something here.
- urxvtcd 4y ago
  Yeah, I share the sentiment. What it boils down to is that you need to write the test two times. The supposed feature that once written the keywords can be reused is mostly a joke. There will be few heavily-used keywords, and a really long tail of one-time keywords, which is a pain. I see a single advantage to these frameworks though. They force you to write a test in a specific way that looks like a real-world scenario with clear delineations between setup stages, assertions, and so on. Not that it's impossible to achieve with regular programming language, obviously.
  - rasjani 4y ago
    > The supposed feature that once written the keywords can be reused is mostly a joke. I've seen well maintained and designed cucumber / rspec / robotframework "code" and i've seen some really horrible shit. Keyword re-use is really a joke if you are not familiar with any good practices to actual software development. Granted, in "low code" scenarios, the worst selling point is "you don't need to know how to write code" - which imho is bullshit and from my pov, statements like i quoted echoes that. Disclaimer; i partake and contribute to RF ecosystem. And i do prefer to write my tests or process automation with it even thought i can write code in multiple languages..
Agoreddah 4y ago
I loved this tool :) ! Few years ago, as a junior I joined one global corporation to my first job and I started to use it with zero knowledge about the python or even testing at all. But learning curve was very easy & fast and as of time, we created more than 3000 E2E & integration tests for our business application, running through the Jenkins. The tests were written in a very human-readable format, so anybody at the team could easily understand the test and identify the issues. Reporting had nice (but quite slow) html & jquery page. Also, Robot editor was a recommended tool to write tests. After a time, when I was starting a startup with a few junior developers which lacked the expertise, we used this tool in combination with Selenium to crawl data. It was fast & easy. Always glad to see this tool is still alive. Good job Pekka Klarck!
screwgoth 4y ago
Robot has it's peak maybe a couple of years ago. Although my company still uses it extensively there are other Automation frameworks that are proving to be better, like , say Cucumber, Cypress, etc. But why is Robot Framework being mentioned and getting attention now? It's not like it's something new and doing anything different from the existing frameworks.
- hpaavola 4y ago
  Cucumber does not really provide anything RF does not, except for Ruby houses. And Cypress is web automation tool, RF being generic one. One could write Cypress library for RF, but as there already are Selenium and Playwright libraries for RF, I don't think it will happen anytime soon at least.
miohtama 4y ago
As an interesting note that it is not surprise that half of showcased business logos are from Finland. Robot Framework is (was?) heavily used by Nokia and then spread from there to its subcontractor network.
- SSLy 4y ago
  Still is.
- rasjani 4y ago
  If i remember right, RF was Pekka's thesis work inside Nokia and it was later open-sourced and a foundation was formed around it.
kh_hk 4y ago
*** Test Cases *** Write my test using a DSL Read examples of the DSL Write a basic test and see that it works Write a complex test and find something missing Learn quirks about DSL Implement missing things on the language that the DSL was implemented on Write part of the test in said language Write part of the test in DSL matching regexes Run the test And it works 9 out of 10 times Install flaky extension Add flaky tag to test And it is green [Teardown] Reflect about my life as a software developer
xylix 4y ago
I develop tooling around Robot Framework as my main job, personally I feel biggest tradeoff of RF in Python world is that it competes against Pytest, which is my personal favourite testing framework out of any I have tried. In JavaScript projects I don't have such a favourite, Jest and Mocha have been okayish when I have tried them but didn't really spark joy. In multilanguage projects (like the Playwright based robotframework-browser which I have been developing) I have enjoyed writing integration tests with Robot Framework. I don't have excessive experience with other comparable QA tools (like cucumber), but I would say Robot Framework's main advantage and disadvantage are both the fact that it is constantly so close to python. (E.g. you can easily do in-line python expressions, but if you were to have a team working with Robot Framework where some are lacking Python competence those might become very confusing pretty fast.) Also for writing advanced libraries Python usage is pretty much mandatory. But I guess it's pretty rare for a DSL to support writing advanced extensions with the same DSL.
alex_suzuki 4y ago
I remember this from a gig I did a while ago. Customer wanted to establish their own automated QA and thought Robot Framework was a good fit, because the language „looks easy“. The goal was to enable IT personnel that lacked a technical background, with no programming experience, to write tests. I was sceptical from the start, and was eventually proven right (I did not enjoy this, it created more not-so-enjoyable work for me). Test code was flaky to the point of being useless, and contained alot of redundancy. IMHO it is really hard to write good tests without having a bit of a „programmers mindset“.
- antman 4y ago
  "IT personnel that lacked a technical background" : Really interested on what the recruitment team was thinking!
  - alex_suzuki 4y ago
    Think support staff, PMs, etc.
- majkinetor 4y ago
  Exactly. You need devs to write good tests in general.
manykarim 4y ago
Robot Framework is a general purpose automation framework. You can use it to automate - Web Applications (with Selenium or Playwright) - Rest APIs - Desktop Applications (Java Swing, WPF, SAP, ..) - All kinds of hardware It offers control-structures like most programming languages (IF/ELSE/FOR/WHILE/TRY/EXCEPT) The Extensions for VS Code or PyCharm/IntelliJ offer Code-Completion, step-wise Debugging, Linting. It is very hackable and can be extended using Python. It has a great API, allowing you to connect it to all kind of other tools. I guess a lot has happened since some of the people here used it. But I guess, if somebody is focused on using a single programming Language (like JavaScript) and is not open to learn Python or the RF Syntax I recommend to look into another native test framework
deng 4y ago
Used it extensively for a while. I think all in all, it's a nice, mature framework. The Gherkin stuff is completely optional, I'm not fond of it and never used it. Using Python, you can create a custom DSL tailored to your needs, so that you can write tests in a tabular form. All your tests will need to adhere to this rigid tabular structure. Whether this makes sense, entirely depends on the circumstances. Compared to just using pytest, I think the main advantage is that tests can also be written and extended by people who are not familiar with Python. So if you have a separate QA team with people who are not programmers, this might make sense. Also, I did like that this rigid tabular structure made all tests very much alike, whereas when using pytest, you often end up with a hodge-podge of different ways to test things and people re-inventing the wheel all the time.
mydevlprplanet 4y ago
I have used RF for many years now at different companies. We use it for automated integration tests, it does not matter in which language your application is written. I do admit that it requires some programming skills, especially for the keywords. Choose the keywords wisely and ensure that the QA people know how to use them. We also use it as part of regression testing and even for acceptance tests with the customer.
elevation 4y ago
I test embedded devices with RF. You write most of your test drivers in python, RF simply orchestrates the tests and collects the results into a report, which works well in simple QA scenarios. Robot lacks features you'd need to support larger-scale embedded test, such as with a device farm, where you need the concept of tests leasing resources. If you build a test stand with 10 testable devices and need to run a suite of 100 tests, it's ideal to run the tests in parallel as devices become available. Some test setups might require more than one instance of a testable device, or instances of more than one type of device (for example, to ensure that a V2 product can still interoperate with a V3 product.) Robot doesn't really support this. While a feature to support this could be made extremely general (resource classes, instances, and leases) the RF developers have been uninterested in incorporating this aspect of test into their framework. The result is everyone who does even mid scale embedded device testing has to writes their own. Another complaint about robot framework is that when you have an expensive setup like a 4 minute flashing operation, you don't want to repeat it more than necessary. So in a file, you might make the expensive setup a suite-level setup, followed by the tests cases that depend upon it. When this file grows, you might want to refactor it into a multi-file test suite in it's own sub-directory. However, these tests no longer share a suite scope (because robot's "suite scope" is actually a file scope" for legacy reasons) so in practice you may need to tolerate 3000 line files to avoid long setups.
- 2rsf 4y ago
  > the RF developers have been uninterested in incorporating this aspect of test Have they been asked? I had the same problem and had different solutions built in house with different levels of success
  - elevation 4y ago
    Pekka has previously expressed that test resource management should not be included in the core and is better implemented as separate keywords. One example would be the limited resource management that Pabot supports (basically a single global semaphore that tests can increment/decrement through a pair of keywords.) The problem with 3-rd party keyword-based solutions is that the resource acquisition becomes a part of the test process and could a test report to look like "ran for 6 minutes, then failed" when what really happened is the actual test couldn't be run at all because resources weren't found. It would be nice to be able to queue tests based on the resources they'll need. Also, another feature that's more difficult with custom implementations is static analysis of the test base. If resource requests were a declarative part of the RF language (much like tags) you could do things like "run all tests that exercise resource X" or "run all tests except those which require more than one Y". This is technically possible with a keyword library but requires an additional global lexing pass of the entire test base, and this is the kind of logic you don't want to decouple from RF and it's test discovery behavior. Perhaps Pekka can be won over in the future.
dbcurtis 4y ago
Stop calling things that are not robots, robots. Does this framework stream sensor data? No. Does this framework control actuators? No. Does this framework represent an actuated device’s configuration space? No. Does this framework perform collision detection? No. Does this framework keep an obstacle map? No. Does this framework have any planners? No. Just stop already. Stop calling things a robot when they are not.
- yaseer 4y ago
  They're referring to "Robotic Process Automation" (RPA). I agree the name is a misnomer, but it's now part of the Enterprise Lexicon, which is filled with jargon. I run an RPA startup, but as much as I hate the term (e.g:https://news.ycombinator.com/item?id=30755118 https://news.ycombinator.com/item?id=30755118) , once something becomes part of common language patterns, you can't unshift it. I hate the modern usage of 'AI' as ML, but I can't change the new definition.
  - dbcurtis 4y ago
    Pardon me on advance for continuing to yell at clouds.
- sheepybloke 4y ago
  I was really excited for a good new ROS like system or something, but instead I got automated testing T.T
otikik 4y ago
Why is it "generic" and what does that mean? I could not find it in the docs
dhanna 4y ago
My only experience with Robot was at Juniper Networks, where it was a poorly maintained bastard QA org tech. Would have been infinitely better if DEV’s and QA had a shared repo of python scripts.
999900000999 4y ago
No. Write some code, or get playwright to generate it for you. Don't use DSLs if you value your sanity.
davemp 4y ago
When I started writing tests for my current project, I looked at what was available and decided forgo existing testing frameworks. I haven't been disappointed. Writing code to call a bunch of test functions then generate a report is really not hard and having control over the whole thing is nice.
dusanh 4y ago
We use Robot at my current job and it works flawlessly. It can test the web front-end (Browser, wrapper for Playwright), the REST APIs (REST, RestInstance), CLI stuff and much more. We selected it specifically for its versatility and the DSL is easy enough so even less technically adept testers can pick it up quickly.
hkfilho 4y ago
I've been using RF for almost 8 years now. I came in touch with it, in the beginning with my C/C++ developer mind: "...what an overhead of work...". But soon a realize I was wrong and start to notice the power of writing tests in robot. And how it allowed to easy the understand of what operation I did on code in the past... like 6 months before, 1 year before... and also the ability to understand more clearly how other teams' features worked and to fix their test and features many times, because we more clearly wrote test using Gherkins in RF... and see how new team members could ramp MUCH more easily inside our environments, by reading tests, most of time. I was even able to implant RF inside some other business after that, bringing together that culture of writing clear tests with RF. Some of those places had the heavy culture of writing top-down requirements precisely. On those places, we integrate RF documentation and test case procedure on our process, generating final documentation using RF test case information. Since them I'm helping to disseminate and evangelizing for the RF use. I saw system analysts embracing the RF, writing the first version of test cases for requirement in matter of minutes. RF provided us with much faster peace in maintaining some very complex environments. We could grab requirements, and write down cases, that latter could be implemented, edited, removed or adjusted. Whatever is the test environment, testing it is a 'living' environment, and need catering from developers. RF had helped greatly maintaining such places, to produce great reports, to create this live ecosystem easy for uses. Is perfect? Surely No. But is much better that all other tools available around. Is open source, easy to expand, allow use of natural language. And when well used, allow us to greatly improve our QA and DEV lives :D

163 points by wiradikusuma 4y ago

dang 4y ago

Related: Robot Framework – test automation in Python - https://news.ycombinator.com/item?id=10631074 https://news.ycombinator.com/item?id=10631074 - Nov 2015 (5 comments)

ithinkso 4y ago

Now that's a name I haven't heard in a long time... Burn. It. With. Fire. I was at Nokia and we used robot, what an absolute pain it was

keewee7 4y ago
The hardware people at my job love it.
- ithinkso 4y ago
  Maybe we used it incorrectly but for testing SW on target it's the worst thing I have ever experienced. The 'human readable' keywords are a hidden evil no one should meet. And the sheer amount of them, matched with random Setups and Teardowns, wierd syntax workarounds to do simple stuff, performance...
  - z33k 4y ago
    What would you rather use for end-to-end acceptance testing? Pabot runs multiple robot tests in parallel, meaning dozens of tests can run in a few minutes. It seems acceptable if you start with the assumption that acceptance testing part of the pipeline doesn't need to be run on every commit.
  - cvak 4y ago
    Yeah this is my experience as well, I understand the appeal, hell I was even the one to frontrun/poc it 6 years ago. But in the end it's just worse dev experience, no sane Keywords autocomplete/lookup in any IDE out there(by that time). In the end you end up programming inside english like DSL with weird quirks, that's somewhere connected to python, with different variable syntax, and not really good debugging. I guess everyone has to pass through that stage at least once, but I will never go back to it.
    - fabioz 4y ago
      Well, nowadays Robot Framework Language Server (which I develop) does have autocomplete/semantic analysis/interactive console/debugger (which can debug Python and Robot Framework at the same time) and a bunch of other things, so, I guess things did improve a bit in the meanwhile...
      - cvak 4y ago
        I suspected LSP would be awesome for it, but nothing existed like 3 years ago - kudos to you sir, I think it would make the experience 10x better.
pjmlp 4y ago
Same here, hello from ex-Nokia Networks GmbH and NectAct migration into Java project. :) I don't miss Robot at all.
deleted 4y ago
[deleted]
fabioz 4y ago
Well, I don't know how things were back then, but Robot Framework had many changes in the latest times, so, it's possible things improved from there (thus your past experiences may not match the current state of affairs). I'd say that it does have some quirks, but if well used it can be a good tool to have in your toolbox. Disclaimer: I'm currently working on the Robot Framework Language Server extension for VSCode (by Robocorp, which uses Robot Framework to work with RPA -- albeit Robocorp supports well both Python as well as Robot Framework in the platform, there's a high traction on the Robot Framework side and many users enjoy it vs using just Python).
- pjmlp 4y ago
  Back in the early Robot days you would write the tests on a tabular form, HTML based. The Python code would parse those HTML tables to execute the code, lovely.
  - p_l 4y ago
    Most common was afaik Excel with export to CSV or TSV, though some people started with Robot IDE (circa 2015-2016 was my last contact)
    - pjmlp 4y ago
      My last contact with Robot was in 2007.
rasjani 4y ago
I was also in nokia and we where pitched to use RF - at the time, we (well, i) made a decision of not to use it - mainly because our actual testing tool was implemented in ruby and using remote library wasnt really possible for our scenario and also i didnt like the test data format then (we where pitched to use a frigin EXCEL or html tables) .. Things have changed a lot from those times though and I am somewhat happy camper with robot in its current state. Its not without its warts but definitely has come a long way since "nokia days".
djvdq 4y ago
Also ex-Nokia employee (in PL) here. I see all of us share the same "love" to RF

frozencell 4y ago

toxicFork 4y ago
I think Russia may have invaded Ukraine sometime recently
- frozencell 4y ago
  Recently, it's at least 2014. And they are about 15 countries in ongoing war now (https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflicts https://en.wikipedia.org/wiki/List_of_ongoing_armed_conflict...).

mkl95 4y ago

A QA consultancy used this tool at my previous gig. They were extremely unproductive, as in several months to deliver a few tests. Honestly if you know your Python you are better off with Selenium, which is probably what Robot uses under the hood anyway.

hpaavola 4y ago
RF is not in any way only web automation tool. Sure, you can use Selenium with it to automate web ui testing. Or you can use playwright. Or you can just not do web ui testing and do mobile ui testing, or REST, or SAP, or pretty much any other automation you wish. What RF provides is by far the best reporting, test tagging, test filtering, and test data injection compared to any other test automation framework. And you can automate pretty much anything using it.
- Ekaros 4y ago
  I agree whole keyword driven thing is quite messy, but the reporting is the part I love about it. Actually being able to reason why things failed and where, if test case are semi-reasonably written.
jtnag 4y ago
Depending what RF is being used for, Selenium may be under the hood wrapped in multi-layers of keywords. In other cases, where testing is not web/REST based (protocol testing, HW testing), what is being used for executing logic is up to devs/testers.
captnswing 4y ago
Agree, except: friends, don't let friends use Selenium. use PlayWright!!
- spikej 4y ago
  or cypess
  - 4y ago

jtnag 4y ago

Blast from the past, I was involved in development of IDE for RF while in Nokia.

hpaavola 4y ago

From QA perspective Robot Framework is by far the best tool there is. First of all, it's not web testing tool, it's not mobile testing tool, it's not REST API testing tool. Or any other specific testing tool. You can automate pretty much any testing activity using RF, including web, mobile and REST, but not in any way limited to those. The killer feature from quality assurance point of view is test tagging combined with really powerful way of selecting what to include in test run and getting good reports which are supported by many many tools. Another killer feature is dead simple test instrumentation. Regarding tagging. Let's take this example: *** Settings *** Force Tags feature-xyz Suite Setup Initialize Tests *** Variables *** ${test_environment} dev *** Test Cases *** Foobar [Tags] jira-id-001 jira-test-id-001 smoke No Operation Lorem [Tags] jira-id-002 No Operation Ipsum [Tags] jira-id-003 bug-in-jira-001 No Operation *** Keywords *** Initialize Tests Connect To Environment ${test_environment} I can select any combination of those test to be included in given test run, specify which environment to connect and send results automatically to Jira. This allows me to run only "smoke" tests against "Pull Request" environment when ever PR is opened. This also allows me to automatically run all tests every hour against "Dev" environment and submit results to Jira. Like this: robot --include smoke --variable test_environment:pr That would only run the one test tagged smoke and Connect To Environment would get the value "pr". robot -i feature-xyz . Would run all tests with tag feature-xyz (and in the example file that would be all tests) against dev environment. And then I could just `curl` the XML result file from the run to Jira (given it has XRAY installed) and Jira would automatically update all the Jira tickets in mentioned in the tags with the test results. If there is no Jira Test tagged in RF test, Jira would automatically create new Jira Test for me. And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc. That way, when you need to know what is you test coverage, just open Jira and see it yourself.

rasjani 4y ago
> And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc. Sadly, that same thing can be also a negative side.. There is really no proper way to expose the results if you are not using Jenkins as your main CI. Ofcourse one can use junit reports or whip up a own listener that will generate some sort of test reports for the given ci platform...

dcanelhas 4y ago

For other roboticists who might stumble upon this: Robot Framework seems to be an automated testing framework (for testing software)

chungus 4y ago

Fun to see this is still around! Used this in 2016, at a big bank. I did not enjoy it, but that can largely be attributed to the way management wanted us to use it. The tests had to be written in our native language (not English) which caused a lot of weird mixed language use behind the scenes. Then a manager would sign off on the test report, which we would deliver to him on paper.

rasjani 4y ago
The latest 6.0 release finaly adds ability to translate most buildin syntax and afaik, keywords.

Cupprum 4y ago

I wrote my bachelor thesis about Robot Framework. The biggest issue i had with Robot Framework was the lack of support in IDEs. Initially you were suppose to work on Robot Framework in RIDE (dedicated IDE build on top of Eclipse) or use a Eclipse extension. I believe both of these are not supported anymore. I like Vim, so i wrote a package, which helps users when developing with Robot Framework. I also created some pull requests for Vim ALE to also support rflint. We also used Robot Framework at work for automated testing, worked like charm.

wraptile 4y ago
There's a quite a popular extension called "Robot Code" for vscode: > An extension which brings support for RobotFramework to Visual Studio Code, including features like code completion, debugging, test explorer, refactoring and more! Have you tried it?
hpaavola 4y ago
RIDE is a totally independent RF editor, not written on top of any other IDE. RED is RF editor built on Eclipse. Don't use either of them. https://marketplace.visualstudio.com/items?itemName=robocorp.robotframework-lsp https://marketplace.visualstudio.com/items?itemName=robocorp... and https://plugins.jetbrains.com/plugin/16086-robot-framework-language-server https://plugins.jetbrains.com/plugin/16086-robot-framework-l... are the way to go. Personally I use SublimeText with the RF LSP server with this https://github.com/Harri/RobotFrameworkSyntax https://github.com/Harri/RobotFrameworkSyntax but that's not "batteries included" like the two other plugins.
rasjani 4y ago
@fabioz who also posted to this discussion has been building "language server protocol" support for robot framework for a while now and is being sponsored to do so by a company called Robocorp. RIDE is not at least officially "not maintained" but my POV is that the current maintainer might not be up to the task -- or at least doesn't have time to dedicate to its maintenance.

Cupprum 4y ago

The developer experience is fairly nice when it comes to Robot Framework. Nice thing is, that Robot Framework was originally designed as automation tool, so also people who do not know a lot about coding can get up to speed fairly easily.

Cupprum 4y ago

If you use Robot Framework and have a lot of tests, which can be executed in parallel. Take a look at pabot [1]. It is a parallel executor for Robot Framework. [1] = https://pabot.org https://pabot.org

lvncelot 4y ago

I am less than lukewarm on those sort of "Natural-Language-Like" DSLs that seem to mostly come out of the QA space. When I worked together with a QA team for automated testing, they used Cucumber, which uses Gherkin as an english-but-not-really DSL. Now the idea is obviously, that someone who doesn't know how to code can still describe and read test scenarios. But what ended up happening eventually is that I spent a lot of time writing code to support different Gherkin expressions, but then had to take over writing the Gherkin expressions as well, because even though Gherkin -looks- like english, it still behaves like a programming language. In the end, it would've been much easier for me to just write the tests in a programming language from the start, instead of using an abstraction layer that turned out to serve no purpose. What I took away from that frustrating experience is that a lot of people think that learning to code is learning a syntax of a programming language, and if you could just do away with that pesky syntax, everybody could code. But, obviously, the syntax ist just the initial barrier, and actually coding is crystallizing and formulating intent - something which programming languages are much better at than natural languages. But for some reason, this low/no-code idea never seems to die down, especially in some spaces, and I have the feeling I'm just missing something here.

urxvtcd 4y ago
Yeah, I share the sentiment. What it boils down to is that you need to write the test two times. The supposed feature that once written the keywords can be reused is mostly a joke. There will be few heavily-used keywords, and a really long tail of one-time keywords, which is a pain. I see a single advantage to these frameworks though. They force you to write a test in a specific way that looks like a real-world scenario with clear delineations between setup stages, assertions, and so on. Not that it's impossible to achieve with regular programming language, obviously.
- rasjani 4y ago
  > The supposed feature that once written the keywords can be reused is mostly a joke. I've seen well maintained and designed cucumber / rspec / robotframework "code" and i've seen some really horrible shit. Keyword re-use is really a joke if you are not familiar with any good practices to actual software development. Granted, in "low code" scenarios, the worst selling point is "you don't need to know how to write code" - which imho is bullshit and from my pov, statements like i quoted echoes that. Disclaimer; i partake and contribute to RF ecosystem. And i do prefer to write my tests or process automation with it even thought i can write code in multiple languages..

Agoreddah 4y ago

I loved this tool :) ! Few years ago, as a junior I joined one global corporation to my first job and I started to use it with zero knowledge about the python or even testing at all. But learning curve was very easy & fast and as of time, we created more than 3000 E2E & integration tests for our business application, running through the Jenkins. The tests were written in a very human-readable format, so anybody at the team could easily understand the test and identify the issues. Reporting had nice (but quite slow) html & jquery page. Also, Robot editor was a recommended tool to write tests. After a time, when I was starting a startup with a few junior developers which lacked the expertise, we used this tool in combination with Selenium to crawl data. It was fast & easy. Always glad to see this tool is still alive. Good job Pekka Klarck!

screwgoth 4y ago

Robot has it's peak maybe a couple of years ago. Although my company still uses it extensively there are other Automation frameworks that are proving to be better, like , say Cucumber, Cypress, etc. But why is Robot Framework being mentioned and getting attention now? It's not like it's something new and doing anything different from the existing frameworks.

hpaavola 4y ago
Cucumber does not really provide anything RF does not, except for Ruby houses. And Cypress is web automation tool, RF being generic one. One could write Cypress library for RF, but as there already are Selenium and Playwright libraries for RF, I don't think it will happen anytime soon at least.

miohtama 4y ago

As an interesting note that it is not surprise that half of showcased business logos are from Finland. Robot Framework is (was?) heavily used by Nokia and then spread from there to its subcontractor network.

SSLy 4y ago
Still is.
rasjani 4y ago
If i remember right, RF was Pekka's thesis work inside Nokia and it was later open-sourced and a foundation was formed around it.

kh_hk 4y ago

*** Test Cases *** Write my test using a DSL Read examples of the DSL Write a basic test and see that it works Write a complex test and find something missing Learn quirks about DSL Implement missing things on the language that the DSL was implemented on Write part of the test in said language Write part of the test in DSL matching regexes Run the test And it works 9 out of 10 times Install flaky extension Add flaky tag to test And it is green [Teardown] Reflect about my life as a software developer

xylix 4y ago

I develop tooling around Robot Framework as my main job, personally I feel biggest tradeoff of RF in Python world is that it competes against Pytest, which is my personal favourite testing framework out of any I have tried. In JavaScript projects I don't have such a favourite, Jest and Mocha have been okayish when I have tried them but didn't really spark joy. In multilanguage projects (like the Playwright based robotframework-browser which I have been developing) I have enjoyed writing integration tests with Robot Framework. I don't have excessive experience with other comparable QA tools (like cucumber), but I would say Robot Framework's main advantage and disadvantage are both the fact that it is constantly so close to python. (E.g. you can easily do in-line python expressions, but if you were to have a team working with Robot Framework where some are lacking Python competence those might become very confusing pretty fast.) Also for writing advanced libraries Python usage is pretty much mandatory. But I guess it's pretty rare for a DSL to support writing advanced extensions with the same DSL.

alex_suzuki 4y ago

I remember this from a gig I did a while ago. Customer wanted to establish their own automated QA and thought Robot Framework was a good fit, because the language „looks easy“. The goal was to enable IT personnel that lacked a technical background, with no programming experience, to write tests. I was sceptical from the start, and was eventually proven right (I did not enjoy this, it created more not-so-enjoyable work for me). Test code was flaky to the point of being useless, and contained alot of redundancy. IMHO it is really hard to write good tests without having a bit of a „programmers mindset“.

antman 4y ago
"IT personnel that lacked a technical background" : Really interested on what the recruitment team was thinking!
- alex_suzuki 4y ago
  Think support staff, PMs, etc.
majkinetor 4y ago
Exactly. You need devs to write good tests in general.

manykarim 4y ago

Robot Framework is a general purpose automation framework. You can use it to automate - Web Applications (with Selenium or Playwright) - Rest APIs - Desktop Applications (Java Swing, WPF, SAP, ..) - All kinds of hardware It offers control-structures like most programming languages (IF/ELSE/FOR/WHILE/TRY/EXCEPT) The Extensions for VS Code or PyCharm/IntelliJ offer Code-Completion, step-wise Debugging, Linting. It is very hackable and can be extended using Python. It has a great API, allowing you to connect it to all kind of other tools. I guess a lot has happened since some of the people here used it. But I guess, if somebody is focused on using a single programming Language (like JavaScript) and is not open to learn Python or the RF Syntax I recommend to look into another native test framework

deng 4y ago

Used it extensively for a while. I think all in all, it's a nice, mature framework. The Gherkin stuff is completely optional, I'm not fond of it and never used it. Using Python, you can create a custom DSL tailored to your needs, so that you can write tests in a tabular form. All your tests will need to adhere to this rigid tabular structure. Whether this makes sense, entirely depends on the circumstances. Compared to just using pytest, I think the main advantage is that tests can also be written and extended by people who are not familiar with Python. So if you have a separate QA team with people who are not programmers, this might make sense. Also, I did like that this rigid tabular structure made all tests very much alike, whereas when using pytest, you often end up with a hodge-podge of different ways to test things and people re-inventing the wheel all the time.

mydevlprplanet 4y ago

I have used RF for many years now at different companies. We use it for automated integration tests, it does not matter in which language your application is written. I do admit that it requires some programming skills, especially for the keywords. Choose the keywords wisely and ensure that the QA people know how to use them. We also use it as part of regression testing and even for acceptance tests with the customer.

elevation 4y ago

I test embedded devices with RF. You write most of your test drivers in python, RF simply orchestrates the tests and collects the results into a report, which works well in simple QA scenarios. Robot lacks features you'd need to support larger-scale embedded test, such as with a device farm, where you need the concept of tests leasing resources. If you build a test stand with 10 testable devices and need to run a suite of 100 tests, it's ideal to run the tests in parallel as devices become available. Some test setups might require more than one instance of a testable device, or instances of more than one type of device (for example, to ensure that a V2 product can still interoperate with a V3 product.) Robot doesn't really support this. While a feature to support this could be made extremely general (resource classes, instances, and leases) the RF developers have been uninterested in incorporating this aspect of test into their framework. The result is everyone who does even mid scale embedded device testing has to writes their own. Another complaint about robot framework is that when you have an expensive setup like a 4 minute flashing operation, you don't want to repeat it more than necessary. So in a file, you might make the expensive setup a suite-level setup, followed by the tests cases that depend upon it. When this file grows, you might want to refactor it into a multi-file test suite in it's own sub-directory. However, these tests no longer share a suite scope (because robot's "suite scope" is actually a file scope" for legacy reasons) so in practice you may need to tolerate 3000 line files to avoid long setups.

2rsf 4y ago
> the RF developers have been uninterested in incorporating this aspect of test Have they been asked? I had the same problem and had different solutions built in house with different levels of success
- elevation 4y ago
  Pekka has previously expressed that test resource management should not be included in the core and is better implemented as separate keywords. One example would be the limited resource management that Pabot supports (basically a single global semaphore that tests can increment/decrement through a pair of keywords.) The problem with 3-rd party keyword-based solutions is that the resource acquisition becomes a part of the test process and could a test report to look like "ran for 6 minutes, then failed" when what really happened is the actual test couldn't be run at all because resources weren't found. It would be nice to be able to queue tests based on the resources they'll need. Also, another feature that's more difficult with custom implementations is static analysis of the test base. If resource requests were a declarative part of the RF language (much like tags) you could do things like "run all tests that exercise resource X" or "run all tests except those which require more than one Y". This is technically possible with a keyword library but requires an additional global lexing pass of the entire test base, and this is the kind of logic you don't want to decouple from RF and it's test discovery behavior. Perhaps Pekka can be won over in the future.

dbcurtis 4y ago

Stop calling things that are not robots, robots. Does this framework stream sensor data? No. Does this framework control actuators? No. Does this framework represent an actuated device’s configuration space? No. Does this framework perform collision detection? No. Does this framework keep an obstacle map? No. Does this framework have any planners? No. Just stop already. Stop calling things a robot when they are not.

yaseer 4y ago
They're referring to "Robotic Process Automation" (RPA). I agree the name is a misnomer, but it's now part of the Enterprise Lexicon, which is filled with jargon. I run an RPA startup, but as much as I hate the term (e.g:https://news.ycombinator.com/item?id=30755118 https://news.ycombinator.com/item?id=30755118) , once something becomes part of common language patterns, you can't unshift it. I hate the modern usage of 'AI' as ML, but I can't change the new definition.
- dbcurtis 4y ago
  Pardon me on advance for continuing to yell at clouds.
sheepybloke 4y ago
I was really excited for a good new ROS like system or something, but instead I got automated testing T.T

otikik 4y ago

Why is it "generic" and what does that mean? I could not find it in the docs

dhanna 4y ago

My only experience with Robot was at Juniper Networks, where it was a poorly maintained bastard QA org tech. Would have been infinitely better if DEV’s and QA had a shared repo of python scripts.

999900000999 4y ago

No. Write some code, or get playwright to generate it for you. Don't use DSLs if you value your sanity.

davemp 4y ago

When I started writing tests for my current project, I looked at what was available and decided forgo existing testing frameworks. I haven't been disappointed. Writing code to call a bunch of test functions then generate a report is really not hard and having control over the whole thing is nice.

dusanh 4y ago

We use Robot at my current job and it works flawlessly. It can test the web front-end (Browser, wrapper for Playwright), the REST APIs (REST, RestInstance), CLI stuff and much more. We selected it specifically for its versatility and the DSL is easy enough so even less technically adept testers can pick it up quickly.

hkfilho 4y ago

I've been using RF for almost 8 years now. I came in touch with it, in the beginning with my C/C++ developer mind: "...what an overhead of work...". But soon a realize I was wrong and start to notice the power of writing tests in robot. And how it allowed to easy the understand of what operation I did on code in the past... like 6 months before, 1 year before... and also the ability to understand more clearly how other teams' features worked and to fix their test and features many times, because we more clearly wrote test using Gherkins in RF... and see how new team members could ramp MUCH more easily inside our environments, by reading tests, most of time. I was even able to implant RF inside some other business after that, bringing together that culture of writing clear tests with RF. Some of those places had the heavy culture of writing top-down requirements precisely. On those places, we integrate RF documentation and test case procedure on our process, generating final documentation using RF test case information. Since them I'm helping to disseminate and evangelizing for the RF use. I saw system analysts embracing the RF, writing the first version of test cases for requirement in matter of minutes. RF provided us with much faster peace in maintaining some very complex environments. We could grab requirements, and write down cases, that latter could be implemented, edited, removed or adjusted. Whatever is the test environment, testing it is a 'living' environment, and need catering from developers. RF had helped greatly maintaining such places, to produce great reports, to create this live ecosystem easy for uses. Is perfect? Surely No. But is much better that all other tools available around. Is open source, easy to expand, allow use of natural language. And when well used, allow us to greatly improve our QA and DEV lives :D