5 ms·

How Secret Used Automated Testing for Their Android Launch

38 points by weitingliu 11y ago

tomasien 11y ago
Secret did NOT get $35M in a Kickstarter campaign, it was venture capital. Or was that a joke I didn't get?
- herbig 11y ago
  Looks like it was edited on the sly: http://webcache.googleusercontent.com/search?q=cache:http://blog.testmunk.com/how-secret-used-automated-testing-for-their-android-launch/ http://webcache.googleusercontent.com/search?q=cache:http://...
  - chimeracoder 11y ago
    Better link, since Google will eventually update their cache: https://archive.is/RAKfE https://archive.is/RAKfE (Yes, I get the recursive irony of archiving a Google cache result).
matthewcanty 11y ago
I find the text hard to read on my screen. Does it have to be so gray?
CodeSheikh 11y ago
It was like a mobile phones orchestra. Amazeballs.
tdumitrescu 11y ago
Given I wait for 5 seconds Then I press view with id "splash_signin_text" Then I wait for 2 seconds Then I clear input field number 1 Then I wait ... Looks like someone never bothered understanding what Given/When/Then are for.
- recursive 11y ago
  I think that describes most people. What's wrong with this?
swanson 11y ago
This is not terribly difficult to get running locally (at a smaller scale). Square has a tool called Spoon[0] that makes it really easy to run a test suite across a bunch of devices. On a recent project, we had 5-7 devices plugged into a spare box that would run a test suite on every push. It worked out great and the ability to capture screenshots was really helpful for the design team to help spot strange UI issues. More details on the setup here: http://www.sep.com/sep-blog/2014/01/06/android-robotium-and-spoon-a-practical-guide-for-testing-on-android-devices/ http://www.sep.com/sep-blog/2014/01/06/android-robotium-and-... But if you need larger device coverage, Testmunk looks pretty awesome :) [0]: https://github.com/square/spoon https://github.com/square/spoon
deleted 11y ago
[deleted]
saclark11 11y ago
I appreciate TestMunk's goal to make automated mobile testing easier and more accessible, and I truly hope they succeed in that, but for the love of testing, please do not view that example feature file at the bottom as an example of proper Gherkin. It is riddled with implementation details and brittle explicit waits. To be fair, they did clearly state: "we generally recommend against fixed waits" but then please do not put this forth as an example to be copied. Scenarios should describe the behavior from a user perspective and leave out implementation details. You should not have to change the your Gherkin no matter how much the underlying implementation changes so long as the expected behavior remains the same. Not to mention the misuse of the Given/When/Then keywords and the "I logout" steps -- that is what before/after hooks are for. I also recognize TestMunk points out that this feature utilizes standard steps to get your first test cases going and to get some screenshots. They also advocate for the page object pattern, which is good, but it might be helpful to be clear that this is not how you'd actually want to write your Gherkin, whose true value comes when written with/for product owners, free of implementation details.
- StavrosK 11y ago
  Can someone explain to me the appeal of Gherkin and other such tools? I don't see why the example on that page are anything but code that wants to "look better", and just ends up introducing another thing to maintain in your application. How is writing a step in English and writing the English to translate it to code better than just writing the exact same steps in code to begin with?
  - Karunamon 11y ago
    Gherkin/Cucumber's big thing is that it lets the people who can't read code (i.e. project managers, suits, and customers) understand what's being tested. Usually along the lines of: "As a user, I want to buy things, so I should be able to put things in the cart and get an accurate total price" or "As a user, I want to view the site in my own language, given I have come from a German network address, I should get the German language site" This frees you from having to explain the finer points of cookies, or geoip, or any number of concepts to people who do not operate on a level that close to the bare metal. The translation isn't really error prone either and isn't hard to implement from the programmer's side. Say that language example above? The actual test would be written like: As a user, I want to vew the site in my own language Given I have come from a German network address, I should get the german language site. Those last two things are recognized via a verbatim regex (basically /^I have come from a German network address$/), and the actual implementation code does something based on that, say, setting a test IP to a known German netblock, and the last line would actually load the site and look for the right localized strings. So they're rather closely coupled together. If you change the wording of the test for whatever reason, the old code won't work anymore (actually, I think Cucumber will tell you that there was no test found), necessitating someone to actually verify that the test as written actually tests what the new requirements are.

38 points by weitingliu 11y ago

tomasien 11y ago

Secret did NOT get $35M in a Kickstarter campaign, it was venture capital. Or was that a joke I didn't get?

herbig 11y ago
Looks like it was edited on the sly: http://webcache.googleusercontent.com/search?q=cache:http://blog.testmunk.com/how-secret-used-automated-testing-for-their-android-launch/ http://webcache.googleusercontent.com/search?q=cache:http://...
- chimeracoder 11y ago
  Better link, since Google will eventually update their cache: https://archive.is/RAKfE https://archive.is/RAKfE (Yes, I get the recursive irony of archiving a Google cache result).

matthewcanty 11y ago

I find the text hard to read on my screen. Does it have to be so gray?

CodeSheikh 11y ago

It was like a mobile phones orchestra. Amazeballs.

tdumitrescu 11y ago

Given I wait for 5 seconds Then I press view with id "splash_signin_text" Then I wait for 2 seconds Then I clear input field number 1 Then I wait ... Looks like someone never bothered understanding what Given/When/Then are for.

recursive 11y ago
I think that describes most people. What's wrong with this?

swanson 11y ago

This is not terribly difficult to get running locally (at a smaller scale). Square has a tool called Spoon[0] that makes it really easy to run a test suite across a bunch of devices. On a recent project, we had 5-7 devices plugged into a spare box that would run a test suite on every push. It worked out great and the ability to capture screenshots was really helpful for the design team to help spot strange UI issues. More details on the setup here: http://www.sep.com/sep-blog/2014/01/06/android-robotium-and-spoon-a-practical-guide-for-testing-on-android-devices/ http://www.sep.com/sep-blog/2014/01/06/android-robotium-and-... But if you need larger device coverage, Testmunk looks pretty awesome :) [0]: https://github.com/square/spoon https://github.com/square/spoon

deleted 11y ago

[deleted]

saclark11 11y ago

I appreciate TestMunk's goal to make automated mobile testing easier and more accessible, and I truly hope they succeed in that, but for the love of testing, please do not view that example feature file at the bottom as an example of proper Gherkin. It is riddled with implementation details and brittle explicit waits. To be fair, they did clearly state: "we generally recommend against fixed waits" but then please do not put this forth as an example to be copied. Scenarios should describe the behavior from a user perspective and leave out implementation details. You should not have to change the your Gherkin no matter how much the underlying implementation changes so long as the expected behavior remains the same. Not to mention the misuse of the Given/When/Then keywords and the "I logout" steps -- that is what before/after hooks are for. I also recognize TestMunk points out that this feature utilizes standard steps to get your first test cases going and to get some screenshots. They also advocate for the page object pattern, which is good, but it might be helpful to be clear that this is not how you'd actually want to write your Gherkin, whose true value comes when written with/for product owners, free of implementation details.

StavrosK 11y ago
Can someone explain to me the appeal of Gherkin and other such tools? I don't see why the example on that page are anything but code that wants to "look better", and just ends up introducing another thing to maintain in your application. How is writing a step in English and writing the English to translate it to code better than just writing the exact same steps in code to begin with?
- Karunamon 11y ago
  Gherkin/Cucumber's big thing is that it lets the people who can't read code (i.e. project managers, suits, and customers) understand what's being tested. Usually along the lines of: "As a user, I want to buy things, so I should be able to put things in the cart and get an accurate total price" or "As a user, I want to view the site in my own language, given I have come from a German network address, I should get the German language site" This frees you from having to explain the finer points of cookies, or geoip, or any number of concepts to people who do not operate on a level that close to the bare metal. The translation isn't really error prone either and isn't hard to implement from the programmer's side. Say that language example above? The actual test would be written like: As a user, I want to vew the site in my own language Given I have come from a German network address, I should get the german language site. Those last two things are recognized via a verbatim regex (basically /^I have come from a German network address$/), and the actual implementation code does something based on that, say, setting a test IP to a known German netblock, and the last line would actually load the site and look for the right localized strings. So they're rather closely coupled together. If you change the wording of the test for whatever reason, the old code won't work anymore (actually, I think Cucumber will tell you that there was no test found), necessitating someone to actually verify that the test as written actually tests what the new requirements are.