20 ms·

Ask HN: Solo app devs, how do you do user testing?

262 points by rhardih 8y ago

Enlisting the help of friends and family can only get you so far, so how do you gather quality feedback pre/post-launch, without shelling out for a professional agency to do the testing?

WA 8y ago
Post-launch: What's your goal? Do you want to test whether your update works correctly on many devices? Do you want to get ideas for new features? How many users do you have? I have about 10,000 active users. I don't do much testing. I test myself on an Android device and an iPhone. Since my app is built with Ionic and based on Cordova, it works anyways and real crashes are not to expected. Feedback is easy: I just wait until people write me an email. It's funny: Sometimes, you don't hear about a feature request or some issue for months. Then 2-3 people write you within a day. Then nothing again. Back when I started, I implemented a lot of features. Now I basically reject almost every feature request, unless it comes up extremely often in emails. I sometimes take notes, but most of the time, I type a quick reply, thanking for the suggestion, archive the email and only try to get a feeling for what people want. Obviously, this has its limits, but since I know my core users and what they want, it works for me. If there's a serious bug, people will let you know soon enough.
- rhardih 8y ago
  Post-launch is mainly vetting that new features makes sense to users and catching UX faux pas, instability or even crashes, etc. before opening up to a wider audience, and possibly scores of bad reviews. I was thinking about just opening up on the app store as well, and just sit back and wait for reviews or emails, but that seems risky, since potentially bad reviews are forever. For a paid app at least, that can have a pretty serious impact on possible uptake. Having at least some level on confidence, that the app is usable and stable before general availability, is the safe play I think. To that affect, closed testing seems the only viable option.
sarabande 8y ago
I've used UserTesting.com for previous tests. If your product affects a lowest-common-denominator user or you are testing mainly UX/interaction, I'd recommend it. At that time I paid around $20 per usability test (I bought a packet of multiple sessions). Testers recorded using my product for 5 to 10 minutes and it was great watching them stumble over what I had designed. They were also trained to be over-communicative and think aloud, which really helped. In total I did four or five of them before I had my first release (since it was still somewhat expensive, I tried to ration my tests by fixing all previous UX bugs before issuing another one). It's been a long time, but if I recall correctly, a few times the testers weren't tech-competent enough to actually do the test, but when I asked UserTesting for a credit or refund to re-do the test because the tester was "bad", they provided one without hassle. You need to be make it as easy as possible to run your user test, because a tester will give up on roadblocks that take more than 60 seconds to solve. I was developing a language learning product, and so needed testers who were interested in language X. In the end, the people I found using the filter [Interested in X language, native English] weren't actually interested in language X. Rather, they came across as people who had accepted the filter to make an extra buck. However, it was good enough for a random internet trial. If I wanted better results I had to do it in person (which I also recommend).
wilperkins 8y ago
I usually hang out in the kitchen at our shared office space and ask people. If that's not enough, then I'll start visiting friends' offices for team breakfast or post-work beers and do the same.
jermaustin1 8y ago
In the past (when I was still a solo developer), after being 90% confident the code won't completely break the environment or do something screwy with user data, sticking the code in production and funneling 50% of the requests to it while watching NewRelic. There are no better user testers than your users, and the risk is only moderate. I could always pull the server down and restore the previous version while trying to fix whatever I broke in the deployment. But if you aren't doing SaaS, user testing as a solo developer is not really feasible. The best user testing I can think of that doesn't "cost" is a Beta/RC Program for your users to volunteer to get potentially broke software for a discount or just early feature releases. For this to work though, you have to have enough users.
- rhardih 8y ago
  I agree, it's easier to do for web apps since you don't have to contend with "app store reviews", that can potentially leave a lasting blemish on your product. So far I've done it as you describe as well and it's the "getting enough users" part that's a pickle. If you get X users on to sign up for a beta, only a fraction of those will actually ever install the app and give it a spin and again, only a fraction of those will give you any real feedback to work from. Finding real testers without breaking the budget, of your (maybe zero-profit) app is the challenge.
  - jermaustin1 8y ago
    I've never trusted a user to provide feedback. They will complain if something if it's terribly broken, but not if it's still usable. That's why I have always used a 3rd party system to collect actual usage analytics combined with server logs.
paul7986 8y ago
For web apps I’ve always used Mechanical Turk and paid each user $10 vs. just a dollar or less. I made each tester download a screen record plug in for chrome, hit record and then go through all the steps in the test. Once finished they emailed me the recording. The above is what you get from usertesting.com and other sites but pay up the nose($50 a test) vs. $10 or less dollars per test. If it’s a downloadable app Just spin up a web UI/UX and have the turkers follow same steps but have them change the browser to say iPhone 6/7/8 view.
- netzone 8y ago
  I don't understand this, why would you want people to test something that's already predefined? You could automate that. You want real users that will use the app for real things. Blind testing from a list seems kind of pointless.
  - paul7986 8y ago
    Umm it’s design keep it simple for all users from grandma to power users.
  - daleco 8y ago
    The goal is to detect usability issues. Your features should have user stories associated to them, come up with few scenarios around these (e.g. book a flight, cancel your flight from the landing page...). Then test your scenarios with multiple users/personas. This will allow you to detect UX/usabilities issues, and prioritize update based on how critical the issue is.
    - paul7986 8y ago
      Exactly and with multiple recordings of each user story you will be able to see if your design is idiot proof or not.
drrob 8y ago
I'd say it depends on the sort of application you're developing. In the next couple of weeks I'm due to release a boxing game (shameless plug: leatherthegame.com), and other than myself my only tester has been my cousin. Having worked on the project for over 3 years I'm aware that there are areas where I "can't see the wood for the trees", but I am confident in my own knowledge, and only wanted an external viewpoint to see how a noob would see things from afresh (both in terms of not having used the app before and also him being only a lite-casual boxing fan, so he wasn't up-to-speed with some terminology). I know full well that once its released the proper boxing fans will get in touch, and I'll change things post-release from that feedback, so my strategy is very much a post-launch end user feedback one. In an ideal world I would have had more friends who could help me test it, but there's only so many opinions I can factor in pre-launch, and managing their experiences and feedback would have been a mini-project to manage purely by itself. When already time-constrained with launch schedules, app store listings and last minute bug hunts and device optimisation I couldn't spare the time to manage a phalanx of testers. This has been a passion project/scratch-my-own-itch project, which is why I feel more secure not having many outside opinions. If you're building something more strategic, to fill a target niche in a specific market say, then my approach will DEFINITELY NOT WORK. In this case you'd be best off firing out a message on LinkedIn or something like that, asking for would-be best testers.
creo 8y ago
I give it to my friends and ask them to do specific things. Most of the time they will point me to something i didn't know.
- rhardih 8y ago
  That's currently where I'm at myself. Do you feel that, that approach scales fine and you get a broad enough idea of where problems are for the general user and where you need to improve/change things?
  - ddebernardy 8y ago
    It doesn't scale, in that you ideally want to test your app's UI/UX with first time users who haven't gotten used to your app's quirks yet. But it allows to catch a lot of stuff early on at the cost of buying them a coffee or a beer (if that).
  - creo 8y ago
    You'll just learn things, but it won't scale at all. One of many reasons is that it's very specific people (it may not even be Your target users).
segmondy 8y ago
Automated testing. I start out with manual tests. If I have a bug at any part of the code I add a unit test. When it gets too big and starts taking too long then I start writing e2e tests.
- gitgud 8y ago
  The menu system being "unclear" is not a bug you can find with automated tests. Its true that unit-tests ensure quality software. But this quality is a waste if real user's can't figure it out.
  - segmondy 8y ago
    True, I'm not talking about UI/UX test, but tests that test that your application is "correct" and does the right thing. For such tests, you need to start out a beta or just wait for user feedback or also be great in product design and thinking like the average user. Adding help popups, and good documentations.
roryisok 8y ago
I have an app in the windows store with a few thousand users. When anyone emails me with feedback or an issue, I usually ask if they want to try the beta version (after I've resolved their issue of course). I've gotten about 15-20 part time testers this way, they've been a mixed bag. Some provide pages if detailed feedback, some barely anything. When I have a decent, bug free mvp of the next version I'm going to fork out for usertesting.com
- rhardih 8y ago
  That actually sounds like a great approach! I can imagine you've triggered a sense of involvement or idea ownership in the most active ones. From my own experience, as a user, reporting a bug er suggesting a feature in a product, only to see it fixed or added, has a tremendous effect in loyalty and even promotion of said product. I think I'll definitely adopt this idea once properly released.
- paul7986 8y ago
  Forget usertesting.com use Mechanical Turk(get testers to download screen record plug in for chrome to record/send you their tests..charge $10 or less a test). Way cheaper and you get more screen recorded tests for less.
- na94 8y ago
  I do something similar. I have a Android app that I sell outside of the app store. I did build some licensing and support infrastructure to allow for automatic notification of updates. There is also a Beta release channel. Therefore, I use the support conversation in order to see if they want to be flipped to that channel. I also use ACRA and Tracepot to capture crashes and anomalous events. Users can also email me a debug log along with their current app settings. This has evolved over a 5 or 6 year period.
nickjj 8y ago
I wouldn't ever pay for an agency to do user testing, and it has nothing to do with money. Your ideal tester is someone who cares enough about your product to use it. I also wouldn't even start developing the product unless I had a decent sized list of people who really want it. In which case, testing it is easy. Just ask those people to test drive your app during development. If your app solves a real problem they will happily do it for free since it solves their problem, but you could also give them perks like giving them 2 months of your service for free when it launches. Your users will naturally give you feedback if things are broken, so that handles post-launch real world testing. Of course you'll want to supplement that with heaps of automated tests too.
- daleco 8y ago
  An agency or proper usability testing should provide you insights and ways to correct the usability issues based on the results. What you describe may work to get feedbacks and new feature requests. But you won't know why a user is not using a feature, where they are struggling, what they don't understand or see.
  - nickjj 8y ago
    > An agency or proper usability testing should provide you insights and ways to correct the usability issues based on the results. What you describe may work to get feedbacks and new feature requests. But you won't know why a user is not using a feature, where they are struggling, what they don't understand or see. If you use heatmaps and other analytical tracking tools you can see how users interact with your app. That's all you really need to see why a user isn't using a feature, struggling with the UI or happen to not see something. An agency would just have X number of paid employees systematically going through your UI and giving you their opinions on what they think is best, but that might not align with what your real users are doing.
eps 8y ago
1. Stick the mailing list sign-up form on a landing page and do a bit of promoting in relevant places. Specifically say it's for beta announcements, preview release testing and picking everyone's brain. This should net you some users to bootstrap the process with, and they will be responsive. 2. Release the beta, mark it as such and ask people for feedback after they used the product for a bit. Listen to the feedback, act on what's relevant, keep improving the beta. Give kudos where they are due. Have public forum, actively participate in all threads. Many people look at how active the support forum is before trying the goods or leaving any feedback. So the more active it is, the better. Also, moderate it aggressively to trim junk and trivialities. 3. Once out of the beta, don't forget to genereously reward everyone who chipped in. Ideally with a special gift, like a specially tagged lifetime premium account, a handful of most expensive licenses, etc. The worst thing to do is to offer a discount on a production version - this comes across as a cheap and greedy move.
superasn 8y ago
I generally find sellers on fiverr. They are willing to record screen and comment for about 30 mins for $5. Not a bad deal and mostly good insightful feedback.
lettergram 8y ago
I do a free beta, then various paid betas. The goal is also to find a price point. A user can either leave feedback in my betas and get that month free, or they can pay. The idea that all they have to do is give feedback for a free service usually works well... Eventually, when it's easier to pay, some users will start leaving money instead if feedback. At some point I'll raise prices a bit and stop the feedback for service system. I've done this for: https://projectpiglet.com/ https://projectpiglet.com/ and https://easy-a.net/ https://easy-a.net/ And a few others, it's always fairly successful - assuming you have a decent landing page and you can advertise the site in your target market for cheap / free.
jordansmithnz 8y ago
Solo dev for an app with hundreds of thousands of users. These are the things I do that ensure smooth releases: • run a beta test group. These users consist mostly of users that have emailed in with feature suggestions or bugs. I ask them if they’d like to join the beta (e.g. they can check their bug is fixed). I frequently get great feedback from a few of these users. • fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100. • learn how to test your own product. It’s hard to do well, but you can learn. It’s a valuable skill that can really set you apart in a regular engineering job too. • used phased releases. Pause the release at 1%, give it a few days for any minor issues to roll in and fix these before releasing to 100%.
- WaltPurvis 8y ago
  > used phased releases. Pause the release at 1% Could you expound on this? How do you do that? (Related: How are you distributing? Through an app store?)
  - faizmokhtar 8y ago
    There's an option to do this on both Google Play and App Store. It's called staged rollouts[1] in Google Play and phased release[2] for App Store. [1]: https://support.google.com/googleplay/android-developer/answer/6346149?hl=en https://support.google.com/googleplay/android-developer/answ... [2]: https://stackoverflow.com/questions/45073264/what-happens-when-i-do-an-itunes-connect-phased-release https://stackoverflow.com/questions/45073264/what-happens-wh...
- hnrodey 8y ago
  This sounds like very actionable advice. Thanks for sharing the info, it's very helpful.
- startupdiscuss 8y ago
  Solo dev and 100ks of users? Wow. What is your tech stack? What is your code-deploy process?
deleted 8y ago
[deleted]
acutesoftware 8y ago
The most important way to do this is to actually be a user of your product, e.g. "Eating your own dog food". This doesn't mean logging in, trying all the features and playing with it for an hour - you have to really use it with your data on a daily basis over a long period of time. You'll come across a lot of bugs and start logging feature requests yourself. Once you can use it without issues [and you'll need to be careful to split the wishlist features you come up with, against the actual bugs which need to be fixed], then you can get others to try it. Best way to get user testing from others? - email your users and ask them if they have had any issues, or want any new features. - ask people to give your product a quick test - for example, https://www.lifepim.com https://www.lifepim.com is my new webapp to easily manage your personal information [shamless plug/warning - notes and tasks work well, contacts needs work, and calendar is not pretty] - pay particular attention to requests from users who pay for your product. Everyone wants software to work in a certain way, but when someone pays, it can be a good indicator to listen to what they want. If they say something isn't working, make it a priority to investigate and address this.
- aggie 8y ago
  Just keep in mind with dogfooding that there is a difference between figuring out how things work and working efficiently/effectively once you have figured out how things work. Dogfooding is really only good for 'testing' the latter, because you already know how everything works better than any normal user would.
  - acutesoftware 8y ago
    The best part I've found with dogfooding is that even though you know exactly how it all works (and yes, you can miss some issues because of this) , actually using the product daily can make you realise - this isn't working well or that feature is kind of a pain in the neck.
fundamental 8y ago
Try to make a simple tutorial video of some sort. Even if it isn't useful in a practical sense it forces yourself (or someone else) to walk through the UI stating their intentions and trying to execute them. Some of the best feedback I've gotten was through watching videos like this for an application that I work on as you can see where the stumbling points are throughout the process.
otterpro 8y ago
There are user-testing companies that does this for small fee: * http://www.trymyui.com/ * https://www.usertesting.com/ * https://Whatusersdo.com/ * https://Userlytics.com/ Disclaimer: I've also worked on couple of those companies as a tester when I needed money and I was out of job, but thankfully I'm currently back working as developer again. On some of them, the tester also needed to record themselves using webcam.
jqbx_jason 8y ago
Using your product daily is essential not only to help steer the ship on what to build but also to cue you in on what's buggy / inefficient. I have the good fortune of having a live chat interface (product is: https://www.jqbx.fm https://www.jqbx.fm) so I get pretty quick feedback if something that's been pushed is causing anyone any issues. However to keep bugs to a minimum I keep most of the logic client side and have a dev environment that I can run on production- I usually test a version for a couple days (see: eat your own dogfood) before releasing anything. Also forcing yourself to review your own code is a good skill and has helped catch some issues right before pulling the trigger.
no_gravity 8y ago
One thing I have found to be very useful is to ask my existing userbase of other projects if they want to test something new and give me feedback. In fact this worked so well that I am currently building a new project around this. A site that lets startups gather feedback from people who are interested in trying out new sites and giving feedback. It is currently in private beta with about 20 startups. If you want an invite, feel free to send me an email.
buf 8y ago
I push the code live and see what the numbers say. I know this isn't the answer most people want to hear, but no one tells the truth like live users.
zer00eyz 8y ago
It doesn't matter how big you are, the same cheep method works for informal but fairly accurate quick and dirty testing. Beer. You think I'm kidding but if you go to a bar, there are lots of people who will test whatever you put in front of them for a free drink. If you want to keep your costs down even further then go during happy hour. Is that "user testing" no... not in the sense we think of it today. Is it testing of the usability of your product: yes it is. Im guessing that most apps issues that are more likely the latter and not the former. Basic things that you think are clear that really aren't (your too close to the problem and the solution). If you explain, point and guide through the same aspect of your feature 10 times in 10 tests people might not get it! These are the problems you might want to solve, dumb it down, change it up, make it better. Does this work? Yes, as long as you can moderate. If you can you be neutral and friendly and avoid "traps" that will alter your results. Saying things like "MY app" or reacting personally will alter your results, and learning when to "guide them" through a frustrating point and when to let them puzzle through it is a bit of an art (this too offers clues if your guiding people at the same place). For me personally a drink or two helps ME in the process... I have also discovered that this process works very well with pairs of people. Two friends already have a rapport and will happily talk to each other to try to get through something problematic. One can encourage or discourage the other and their dialog can be enlightening. Letting them play together with your app and drink free drinks can give you a LOT more insight than the next three individuals testing. Anything more than two people becomes problematic so look to avoid large groups.
- lucasverra 8y ago
  great advice, will be implemented :)
- daleco 8y ago
  Guerilla testing is a great starting point. Do you usualy give one or multiple scenarios to complete? Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...)
  - 8y ago
bitwize 8y ago
Test it myself, and occasionally get a friend to bang on it.
deleted 8y ago
[deleted]
sudouser 8y ago
external agency, 1-2 weekly tests, specially critical in the first stages of the product. good price, cheap compared to a dev’s salary, or compared to my time, paying less than $500 per month. do 5-second tests, usability, card sorting, path testing, all ux and ui testing for greater chance of success
coreyjwaldin 8y ago
Checkout helio.app
deleted 8y ago
[deleted]
upbeatlinux 8y ago
In production (sarcasm).
Radeo 8y ago
Probably it was already said, but quite crucial in the process would be proper analytics. To see where the user's flow is concentrated or stops at all.
mesozoic 8y ago
Integrated tests are key. Not a solo dev but my company won't give me any QA resources so the only way forward is integration tests.
- segmondy 8y ago
  On projects I do solo dev, I do integrated tests because after a while the project becomes big enough that trying to cover all paths takes too damn long. My current project got to be about 70% testing/30% code and I was miserable. I stopped all coding and moved to e2e tests and now I'm back to 95% coding / 5% coding and quite happy. Most companies surprisingly don't care about full integration tests, they talk about it but are quite happy to have fingers on the keyboard and mouse pecking and clicking away.
  - mesozoic 8y ago
    Yeah pretty much the same here. It's funny when you start listening to actions and incentives instead of words. You realize its all doublespeak.
euonotn34ntuent 8y ago
I used to be a solo dev working in a niche field where I made plug-ins for some creative apps. I found that one thing that really helped was being active in the forums where users were, and (if allowed by the forum) posting that you have a product and you'd like to get some people to test it. I'd then require them to fill out a simple form that gathered some information about how they work. The form served 2 purposes: 1) It gave me the ability to balance different types of users. Hobbyists vs. professionals. People in this specialty vs. that specialty. etc. 2) It weeded out people who just wanted something free and weren't interested in giving feedback. Seriously, a simple form requiring a name, an email, and the answer to 3-5 questions helped get rid of the freeloaders. But don't make it too onerous or you won't end up with any testers. I'd then usually give a free license for the released product to anyone who provided useful feedback.
jakobegger 8y ago
If I need to get feedback on a feature, I build a prototype, and then show it to one or two people (friends, family, customers that sent feature requests). I tell them roughly what they should do, and watch if they can figure it out. Then I change things as necessary, and show it to the next person. Usually a handful of people are enough to find the the major issues. - don't "burn" beta testers. show a prototype to one or two people, and don't show it to more people before you fixed the issues that have come up. otherwise you just get duplicate feedback - ask people personally. Emails to a group of people are ignored by 99% of recipients - feel free to ignore advise from testers. everyone always wants to give advise, but that is irrelevant. the important part is: did the tester manage to complete the task? - do not ignore people struggling with your app. the natural reaction to people failing to use your app is to think they are stupid. they aren't. most of the time your ui is stupid and you need to fix it. it doesn't matter how brilliant you think the ui is, if testers can't use it, change it. - sometimes the answer is that what you built doesn't work and you just wasted a lot of time and you need to throw it away and start over. do not ignore this possibility
deleted 8y ago
[deleted]
gotrythis 8y ago
I didn't read all the comments, so don't know if anyone else suggested this. A few times now, I have successfully got my first few hundred users by offering a lifetime discount and positioning it as a limited access beta testing program. Got lots of feedback, and most users kept their accounts for a very long time, even if they weren't using the software, just do they wouldn't lose their discount.
troycarlson 8y ago
One tool I use is Mouseflow, primarily for session playback. It's super helpful to see how users are interacting with the app, qualitatively, and has helped me find bugs and UX issues that never made it into the error logs. https://mouseflow.com https://mouseflow.com

262 points by rhardih 8y ago

Enlisting the help of friends and family can only get you so far, so how do you gather quality feedback pre/post-launch, without shelling out for a professional agency to do the testing?

WA 8y ago

Post-launch: What's your goal? Do you want to test whether your update works correctly on many devices? Do you want to get ideas for new features? How many users do you have? I have about 10,000 active users. I don't do much testing. I test myself on an Android device and an iPhone. Since my app is built with Ionic and based on Cordova, it works anyways and real crashes are not to expected. Feedback is easy: I just wait until people write me an email. It's funny: Sometimes, you don't hear about a feature request or some issue for months. Then 2-3 people write you within a day. Then nothing again. Back when I started, I implemented a lot of features. Now I basically reject almost every feature request, unless it comes up extremely often in emails. I sometimes take notes, but most of the time, I type a quick reply, thanking for the suggestion, archive the email and only try to get a feeling for what people want. Obviously, this has its limits, but since I know my core users and what they want, it works for me. If there's a serious bug, people will let you know soon enough.

rhardih 8y ago
Post-launch is mainly vetting that new features makes sense to users and catching UX faux pas, instability or even crashes, etc. before opening up to a wider audience, and possibly scores of bad reviews. I was thinking about just opening up on the app store as well, and just sit back and wait for reviews or emails, but that seems risky, since potentially bad reviews are forever. For a paid app at least, that can have a pretty serious impact on possible uptake. Having at least some level on confidence, that the app is usable and stable before general availability, is the safe play I think. To that affect, closed testing seems the only viable option.

sarabande 8y ago

I've used UserTesting.com for previous tests. If your product affects a lowest-common-denominator user or you are testing mainly UX/interaction, I'd recommend it. At that time I paid around $20 per usability test (I bought a packet of multiple sessions). Testers recorded using my product for 5 to 10 minutes and it was great watching them stumble over what I had designed. They were also trained to be over-communicative and think aloud, which really helped. In total I did four or five of them before I had my first release (since it was still somewhat expensive, I tried to ration my tests by fixing all previous UX bugs before issuing another one). It's been a long time, but if I recall correctly, a few times the testers weren't tech-competent enough to actually do the test, but when I asked UserTesting for a credit or refund to re-do the test because the tester was "bad", they provided one without hassle. You need to be make it as easy as possible to run your user test, because a tester will give up on roadblocks that take more than 60 seconds to solve. I was developing a language learning product, and so needed testers who were interested in language X. In the end, the people I found using the filter [Interested in X language, native English] weren't actually interested in language X. Rather, they came across as people who had accepted the filter to make an extra buck. However, it was good enough for a random internet trial. If I wanted better results I had to do it in person (which I also recommend).

wilperkins 8y ago

I usually hang out in the kitchen at our shared office space and ask people. If that's not enough, then I'll start visiting friends' offices for team breakfast or post-work beers and do the same.

jermaustin1 8y ago

In the past (when I was still a solo developer), after being 90% confident the code won't completely break the environment or do something screwy with user data, sticking the code in production and funneling 50% of the requests to it while watching NewRelic. There are no better user testers than your users, and the risk is only moderate. I could always pull the server down and restore the previous version while trying to fix whatever I broke in the deployment. But if you aren't doing SaaS, user testing as a solo developer is not really feasible. The best user testing I can think of that doesn't "cost" is a Beta/RC Program for your users to volunteer to get potentially broke software for a discount or just early feature releases. For this to work though, you have to have enough users.

rhardih 8y ago
I agree, it's easier to do for web apps since you don't have to contend with "app store reviews", that can potentially leave a lasting blemish on your product. So far I've done it as you describe as well and it's the "getting enough users" part that's a pickle. If you get X users on to sign up for a beta, only a fraction of those will actually ever install the app and give it a spin and again, only a fraction of those will give you any real feedback to work from. Finding real testers without breaking the budget, of your (maybe zero-profit) app is the challenge.
- jermaustin1 8y ago
  I've never trusted a user to provide feedback. They will complain if something if it's terribly broken, but not if it's still usable. That's why I have always used a 3rd party system to collect actual usage analytics combined with server logs.

paul7986 8y ago

For web apps I’ve always used Mechanical Turk and paid each user $10 vs. just a dollar or less. I made each tester download a screen record plug in for chrome, hit record and then go through all the steps in the test. Once finished they emailed me the recording. The above is what you get from usertesting.com and other sites but pay up the nose($50 a test) vs. $10 or less dollars per test. If it’s a downloadable app Just spin up a web UI/UX and have the turkers follow same steps but have them change the browser to say iPhone 6/7/8 view.

netzone 8y ago
I don't understand this, why would you want people to test something that's already predefined? You could automate that. You want real users that will use the app for real things. Blind testing from a list seems kind of pointless.
- paul7986 8y ago
  Umm it’s design keep it simple for all users from grandma to power users.
- daleco 8y ago
  The goal is to detect usability issues. Your features should have user stories associated to them, come up with few scenarios around these (e.g. book a flight, cancel your flight from the landing page...). Then test your scenarios with multiple users/personas. This will allow you to detect UX/usabilities issues, and prioritize update based on how critical the issue is.
  - paul7986 8y ago
    Exactly and with multiple recordings of each user story you will be able to see if your design is idiot proof or not.

drrob 8y ago

I'd say it depends on the sort of application you're developing. In the next couple of weeks I'm due to release a boxing game (shameless plug: leatherthegame.com), and other than myself my only tester has been my cousin. Having worked on the project for over 3 years I'm aware that there are areas where I "can't see the wood for the trees", but I am confident in my own knowledge, and only wanted an external viewpoint to see how a noob would see things from afresh (both in terms of not having used the app before and also him being only a lite-casual boxing fan, so he wasn't up-to-speed with some terminology). I know full well that once its released the proper boxing fans will get in touch, and I'll change things post-release from that feedback, so my strategy is very much a post-launch end user feedback one. In an ideal world I would have had more friends who could help me test it, but there's only so many opinions I can factor in pre-launch, and managing their experiences and feedback would have been a mini-project to manage purely by itself. When already time-constrained with launch schedules, app store listings and last minute bug hunts and device optimisation I couldn't spare the time to manage a phalanx of testers. This has been a passion project/scratch-my-own-itch project, which is why I feel more secure not having many outside opinions. If you're building something more strategic, to fill a target niche in a specific market say, then my approach will DEFINITELY NOT WORK. In this case you'd be best off firing out a message on LinkedIn or something like that, asking for would-be best testers.

creo 8y ago

I give it to my friends and ask them to do specific things. Most of the time they will point me to something i didn't know.

rhardih 8y ago
That's currently where I'm at myself. Do you feel that, that approach scales fine and you get a broad enough idea of where problems are for the general user and where you need to improve/change things?
- ddebernardy 8y ago
  It doesn't scale, in that you ideally want to test your app's UI/UX with first time users who haven't gotten used to your app's quirks yet. But it allows to catch a lot of stuff early on at the cost of buying them a coffee or a beer (if that).
- creo 8y ago
  You'll just learn things, but it won't scale at all. One of many reasons is that it's very specific people (it may not even be Your target users).

segmondy 8y ago

Automated testing. I start out with manual tests. If I have a bug at any part of the code I add a unit test. When it gets too big and starts taking too long then I start writing e2e tests.

gitgud 8y ago
The menu system being "unclear" is not a bug you can find with automated tests. Its true that unit-tests ensure quality software. But this quality is a waste if real user's can't figure it out.
- segmondy 8y ago
  True, I'm not talking about UI/UX test, but tests that test that your application is "correct" and does the right thing. For such tests, you need to start out a beta or just wait for user feedback or also be great in product design and thinking like the average user. Adding help popups, and good documentations.

roryisok 8y ago

I have an app in the windows store with a few thousand users. When anyone emails me with feedback or an issue, I usually ask if they want to try the beta version (after I've resolved their issue of course). I've gotten about 15-20 part time testers this way, they've been a mixed bag. Some provide pages if detailed feedback, some barely anything. When I have a decent, bug free mvp of the next version I'm going to fork out for usertesting.com

rhardih 8y ago
That actually sounds like a great approach! I can imagine you've triggered a sense of involvement or idea ownership in the most active ones. From my own experience, as a user, reporting a bug er suggesting a feature in a product, only to see it fixed or added, has a tremendous effect in loyalty and even promotion of said product. I think I'll definitely adopt this idea once properly released.
paul7986 8y ago
Forget usertesting.com use Mechanical Turk(get testers to download screen record plug in for chrome to record/send you their tests..charge $10 or less a test). Way cheaper and you get more screen recorded tests for less.
na94 8y ago
I do something similar. I have a Android app that I sell outside of the app store. I did build some licensing and support infrastructure to allow for automatic notification of updates. There is also a Beta release channel. Therefore, I use the support conversation in order to see if they want to be flipped to that channel. I also use ACRA and Tracepot to capture crashes and anomalous events. Users can also email me a debug log along with their current app settings. This has evolved over a 5 or 6 year period.

nickjj 8y ago

I wouldn't ever pay for an agency to do user testing, and it has nothing to do with money. Your ideal tester is someone who cares enough about your product to use it. I also wouldn't even start developing the product unless I had a decent sized list of people who really want it. In which case, testing it is easy. Just ask those people to test drive your app during development. If your app solves a real problem they will happily do it for free since it solves their problem, but you could also give them perks like giving them 2 months of your service for free when it launches. Your users will naturally give you feedback if things are broken, so that handles post-launch real world testing. Of course you'll want to supplement that with heaps of automated tests too.

daleco 8y ago
An agency or proper usability testing should provide you insights and ways to correct the usability issues based on the results. What you describe may work to get feedbacks and new feature requests. But you won't know why a user is not using a feature, where they are struggling, what they don't understand or see.
- nickjj 8y ago
  > An agency or proper usability testing should provide you insights and ways to correct the usability issues based on the results. What you describe may work to get feedbacks and new feature requests. But you won't know why a user is not using a feature, where they are struggling, what they don't understand or see. If you use heatmaps and other analytical tracking tools you can see how users interact with your app. That's all you really need to see why a user isn't using a feature, struggling with the UI or happen to not see something. An agency would just have X number of paid employees systematically going through your UI and giving you their opinions on what they think is best, but that might not align with what your real users are doing.

eps 8y ago

1. Stick the mailing list sign-up form on a landing page and do a bit of promoting in relevant places. Specifically say it's for beta announcements, preview release testing and picking everyone's brain. This should net you some users to bootstrap the process with, and they will be responsive. 2. Release the beta, mark it as such and ask people for feedback after they used the product for a bit. Listen to the feedback, act on what's relevant, keep improving the beta. Give kudos where they are due. Have public forum, actively participate in all threads. Many people look at how active the support forum is before trying the goods or leaving any feedback. So the more active it is, the better. Also, moderate it aggressively to trim junk and trivialities. 3. Once out of the beta, don't forget to genereously reward everyone who chipped in. Ideally with a special gift, like a specially tagged lifetime premium account, a handful of most expensive licenses, etc. The worst thing to do is to offer a discount on a production version - this comes across as a cheap and greedy move.

superasn 8y ago

I generally find sellers on fiverr. They are willing to record screen and comment for about 30 mins for $5. Not a bad deal and mostly good insightful feedback.

lettergram 8y ago

I do a free beta, then various paid betas. The goal is also to find a price point. A user can either leave feedback in my betas and get that month free, or they can pay. The idea that all they have to do is give feedback for a free service usually works well... Eventually, when it's easier to pay, some users will start leaving money instead if feedback. At some point I'll raise prices a bit and stop the feedback for service system. I've done this for: https://projectpiglet.com/ https://projectpiglet.com/ and https://easy-a.net/ https://easy-a.net/ And a few others, it's always fairly successful - assuming you have a decent landing page and you can advertise the site in your target market for cheap / free.

jordansmithnz 8y ago

Solo dev for an app with hundreds of thousands of users. These are the things I do that ensure smooth releases: • run a beta test group. These users consist mostly of users that have emailed in with feature suggestions or bugs. I ask them if they’d like to join the beta (e.g. they can check their bug is fixed). I frequently get great feedback from a few of these users. • fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100. • learn how to test your own product. It’s hard to do well, but you can learn. It’s a valuable skill that can really set you apart in a regular engineering job too. • used phased releases. Pause the release at 1%, give it a few days for any minor issues to roll in and fix these before releasing to 100%.

WaltPurvis 8y ago
> used phased releases. Pause the release at 1% Could you expound on this? How do you do that? (Related: How are you distributing? Through an app store?)
- faizmokhtar 8y ago
  There's an option to do this on both Google Play and App Store. It's called staged rollouts[1] in Google Play and phased release[2] for App Store. [1]: https://support.google.com/googleplay/android-developer/answer/6346149?hl=en https://support.google.com/googleplay/android-developer/answ... [2]: https://stackoverflow.com/questions/45073264/what-happens-when-i-do-an-itunes-connect-phased-release https://stackoverflow.com/questions/45073264/what-happens-wh...
hnrodey 8y ago
This sounds like very actionable advice. Thanks for sharing the info, it's very helpful.
startupdiscuss 8y ago
Solo dev and 100ks of users? Wow. What is your tech stack? What is your code-deploy process?

deleted 8y ago

[deleted]

acutesoftware 8y ago

The most important way to do this is to actually be a user of your product, e.g. "Eating your own dog food". This doesn't mean logging in, trying all the features and playing with it for an hour - you have to really use it with your data on a daily basis over a long period of time. You'll come across a lot of bugs and start logging feature requests yourself. Once you can use it without issues [and you'll need to be careful to split the wishlist features you come up with, against the actual bugs which need to be fixed], then you can get others to try it. Best way to get user testing from others? - email your users and ask them if they have had any issues, or want any new features. - ask people to give your product a quick test - for example, https://www.lifepim.com https://www.lifepim.com is my new webapp to easily manage your personal information [shamless plug/warning - notes and tasks work well, contacts needs work, and calendar is not pretty] - pay particular attention to requests from users who pay for your product. Everyone wants software to work in a certain way, but when someone pays, it can be a good indicator to listen to what they want. If they say something isn't working, make it a priority to investigate and address this.

aggie 8y ago
Just keep in mind with dogfooding that there is a difference between figuring out how things work and working efficiently/effectively once you have figured out how things work. Dogfooding is really only good for 'testing' the latter, because you already know how everything works better than any normal user would.
- acutesoftware 8y ago
  The best part I've found with dogfooding is that even though you know exactly how it all works (and yes, you can miss some issues because of this) , actually using the product daily can make you realise - this isn't working well or that feature is kind of a pain in the neck.

fundamental 8y ago

Try to make a simple tutorial video of some sort. Even if it isn't useful in a practical sense it forces yourself (or someone else) to walk through the UI stating their intentions and trying to execute them. Some of the best feedback I've gotten was through watching videos like this for an application that I work on as you can see where the stumbling points are throughout the process.

otterpro 8y ago

There are user-testing companies that does this for small fee: * http://www.trymyui.com/ * https://www.usertesting.com/ * https://Whatusersdo.com/ * https://Userlytics.com/ Disclaimer: I've also worked on couple of those companies as a tester when I needed money and I was out of job, but thankfully I'm currently back working as developer again. On some of them, the tester also needed to record themselves using webcam.

jqbx_jason 8y ago

Using your product daily is essential not only to help steer the ship on what to build but also to cue you in on what's buggy / inefficient. I have the good fortune of having a live chat interface (product is: https://www.jqbx.fm https://www.jqbx.fm) so I get pretty quick feedback if something that's been pushed is causing anyone any issues. However to keep bugs to a minimum I keep most of the logic client side and have a dev environment that I can run on production- I usually test a version for a couple days (see: eat your own dogfood) before releasing anything. Also forcing yourself to review your own code is a good skill and has helped catch some issues right before pulling the trigger.

no_gravity 8y ago

One thing I have found to be very useful is to ask my existing userbase of other projects if they want to test something new and give me feedback. In fact this worked so well that I am currently building a new project around this. A site that lets startups gather feedback from people who are interested in trying out new sites and giving feedback. It is currently in private beta with about 20 startups. If you want an invite, feel free to send me an email.

buf 8y ago

I push the code live and see what the numbers say. I know this isn't the answer most people want to hear, but no one tells the truth like live users.

zer00eyz 8y ago

It doesn't matter how big you are, the same cheep method works for informal but fairly accurate quick and dirty testing. Beer. You think I'm kidding but if you go to a bar, there are lots of people who will test whatever you put in front of them for a free drink. If you want to keep your costs down even further then go during happy hour. Is that "user testing" no... not in the sense we think of it today. Is it testing of the usability of your product: yes it is. Im guessing that most apps issues that are more likely the latter and not the former. Basic things that you think are clear that really aren't (your too close to the problem and the solution). If you explain, point and guide through the same aspect of your feature 10 times in 10 tests people might not get it! These are the problems you might want to solve, dumb it down, change it up, make it better. Does this work? Yes, as long as you can moderate. If you can you be neutral and friendly and avoid "traps" that will alter your results. Saying things like "MY app" or reacting personally will alter your results, and learning when to "guide them" through a frustrating point and when to let them puzzle through it is a bit of an art (this too offers clues if your guiding people at the same place). For me personally a drink or two helps ME in the process... I have also discovered that this process works very well with pairs of people. Two friends already have a rapport and will happily talk to each other to try to get through something problematic. One can encourage or discourage the other and their dialog can be enlightening. Letting them play together with your app and drink free drinks can give you a LOT more insight than the next three individuals testing. Anything more than two people becomes problematic so look to avoid large groups.

lucasverra 8y ago
great advice, will be implemented :)
daleco 8y ago
Guerilla testing is a great starting point. Do you usualy give one or multiple scenarios to complete? Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...)
- 8y ago

bitwize 8y ago

Test it myself, and occasionally get a friend to bang on it.

deleted 8y ago

[deleted]

sudouser 8y ago

external agency, 1-2 weekly tests, specially critical in the first stages of the product. good price, cheap compared to a dev’s salary, or compared to my time, paying less than $500 per month. do 5-second tests, usability, card sorting, path testing, all ux and ui testing for greater chance of success

coreyjwaldin 8y ago

Checkout helio.app

deleted 8y ago

[deleted]

upbeatlinux 8y ago

In production (sarcasm).

Radeo 8y ago

Probably it was already said, but quite crucial in the process would be proper analytics. To see where the user's flow is concentrated or stops at all.

mesozoic 8y ago

Integrated tests are key. Not a solo dev but my company won't give me any QA resources so the only way forward is integration tests.

segmondy 8y ago
On projects I do solo dev, I do integrated tests because after a while the project becomes big enough that trying to cover all paths takes too damn long. My current project got to be about 70% testing/30% code and I was miserable. I stopped all coding and moved to e2e tests and now I'm back to 95% coding / 5% coding and quite happy. Most companies surprisingly don't care about full integration tests, they talk about it but are quite happy to have fingers on the keyboard and mouse pecking and clicking away.
- mesozoic 8y ago
  Yeah pretty much the same here. It's funny when you start listening to actions and incentives instead of words. You realize its all doublespeak.

euonotn34ntuent 8y ago

I used to be a solo dev working in a niche field where I made plug-ins for some creative apps. I found that one thing that really helped was being active in the forums where users were, and (if allowed by the forum) posting that you have a product and you'd like to get some people to test it. I'd then require them to fill out a simple form that gathered some information about how they work. The form served 2 purposes: 1) It gave me the ability to balance different types of users. Hobbyists vs. professionals. People in this specialty vs. that specialty. etc. 2) It weeded out people who just wanted something free and weren't interested in giving feedback. Seriously, a simple form requiring a name, an email, and the answer to 3-5 questions helped get rid of the freeloaders. But don't make it too onerous or you won't end up with any testers. I'd then usually give a free license for the released product to anyone who provided useful feedback.

jakobegger 8y ago

If I need to get feedback on a feature, I build a prototype, and then show it to one or two people (friends, family, customers that sent feature requests). I tell them roughly what they should do, and watch if they can figure it out. Then I change things as necessary, and show it to the next person. Usually a handful of people are enough to find the the major issues. - don't "burn" beta testers. show a prototype to one or two people, and don't show it to more people before you fixed the issues that have come up. otherwise you just get duplicate feedback - ask people personally. Emails to a group of people are ignored by 99% of recipients - feel free to ignore advise from testers. everyone always wants to give advise, but that is irrelevant. the important part is: did the tester manage to complete the task? - do not ignore people struggling with your app. the natural reaction to people failing to use your app is to think they are stupid. they aren't. most of the time your ui is stupid and you need to fix it. it doesn't matter how brilliant you think the ui is, if testers can't use it, change it. - sometimes the answer is that what you built doesn't work and you just wasted a lot of time and you need to throw it away and start over. do not ignore this possibility

deleted 8y ago

[deleted]

gotrythis 8y ago

I didn't read all the comments, so don't know if anyone else suggested this. A few times now, I have successfully got my first few hundred users by offering a lifetime discount and positioning it as a limited access beta testing program. Got lots of feedback, and most users kept their accounts for a very long time, even if they weren't using the software, just do they wouldn't lose their discount.

troycarlson 8y ago

One tool I use is Mouseflow, primarily for session playback. It's super helpful to see how users are interacting with the app, qualitatively, and has helped me find bugs and UX issues that never made it into the error logs. https://mouseflow.com https://mouseflow.com