4 ms·

This kind of process is extremely valuable and should be done by devs more often. Start from the start and follow whatever your application tells you to do. Not

by IanCal 3y ago

This kind of process is extremely valuable and should be done by devs more often. Start from the start and follow whatever your application tells you to do. Note down when it doesn't tell you where to go or what to do. You'd be surprised by just how many things you do automatically while working because you know the little tricks and things to get by, and that wording doesn't necessarily match what the app requires now. Side note - this kind of this is why good QA people are awesome. They'll show you what users will actually do. I'll add in something here. Element the app said they were logging into matrix.org. matrix.org has a "try matrix". The first thing is it tells me to choose a client (this feels like a loop), then says to choose a server but also maybe I don't need to, then has a create account button. The create account button takes me to a docs page. Which tells me to go to the element site, and then create an account with matrix. So that's matrix -> use element -> element says to use matrix -> matrix says to use someone else, ok you can use us -> to use us go to element -> element says you're making an account with matrix. edit - oh you can and should also do this with your dev process. Create an empty folder, check out the repo and follow the readme. Do you actually get a running system for local dev? Can you successfully run the tests? If you are able to, do this on a clean machine (maybe load up a docker image and see if you can follow it in a truly clear system). Does it turn out it assumes you already have tool X installed because your developers already have it from another project? Do you actually need postgres running with a specific user with specific login details? If you're like me you don't like writing docs, so this may actually just push you to add scripts that do the setup required. To not sound sanctimonious about this every time I've done this with my own code I've found issues with the documentation.

INTPenis 3y ago
>This kind of process is extremely valuable and should be done by devs more often. The fact that it's not being done doesn't bode well for their perceived engagement to this project. I remember when it launched and how much they hyped it up to be the future of secure messaging. That was how many years ago now? It was pre-pandemic. I'm a lover of all selfhostable federated solutions so I actually hosted a Matrix server for a couple of years. My conclusion is that it's just not ready for production scalability. And you can't migrate easily between implementations because of their unique database design.
vallode 3y ago
QA is a massively underappreciated position. A QA person that knows when to automate, when to manually test, and how to report and file issues relevant to the project can save a significant proportion of hours on a project overall. I wish many more companies included budget for QA, it saves developers a lot of time. A bit of a side-note: this sort of analysis is a great answer to "I want to contribute to open source, how?". Some fairly simple wins for significantly better user experience, and no coding required!
- ChrisMarshallNY 3y ago
  The old wisdom in the US, is that if you have "Quality" in your job title, your career is over. At the Japanese company that I used to work for, it meant that you were one of the most powerful people in the corporation, and was a sought-after adornment. Different strokes, and all that...
  - ivan_gammel 3y ago
    I always give the power to my QA team to block any release no matter what and to give higher priority to tickets than product manager. If CEO wants to override, I cover them and take the blame. This is not a guarantee that there will be no bugs in production, but it saved us a few times.
    - ChrisMarshallNY 3y ago
      The Japanese testers were the best I'd ever seen. They never reported a "NotABug." They could back up every report, and give exact reproduction steps. They found weird, obscure corner cases, and that was by hand (they hated automation tools). They had 3,000-line Excel spreadsheets. If even one of those rows failed, the whole shooting match (like an entire product line) could come to a halt (so that meant they had to cross their t's, and dot their i's). They seldom had "opinion-based" reports, and, when they did, the report was presented by the manager, after long discussions. The company I worked for, was renowned as one of the highest-Quality optical corporations in the world.
thibaultamartin 3y ago
Hi, I'm the Thib person mention in this article, and I agree that QA is super important. I can mostly talk about matrix.org, since I have little power over the Element clients. Disclaimer though: I'm technically employed by Element (to make paperwork simpler since I'm France-based, Element has an entity in France, and the Foundation is UK-based), but I'm working for the Foundation full time. This kind of article is super valuable since it gives us the perspective of a new user. I opened https://github.com/matrix-org/matrix.org/issues/2178 https://github.com/matrix-org/matrix.org/issues/2178 to translate the gripes mentioned in the issue into actionable items for us. I took action on the most urgent one (updating the Try Matrix page), but want to take the time to go beyond the surface symptoms and address the root cause of the other gripes. On the Foundation side, we're a small but mighty team of four. The website is currently maintained part time by me and a volunteer who is doing an excellent job at it. As I wrote recently in a blog post "Tracking what works, not people" (https://ergaster.org/posts/2024/01/24-tracking-what-works/ https://ergaster.org/posts/2024/01/24-tracking-what-works/), I would love to have the resources to conduct user research and user testing on the website but I unfortunately don't. We deployed privacy-preserving analytics to see where people drop and what confuses them. It's not nearly as good as proper QA and user testing, but that's what we can afford for now. Overall I'm grateful to the author for documenting their frustration, and even more grateful for reacting constructively to our responses and integrating them in the blog post! One of the strengths of open source is to find and address issues collectively. I consider this blog post to be a good open source contribution. If people around believe in our mission and want to help us with their brainpower, I invite them to join our "Office of the Matrix.org Foundation" room: https://matrix.to/#/%23foundation-office:matrix.org https://matrix.to/#/%23foundation-office:matrix.org For those aligned with our mission and who want to support us financially, the https://matrix.org/support/ https://matrix.org/support/ page should give you all the information you need to help us out.
- gowings97 3y ago
  Do you have any thoughts on how you might improve this workflow?
  - thibaultamartin 3y ago
justin_oaks 3y ago
If you're trying to make a good onboarding user experience then you should do your onboarding testing with people who've never seen the product before, not devs or QA. Once people are familiar with the product (devs, QA, and anyone who has used it before) then they're "tainted". They'll remember the weird way that they had to work around an issue, and that'll just end up being "the way it is" rather than something to fix. I've read that a strategy for this is create an ad and pay people $50 to come in and try to use your software. Tell them to do something in your software and see what they get hung up on. The worst UX problems will be hit by nearly every user. As simple as that is, none of my employers have ever done this. The closest was one of the bosses asking his wife to try out the software.
- IanCal 3y ago
  You can go one simpler if you need, you can use something like https://www.usertesting.com/ https://www.usertesting.com/ which will do the ad side of it for you. I used to do testing for them for some beer money. You'd use something for the first time and try and achieve some task, often talking through it, and it's all either screen recorded or it used to be videoed for phones. Having someone in person can be extremely valuable for other reasons, but this can be a quick approach. For larger customers, going onsite and watching them use your tools is so valuable. > Once people are familiar with the product (devs, QA, and anyone who has used it before) then they're "tainted". I broadly agree, though this is where I'd split out really good QA people I've worked with. The added advantage is they can also explain the change required that would get some user X to have a better experience (e.g. how your autistic users may get more stuck at a certain place, or how to change the flow such that a 3-4 year old can navigate a UI).

by IanCal 3y ago

INTPenis 3y ago

>This kind of process is extremely valuable and should be done by devs more often. The fact that it's not being done doesn't bode well for their perceived engagement to this project. I remember when it launched and how much they hyped it up to be the future of secure messaging. That was how many years ago now? It was pre-pandemic. I'm a lover of all selfhostable federated solutions so I actually hosted a Matrix server for a couple of years. My conclusion is that it's just not ready for production scalability. And you can't migrate easily between implementations because of their unique database design.

vallode 3y ago

QA is a massively underappreciated position. A QA person that knows when to automate, when to manually test, and how to report and file issues relevant to the project can save a significant proportion of hours on a project overall. I wish many more companies included budget for QA, it saves developers a lot of time. A bit of a side-note: this sort of analysis is a great answer to "I want to contribute to open source, how?". Some fairly simple wins for significantly better user experience, and no coding required!

ChrisMarshallNY 3y ago
The old wisdom in the US, is that if you have "Quality" in your job title, your career is over. At the Japanese company that I used to work for, it meant that you were one of the most powerful people in the corporation, and was a sought-after adornment. Different strokes, and all that...
- ivan_gammel 3y ago
  I always give the power to my QA team to block any release no matter what and to give higher priority to tickets than product manager. If CEO wants to override, I cover them and take the blame. This is not a guarantee that there will be no bugs in production, but it saved us a few times.
  - ChrisMarshallNY 3y ago
    The Japanese testers were the best I'd ever seen. They never reported a "NotABug." They could back up every report, and give exact reproduction steps. They found weird, obscure corner cases, and that was by hand (they hated automation tools). They had 3,000-line Excel spreadsheets. If even one of those rows failed, the whole shooting match (like an entire product line) could come to a halt (so that meant they had to cross their t's, and dot their i's). They seldom had "opinion-based" reports, and, when they did, the report was presented by the manager, after long discussions. The company I worked for, was renowned as one of the highest-Quality optical corporations in the world.

thibaultamartin 3y ago

Hi, I'm the Thib person mention in this article, and I agree that QA is super important. I can mostly talk about matrix.org, since I have little power over the Element clients. Disclaimer though: I'm technically employed by Element (to make paperwork simpler since I'm France-based, Element has an entity in France, and the Foundation is UK-based), but I'm working for the Foundation full time. This kind of article is super valuable since it gives us the perspective of a new user. I opened https://github.com/matrix-org/matrix.org/issues/2178 https://github.com/matrix-org/matrix.org/issues/2178 to translate the gripes mentioned in the issue into actionable items for us. I took action on the most urgent one (updating the Try Matrix page), but want to take the time to go beyond the surface symptoms and address the root cause of the other gripes. On the Foundation side, we're a small but mighty team of four. The website is currently maintained part time by me and a volunteer who is doing an excellent job at it. As I wrote recently in a blog post "Tracking what works, not people" (https://ergaster.org/posts/2024/01/24-tracking-what-works/ https://ergaster.org/posts/2024/01/24-tracking-what-works/), I would love to have the resources to conduct user research and user testing on the website but I unfortunately don't. We deployed privacy-preserving analytics to see where people drop and what confuses them. It's not nearly as good as proper QA and user testing, but that's what we can afford for now. Overall I'm grateful to the author for documenting their frustration, and even more grateful for reacting constructively to our responses and integrating them in the blog post! One of the strengths of open source is to find and address issues collectively. I consider this blog post to be a good open source contribution. If people around believe in our mission and want to help us with their brainpower, I invite them to join our "Office of the Matrix.org Foundation" room: https://matrix.to/#/%23foundation-office:matrix.org https://matrix.to/#/%23foundation-office:matrix.org For those aligned with our mission and who want to support us financially, the https://matrix.org/support/ https://matrix.org/support/ page should give you all the information you need to help us out.

gowings97 3y ago
Do you have any thoughts on how you might improve this workflow?
- thibaultamartin 3y ago

justin_oaks 3y ago

If you're trying to make a good onboarding user experience then you should do your onboarding testing with people who've never seen the product before, not devs or QA. Once people are familiar with the product (devs, QA, and anyone who has used it before) then they're "tainted". They'll remember the weird way that they had to work around an issue, and that'll just end up being "the way it is" rather than something to fix. I've read that a strategy for this is create an ad and pay people $50 to come in and try to use your software. Tell them to do something in your software and see what they get hung up on. The worst UX problems will be hit by nearly every user. As simple as that is, none of my employers have ever done this. The closest was one of the bosses asking his wife to try out the software.

IanCal 3y ago
You can go one simpler if you need, you can use something like https://www.usertesting.com/ https://www.usertesting.com/ which will do the ad side of it for you. I used to do testing for them for some beer money. You'd use something for the first time and try and achieve some task, often talking through it, and it's all either screen recorded or it used to be videoed for phones. Having someone in person can be extremely valuable for other reasons, but this can be a quick approach. For larger customers, going onsite and watching them use your tools is so valuable. > Once people are familiar with the product (devs, QA, and anyone who has used it before) then they're "tainted". I broadly agree, though this is where I'd split out really good QA people I've worked with. The added advantage is they can also explain the change required that would get some user X to have a better experience (e.g. how your autistic users may get more stuck at a certain place, or how to change the flow such that a 3-4 year old can navigate a UI).