A transcript of Episode 233 of UX Podcast. James Royal-Lawson, Per Axbom, and Richard Whitehand talk about the benefits of usability testing, which are more than just discovering issues with your design.
This transcript has been machine generated and checked by Stephanie Staudinger.
Transcript
Per Axbom
Thanks to Renaud, Lea, Alesha and Johan for supporting us podcast. We really appreciate it.
Computer voice
UX podcast episode 233
[Music]
James Royal-Lawson
Hello everybody. Welcome to UX Podcast coming to you from Stockholm, Sweden. We are your hosts James Royal-Lawson
Per Axbom
and Per Axbom
James Royal-Lawson
with listeners in 192 countries, from Rwanda to Russia,
Per Axbom
and today’s guest, Richard Whitehand has more than 25 years of experience working with usability and user centred design. And his consultancy work has always focused on improving products and systems from a user’s perspective.
James Royal-Lawson
Now, both Per and I have known Richard for a really long time.
Per Axbom
Yeah, I think you much longer than I, I think
James Royal-Lawson
I think it’s the very end of the 80s when Richard and I were both running the same type of bulletin board system. Back in the day when it was all very new and different to the websites of today.
Per Axbom
Yeah, late 90s, in my case that when I met Richard for the first time
James Royal-Lawson
But he, he runs probably the most modern usability lab here in Stockholm.
Per Axbom
Yeah, it’s so much fun to walk around there and have him brag about the ventilation system, which you’ll hear more about in our interview as well.
James Royal-Lawson
We were hoping to actually be in the lab to record the interview. But we couldn’t actually be in person in the same place.
Per Axbom
Yeah. Anyway, always impressed as we are by the whole labs concept. We sat down with Richard to talk more about usability testing in general and the benefits and challenges of usability labs in particular. For instance, should we even be calling them labs.
James Royal-Lawson
Richard, maybe you could take a moment just to describe the usability lab to people. I mean, I can see that you’ve sat in your usability lab at the moment, but just describe it for for the listeners.
Richard Whitehand
Well, essentially it’s two rooms with a one way glass between the two rooms. And we try and set up the one room where the test participant sets appropriately for the sort of test material that we’re working with. So if it’s, for example, digital TV app or something like that, then it could be set up as a living room with sofas and chairs, and coffee table.
If it’s a typical sort of website, then we’ve just got a desk and chair here. And so we’ve got a room where the test participant sits and then you’ve got a room where the observant, observers that the team that is usually from the company that we’re working for sits. And that can consist of anything from project managers to developers, to in some cases board members that can observe these things.
James Royal-Lawson
And I’ve I know that you have quite an impressive ventilation system there as well.
Richard Whitehand
Yes… yes, we like to we like to show, show people that. That was one of the biggest problems any, anyone that’s that’s been in a usability lab or two will probably tell you that the biggest problem is ventilation. Not not in the test room with the with the test participant but in the observation room.
There are many labs around the place which are essentially can be converted meeting rooms or converted office rooms where the observation room is patched dimension for two or three people and then you’ve got masses of observers cramming into there to watch. And they run out of air. And it’s, it’s sound insulated. It’s also fairly insulated as far as the air is concerned. So unless you’ve got a good ventilation system, then people start to drop off, which is not what you want when you’re observing usability tests.
So this place which we built a year and a half ago, then we decided to really properly. We got into big discussions with the architect who, at one point simply refused to go any further with how much air we wanted to put into the observation room. It’s really fantastic, in fact, so we can adjust it depending on the number of people.
James Royal-Lawson
I really wish a lot of the meeting rooms I’ve been in over the years, had anything like the ventilation you’ve got in the observation room.
Per Axbom
Well, that’s the thing, isn’t it? The experience design that goes into just the building of the premises, and the care you’re taking for the people who are coming to do the tests. In a way that nobody else does, if you’re inviting people to your office or something like that. You never think about those things. But it’s obvious that you’d think a lot about it.
Richard Whitehand
Well, we’ve seen a huge change. Well, certainly amongst our clients in the sorts of people that are interested in coming and watching these sorts of studies. When we were doing these studies 10 or 20 years ago, in my case, then very few people used to come and watch. And if you did manage to come along then it would be designers and developers.
These days, well we just did a test last week where they were board members of a big organisation, that are invited to come along. So it’s very much now something for working with or encouraging really the whole organisation to become user-centric. And in fact, one or two of our clients are stating that outright which is great to hear, you know, with this has been really powerful tool. For helping to make the organisation more user-centric to understand really feel for our customers.
James Royal-Lawson
So you’ve maybe, so you’ve seen the move from maybe where the report at the end of usability testing was the main deliverable to a time where perhaps something a little bit less concrete is now one of the main deliverables.
Richard Whitehand
In a way yes. We think that – we like to think anyway that at least half the value of doing this sort of study is the impact that it has on the organisation as a whole. At least those – those people from different parts of the organisation that are invited to participate. And it really was a wide range last week. People from from marketing and sales people from support. Board members, IT and of course, the UX staff as well. And they, you see how engaged they become in what’s going on when they’re sitting there on the other side of one-way glass, like the customers just on the other side is struggling with whatever it is in the interface. And they really feel for that person, they really understand what’s going on. And they take that message back with them. And it’s not just an understanding of what that particular problem was and how they might fix it. But it’s a better understanding of who they’re actually designing for, and the sorts of challenges that they’re facing. So they can apply that knowledge in all sorts of different levels as it were.
So, we think and more and more of our clients really thinking this is this is one of the biggest values of doing this. The problem we have is that more and more people want to come and watch these things live. So when we built this place, and we built slightly larger observation room, but it starts to get, you know, the usability test starts to be about handling all the observers. Rather than the test participants because it’s still one on one testing we’re doing. There’s only one test participant. And we also don’t like to have a situation where there can be perhaps too many people observing because you know, if you’re sitting with a participant and the participant asks, tell me exactly how many people on the other side of the glass there? We answer honestly. We’re not going to lie to people. So it’s nice to be able to say, well, there is five or six people that are working with the development of the website.
James Royal-Lawson
I mean, imagine it so you don’t really want it to become usability theatre. Where you’re putting on a performance with like 200 people in the audience.
Richard Whitehand
Yes, strange you should say that. But I recall many years ago, Jared Spool, for example, talking about cases where he would have almost like that, in the front of a large meeting room, where the user would sit with the moderator, and then behind would be all these people that were asked to keep quiet. It’s – I’m sure it can work if it’s handled very carefully, but we prefer a situation where we’ve got a little bit more control over our observers. And they do get itchy feet sometimes as well, you know. Some of the, some of the designers or developers like, the user needs to be told this, you know. Can’t you…can’t you just tell them that? It’s like, well, no. We want to understand the real situation and what’s actually – what they would do if they were sitting at home or at their office.
James Royal-Lawson
I think that’s a that’s a very important point the – that feeling you get when you’re indirect – when you’re directly observing a user and witnessing them struggle or fail to do what you thought was quite obvious perhaps because you’ve been part of designing it. Compared to the abstract nature of many of the other testing tools we’re encouraged to use nowadays. And I’m thinking on – and I’m thinking about things like when we were using analytics tools or just A/B testing, for example. Where you, there’s a, there’s an absolute disconnection from the context and reality, there the actual moments of the of the user doing something.
Richard Whitehand
Yeah. Now, I mean, we’ve essentially should say that because just recently, we’ve had now two or three clients who have seemingly spent years doing, working with their analytics and different types of A/B testing and so on. We had a small startup company here actually. Strangely enough in the in the funeral business – startup – funeral company, byrå, and they been doing a huge amount of work and good work with – with, on their analytics, a lot of A/B testing. Following up on all sorts of different statistics also connecting it up with a call centre statistics and so on.
I think it was quite an eye opener for them to actually come into this environment to watch users one at a time to understand the actual whole situation in which they’re trying to use the website. Why? Why this interaction is occurring? It’s one thing to sort of get data about what happens? Where do they click? But how does that come about? What are the reasons behind that? How are users thinking? It’s really important to understand when you want to actually make important design changes or design decisions. And we feel that talking to some of these companies that there’s a lot of focus on, Okay, well, should I should I put an extra button there? Or should I word that button differently? Or, these sorts of changes in design where they’ll try and follow up on the statistics. So I added that in. That was an improvement. Great! You know, I’m going in the right direction. Whereas they can, in fact, being missing the big picture.
You know, you can maybe achieve a very small improvement by renaming a button or adding another button. Or whatever it may be. But if you don’t really understand what’s going on, what the user is trying to do and why, then you can miss the really big changes that need to be made to the design so that it’s actually going to have a big impact on the UX at the end.
Per Axbom
There’s also the aspect of today, most UX teams, both do the research and design, but also do the testing of their own designs. And whereas always of course, when someone comes to you, you’re an independent tester. You haven’t been involved in the building of the site. So that means a lot, because that’s what I thought of when I heard, so there’s a designer into the observation room who wants to raise their hand and say, no, we have to tell the user this. That unfortunately, of course happens in user testing where you’re testing your own product. That people actually interrupt the person who is testing your product and do that. So I think that’s a huge value you’re bringing to the table as well.
Richard Whitehand
Yeah, and we also do a certain amount of design work in certain projects. So we end up in that situation ourselves. And we have the absolute rule that if you’ve been involved in, I mean, even if you haven’t really worked with the design, but you’ve been a stakeholder in the design, or you’ve contributed in design decision making, that you are not the one that runs the test. To find someone that’s not been involved, and to run the tests. And I find it difficult to believe that an interface designer that’s really worked with their own design a lot can actually run a test in an unbiased manner. It’s not possible. And you’re too much in a situation where you want to, well if I just… you know, if they just got it as it were. They’re not getting it. That’s, you know, they’re getting something else.
James Royal-Lawson
I think, that’s an excellent point. Is that I know that… you know that in teams I’ve worked with, that amongst developers it’s very common that you would get another developer to check your code. To check what you’ve programmed for exactly that reason. That your ability to judge and see – you know because there’s so many ways you can program things. Then you know, you become blind to the fact that, yes it might work but maybe it’s not – maybe the most efficient way to do. Or maybe there’s other ways you’re doing it. Or maybe you’ve got standards that you haven’t quite followed, you forgot one of them. So exactly the same benefits for programmers. You get another programmer to check it and give you feedback. And like you’ve just said with usability testing, sounds perfect for designers to not test their own designs.
Richard Whitehand
Absolutely. And then of course you come into the question of cost and time. And naturally in some situations then there isn’t an alternative. And we would always then argue, well, doing something is better than nothing as it were when it comes to user involvement. The key messages is to at least keep quiet. A user test is not about…it’s not an interview. If you can simply get to the point where you can give people a situation in which they need to achieve something with your site or your app. They understand from their own perspective what it is they need to achieve. And let them do that and watch them. Then you can come a long way.
I’ve seen also too many usability tests where, and it’s partly brought about again by the people that are watching, where they feel well, you know, the moderator is not asking any questions. What are you doing?! You know, the thing happening here. As if it’s some sort of interview, and I’ve seen tests where really essentially every time a user clicks on something or presses on something that the moderator says, What were you expecting to happen there? And what do you think of this page? And do you like the colour scheme? Whereupon the poor user has completely lost track of what it was they were trying to do.
James Royal-Lawson
Now you’ve been doing formal usability testing, Richard for for over 20 years. Would you say that usability testing now is more prevalent than it was 20 years ago? Take into account the fact that we’re more designers now than we were 20 plus years ago.
Richard Whitehand
Difficult question. I mean, yes and no. 20 years or so a bit ago – I mean, we’ve got the great expansion in the web, of course, the mid 90s. Let’s say 25 years ago and there you saw a sudden realisation at some level that now we’re designing for the general public. You know, until that point people were designing with a more specific target group in mind. When it came to most development projects that were going on, it was internal systems that may be that we’re working with. But now we were designing websites for the general public, there was a huge expansion at that stage in a number of designers working and we saw a growth or the beginning of that growth in usability testing.
Now, of course, there are a lot of people working with – with UX, whatever UX is. At…it’s…in different ways and there are lots of different approaches to user testing. So I mean if you sum all those up, all the different approaches. Whether we’re talking about remote moderated, unmoderated, in person. So these are different kinds. And I’m sure there’s a lot more going on than back then. I would argue not enough in many cases. What we should be saying when you…you said earlier that I’ve been working with formal usability testing for over 20 years. And that’s, that’s true. But I would say, most of the testing that I’ve been working with is not what I would call formal. We need to see a lot more informal usability testing going on. Much more small scale rapid user testing.
And what we’re starting to see here, because we also, I didn’t mention that we also rent out the facilities here to other organisations. And we’re starting to see people coming to us and say, Well, you know, we just want to book one day here, one day there at rapid intervals. Where they’re going to be doing a series of tests in a relatively short period of time. And that’s very important to get this rapid, iterative testing into agile processes or whatever’s going on in the organisation. We understand that we can’t just test once. We need to test a number of times. It’s not just a question of testing, oh when have we found all the problems and we know what to do. And many organisations are starting to [indiscernable] that now which is nice.
James Royal-Lawson
I mean it’s good that you picked up on my use of ‘formal’ there because of course what I was meaning was lab based usability testing. The planned, structured lab based usability testing. As opposed to very many other flavours of testing that we do and ad hoc and geurrilla… sketches at Starbucks. All these kind of things, which all have…
Per Axbom
…remote online tools.
James Royal-Lawson
Yeah.
Richard Whitehand
Yes, I suppose it does come across as maybe a little bit more formal. To me it’s more just an environment, which enables us to involve the designers and the developers in a different way to, you know, going down to a cafe or something like that. It’s not a replacement for that either. I mean, I would say it’s probably about maybe just over just over half of the studies that we do that are in the lab. There are many studies that for good reasons need to be done in the field. And you’re always weighing up the benefit of being able to have the design team present and the value that that gives, which we see is great. Compared to, okay, what are we losing by not doing these studies in a particular context?
Per Axbom
That’s interesting because that means there’s a maturity factor and there’s a factor of where you are with the design, how fast you need to go between the tests. So you’re obviously weighing in a lot of things because when I’m going to conferences these days very few people even talk about lab testing. It’s not on the agenda for a lot of designers. And it seems unfortunate in a way because the way you’re describing it is it’s really set up to weigh the pros and cons of doing it this and that way.
Richard Whitehand
I feel that as well to some extent. At the same time… I remember 20 years ago or more when we were attending exhibitions and just for fun we went round in white lab coats and made it all a bit sort of scientific. It got attention but I’m not sure it’s the right image one wants to have of a lab, or even fact calling it a lab. But I mean, that’s what it’s always been called. It’s to me very much about facilitating involvement of the design as a design team. And it shouldn’t be seen as such a big special thing in a way. But we do feel, I mean, even though the one way glass of course looks a little bit, I don’t know, can look off putting we tried to we try to not have the participants even facing the glass, in fact, but it’s very much about connecting the the observers, the designers, developers, project managers with what’s going on.
So it’s quite a practical setup here. I know there were many or many, there were a few organisations if you go back 20-25 years. Some of the big banks for example in the UK. Ericsson in Sweden. Having quite fancy labs because they were both used for testing but they’re also used to sort of wheel in key clients and so on to show, Look! This is our testing facility. Doesn’t it look wonderful.
James Royal-Lawson
Yeah they were demo facilities as well. That you – you’ve got to show off the kind of the future world that they could be. I mean, Ericsson had a lot of those kind of environments to show off the next technology that was coming.
Richard Whitehand
I think we all need to better understand the people behind our data. Understand the why and the how, for the user interaction that’s going on in our interfaces. And this is an extremely powerful tool that I think some people have bought into in a big way. We’ve got a number of clients that come back many times each year to do this, and other people perhaps still see it as you know, this scientific lab that’s not a representative environment. And it’s not realistic to run tests in that way and it’s too expensive, but there’s lots of arguments. I think it’s got some huge strong points.
Per Axbom
Certainly does and I think more people are aware now. I always enjoy talking about these things because I mean, I hear both sides. I hear all sides all the time. But it’s so… you’re so balanced about it. There are pros and cons to everything. And so you just need to think about those things. You need to think about what you’re testing and what you’re trying to find out. And what…who you’re trying to influence at the end.
Richard Whitehand
I mean, we’d all love to do usability testing that’s, you know, 100% realistic in every respect. It’s not, it’s not possible. And I remember reading an article. I’m trying to think who wrote it but someone that was conducting, regularly conducting research in people’s homes. And this is often brought up, you know, if we’re going to be conducting user testing of a website or an app that’s used in people’s homes, then we should be testing it in people’s homes. Well, this lady discovered that it took several days of being in someone’s home before she felt that people started to act, you know reasonably normal as it were.
And sure, if you’re prepared to invest that amount of time in your research for one participant then go for it. I think when, certainly in Sweden – I’m sure it applies in other countries but if you were going into people’s homes as a relatively unknown person, of course they’ve tied everything up and their very conscious about things that are going on and want to make an impression, and it is not realistic. So you can’t have fully realistic. What I think you must look at is what’s the biggest… what’s going to make the biggest difference to my project? What is going to bring the biggest benefit? Not just for me as the UX researcher. But for the project and the organisation as a whole.
And these days, I truly believe we have to get our colleagues, our managers, our board members on the user-centric bandwagon. And they need to understand why this works important. They need to…they’re the ones, you know, investing in us as designers and developers. They’re the ones sitting on the money as it were. If they really understand the benefits of this kind of user research, then there’s more potential for us to do the things that we need to do. To make a difference to design.
James Royal-Lawson
Definitely
Per Axbom
Perfect note to end on. Thank you for joining us, Richard.
Richard Whitehand
Thank you.
James Royal-Lawson
Thanks a lot.
[Music]
James Royal-Lawson
So I think one of the often or the most overlooked perhaps aspect of usability testing is the whole thing to do with how it helps create a culture around human centred design, that you expose people in your team, to the sometimes suffering, that your products and designs actually causing people at firsthand.
Per Axbom
Just that, yeah ,that just that aspect of involving people in the usability testing is what will help the team forward and have it top of mind always when designing in the future.
James Royal-Lawson
And this is, this is something that Jared Spool calls is exposure hours.
Per Axbom
Oh right. Yeah.
James Royal-Lawson
And what Jared says is, what we talked to him before about it is that the ability to when your team is maturing, and a better answer to the five to eight number of participants in usability testing is you should aim to increase your team’s exposure hours. Everyone in your team. If that beyond your team, as Richard pointed out in the interview, that you should be looking at making sure even other stakeholders, CEOs even would be involved in your testing to witness what’s going on.
Per Axbom
Exactly. And it’s such a balanced way of putting it. Talking about how lab testing is not always the right thing. Sometimes it’s the right thing. You always have to figure out what are you trying to achieve? And that’s what I’m really taking away from this interview. It’s so important to always figure out what is the goal of usability test, and it can be a lot of different things.
James Royal-Lawson
Yeah. And by and large, the more contact you…exposure and contact you have with your users, the better.
Per Axbom
And funny you should mention Jared, of course, because I recommended listening his Episode 179 with the Jared Spool. Where we talked about NPS Net Promoter Score, and five to eight usability testing myth. Five to eight participants usability testing myth.
James Royal-Lawson
Don’t forget to check out when our next fika is, by visiting uxpodcast.com/fika And you can also help support us podcast financially by visiting us podcast.com/support
Per Axbom
Remember to keep moving.
James Royal-Lawson
See you on the other side.
[Music]
James Royal-Lawson
Why did the PowerPoint presentation cross the road?
Per Axbom
I don’t know James. Why did the PowerPoint presentation cross the road?
James Royal-Lawson
To get to the other slide
Per Axbom
[Laughs]
This is a transcript of a conversation between James Royal-Lawson, Per Axbom, and Richard Whitehand recorded in March 2020 and published as Episode 233 of UX Podcast.