A transcript of Episode 229 of UX Podcast. James Royal-Lawson and Per Axbom and Erika Hall discuss research and Erika’s book Just Enough Research. 

This transcript has been machine generated and checked by Lizzie Hedges.

Transcript

Per Axbom
Thanks to Johnny, Cristine, Joyce and Duncan for supporting UX Podcast. We really appreciate it.

James Royal-Lawson
And there are three ways you can help support UX Podcast:

• One of them is, visit uxpodcast.com/support and contribute with a donation.
• Two, you can email uxpodcast@uxpodcast.com and volunteer to help proofread our transcripts.
• And thirdly, you can buy a ticket to From Business to Buttons this May in Stockholm. If you use the code uxpodcast, just one word, you get 10% off and we’ll get a donation from the conference.

Computer voice
UX Podcast episode 229

[Music]

James Royal-Lawson
Hello, everybody. Welcome to UX Podcast coming to you from Stockholm, Sweden. We are your hosts James Royal-Lawson

Per Axbom
and Per Axbom.

James Royal-Lawson
We have listeners in 191 countries from Namibia to Italy.

Per Axbom
Erika Hall, co-founder of Mule design studio and author of Just Enough Research will be speaking here in Stockholm in May, at From Business to Buttons.

James Royal-Lawson
And six years have passed now since Erika published Just Enough Research through A Book Apart. It’s a guide book with trusted research methods.

Per Axbom
And recently Erika released an updated second edition. We didn’t get around to talking to her when the book originally came out, so we took the chance to chat to her now.

[Music]

James Royal-Lawson
So Erika, the first chapter in Just Enough Research is ‘enough is enough’. But when I looked at it, it actually is what is research and I thought oh, is that a strange question? I when I thought a little bit more, I thought no, actually, that’s an excellent question. So what is research?

Erika Hall
Research is simply one way to put it is systematic inquiry. It’s just trying to answer a question in a somewhat formalised manner, in just an organised manner really. Like we do it all the time in daily life, where you you think of something you want to know, you do some activity in order to answer that question. And then you decide when you’ve attained a level of confidence, and you think to yourself, oh, I feel sufficiently confident that I have answered my question.

That little process we repeat countless numbers of times in our days in our work lives. And the reason I wanted to define it in the book is because people freak out a little bit in a business context and a lot of people who are unfamiliar with the concept of just applied research in a professional setting, imagine the report. But research is not writing a report, research is simply going through that process of identifying a question and answering the question to a level of confidence that you establish sort of in advance or over the course of doing the work.

Per Axbom
Which is a problem then because everyone lies?

Erika Hall
No, yes, yes.That’s what makes research with humans fun and interesting is that everyone is lying to themselves and others all the time.

James Royal-Lawson
So if everyone lies, isn’t isn’t that a little bit paradoxical, in the sense that we’re doing research to to get some answers to our questions. So okay, how do we stop our research, from being just guesswork dressed up in the lies that we hear?

Erika Hall
Well, that’s where the systematic part comes in. And one of the greatest areas of confusion. And I didn’t even really fully come to terms with this until within the last couple of years. Because, you know, I do workshops around research and I consulted on research, and I finally realised the extent to which people confuse the research question what you want to find out with an interview question or a survey question, which is what you ask someone directly, like this is the hugest area of confusion.

I think that leads to a lot of bad research and a lot of taking lies at face value. Because just because you want to know something, such as, you know, how likely are people to use my product, perhaps, that’s a very popular question. You can’t ask someone that but if you start looking at surveys or an interviews people will ask questions like, how likely are you to use this product? You cannot ask someone that question because it’s impossible for them to answer. However, there are other types of research you can do and other things you can ask people to get information that will help you answer that question. But it’s not as simple as asking your question directly.

Per Axbom
And it must be so easy to be I mean, you think something is true. You hear someone say it kind of like it’s true when you want to believe it. So you that you’re drawn into those conclusions. How do you fight off yourself and your own bias?

Erika Hall
Well, the first thing you have to do is recognise it and acknowledge that we all have it. Sometimes people in business use bias as an excuse not to do research. They say, oh, we’re all biassed so why even bother. But if you understand that we’re biassed, then you can correct for it, then you can when you design your research, you can talk with your team, what sorts of bias are likely to creep in? And what can we do to mitigate those and you can do it in terms of your recruiting to make sure that you’re really recruiting a representative sample of people. You can do it in how you design the conversations, how you analyse the, the notes you have afterwards, you can say, oh, we heard this from one of the people we interviewed, how likely is it that the way that they reported their behaviour is true?

And because really one of the biggest ways to prevent bias is not asking people direct questions about you know, their thoughts and feelings and taking those as fact, what you can ask people about are behaviours that they don’t feel self conscious about in a very descriptive way. You can ask someone to like a really good example, you can ask somebody to walk the walk you through their day. Yesterday, you say, okay, just tell me what you did from the time you woke up to the time you you went to sleep, just just walk me through and then you ask them, tell me more about this, tell me more about that. And you refrain from asking them very direct pointed questions about things like how much money did you spend on something? Or how do you decide how to spend money like questions like that often make people self conscious, and then they start telling you a different version of the truth.

Per Axbom
Exactly. So you mentioned when you were defining research there that when you feel sufficiently confident, how do you know how do you know when you’re sufficiently confident, how do you know when to stop doing the research?

Erika Hall
Well, if you’re doing quantitative research, you establish, you know, a level of statistical confidence. And when you’re doing qualitative research you, you get to something called saturation where you’re hearing sort of, you’re hearing the same patterns again and again and again. And you might, you know, you always have to be careful to make sure that you’re not being overconfident. And these are real patterns. And a lot of that is just using critical thinking. And I really strongly encourage people to do, you know, research work and analysis with their teams, in order to really help make sure that confirmation bias isn’t creeping in.

You know, you really want to get to a point of like, okay, we feel pretty confident about this. And a lot of times, it’s good to have a discussion in advance, like what’s our standard of confidence? And you can you can adjust over time. But it really is. It is a sense that you develop, and you’re never going to be, you know absolutely correct, right, you’re never going to be certain. And I think that’s what trips people up is this idea of you do research until you’re certain. We’re never certain. But in life, we do these things, and you know, like, oh, I’m pretty certain, like I’ve done enough research, I’m pretty certain this is the bicycle that will be right for me, and I will spend $1,000 on this bicycle. And it all, it all depends on how big of an investment you’re making based on that confidence. If it’s a small investment of time, or resources or money, whatever, you maybe have less of a standard of confidence. But if you’re making a really huge investment, you want to be a little more confident that your assumptions or your conclusions reflect reality.

James Royal-Lawson
Of course, because you don’t want to waste a lot more. I mean, if you just if you’re just making a tweak, then it’s not going to be so expensive to do another tweak. What if you’re building a new product, then it’s gonna cost a lot of money to do another new product if you got it wrong.

Erika Hall
Right. And And the funny thing is, so many companies make huge investments based on no research. So and then they then they get very anxious and stressed about doing research and are we talking to enough people? So they want to avoid it. It’s a very, that’s the biggest paradox to me, is how many organisations don’t want to talk to users or customers or observe people out in the world because they say, oh, we can’t be certain so why bother? And yet they’ll make huge investments based on what the CEO feels like or what another company is doing without really thinking like, oh, does that apply to us does that apply to our market. And they’ll just make these huge guesses based on total feelings or wishful thinking where they could do some research and at least have some confidence in that what they’re doing reflects the real world.

James Royal-Lawson
I think that’s pretty good. That brings me on to organisational research, that you have a chapter on in the book, which I think is you class it as a type of research. But it’s I think it’s often overlooked about how much you need to understand about the environment you work in, in order to even start doing research out there in the rest of the world.

Erika Hall
Yeah, that’s, that’s absolutely correct. And this is why a lot of a lot of design research is really focused on user research, and sometimes I will hear myself referred to as a user researcher. And that’s not how I see myself because I think that understanding the user is one small part and in many ways, the easiest part of the research to do. And thinking about the organisation, and how it works and how it makes decisions. That is often what has the greatest influence on the type of design work or the type of products that make it out into the world. I describe organisations as the social context in which decisions are made. And if you don’t understand that, then you might come up with something that makes perfect sense for the user or for the customer, and the organisation can’t produce, can’t support, will resist it.

Erika Hall
And I think a lot of designers and researchers get very confused by this. They say well, we but we did all our research, and why is the organisation rejecting our conclusions or ignoring us or doing something differently? And it’s because they don’t understand the organisation and I think this is the greatest gap in design thinking is design thinking doesn’t touch on the organisation at all, when it it puts itself forward as like a methodology for helping to make design decisions or help solve problems, like there’s a huge blind spot that just sort of assumes that the organisation is willing and capable to follow through on the the ideas and recommendations that come out of that sort of process.

Per Axbom
Right. Now I have a situation like, where in research, I go out, and I have a problem to solve. And I want to get information on that. But I find all these other problems, that actually basically nullify the initial problem because I realised that the initial problem is the wrong problem. Then you have to go back to leadership and convince them. I mean, so it can get really, really complex unless you’re happy, just okay, I’m just going to work with the interface or I’m going to realise, no, the people I’m actually interviewing, they don’t need a better interface, if we just bought bigger screens for them that would solve the initial problem, and more. Do you come into situations where you have to have to argue that my research found something that we weren’t looking for, but really, really is more important than what we were working on before?

Erika Hall
Oh, yeah. All the time. And the way to do that, and the thing that, again, a lot of designers and researchers shy away from because it’s uncomfortable, is establishing that relationship and that understanding of and with the leadership. If you aren’t clear on the business goals, you shouldn’t be doing any research or any design. Like that is the most important first step that often gets neglected people go straight to what do the users need? And it’s good to step back and say, Well, what is the leadership vision? What are our business priorities? Is it we’re just trying to find the problem space, trying to find a user need out there that we can make a business around solving? Or do we already have a an operational company, we already have a business model that this has to work within. So those should be the first constraints of problem solving is what will make the business successful. And then you use that to guide making your research questions. And that will help prevent that situation where you’re looking in the wrong area. You know, it’s just you really have to be clear on what type of question you’re asking like, are you really trying to just find new ideas for problems to solve? Or are you trying to learn things that will help make your business or your organisation more successful in a particular way.

Per Axbom
And this sounds like it’s really important to talk about these things before research even begins.

Erika Hall
Oh, yes. Oh, so important.

Per Axbom
Yeah. Because I feel that a lot of especially when you’re a junior UX designer, you know, you want to do research, I go out and interview people, then you’re not set up to actually take that step back and make those judgement calls. But that is what actually grows our industry and makes people feel respect and want to want to use us for this type of work is that we actually do solve more things than we actually were asked to do in the beginning.

Erika Hall
Right, and it’s, it’s not just junior UX people, a lot of times some very experienced and proficient researchers, if they come out of academia and they aren’t used to working in industry, they might be very skilled at doing very rigorous research but they are completely blindsided and surprised by the political environment. And they don’t think to study that environment as an important factor in the design. And so I’ve I’ve talked to people who are very, very senior, who don’t necessarily have the tools to set up these projects for success in advance, like because it it, they haven’t been in that sort of environment. So that is very new to them. And it’s just always really important to not go straight to the technique or the methodology. This is where there’s so much focus on the discussion around researchers, what tools, what method, how do you interview people, and I just really encourage people to step back and say, are you in an environment where the decision making process is clear where the decision making process is based in evidence, like, if you you don’t have that, before you try to choose a question or choose a tool, choose a platform, all of these other things, your research will not have an effect. And it will be just as bad as if you didn’t do any at all.

James Royal-Lawson
I wonder how much of this is is related to the ability of critical thinking. You’ve mentioned the processes and this this problem is down to that we’re with a lot of time very rigidly stuck to our processes that we’ve learned. And you say about stepping back and that that is a form of of critical thinking, we have to look at your tools. Is that maybe the thing that we’ve dropped the ball on a bit in, in our industry, on the way?

Erika Hall
Oh, yeah, I definitely agree with that. I think we’ve gotten so excited about the creative part of design and you know, understandably so, it’s exciting to put new things out into the world and there’s the whole of, of, of the critical thinking where you’re really thinking, critically thinking about how you’re making judgments what your basis of decision making is. And then there’s the other part, which is the criticism part of design, where you’re asking, why are we designing this? Why are these our goals like, is this an appropriate solution to this problem? And I think there’s sometimes designers can feel, I don’t know if it comes from insecurity, but there’s a sense of, of criticising the process or criticising the work is mean spirited rather than a core part of the work. You know, asking those questions about why you’re doing things and thinking about whether you’re taking the right course of action or doing things for the right reason, or asking whether a particular solution really does measure up to the goals you’ve set out. All of those are just as fundamental a part of the design practice, as you know, creating things and making things and it’s not negative or bad or mean. It’s a core part of the practice of getting really good, effective design out into the world.

James Royal-Lawson
I wanted to ask you about the the second edition of Just Enough Research has just come out and you’ve added a new chapter, and the new of chapter is all about surveys, but I haven’t read it so I don’t know any more than that.

Erika Hall
I left surveys out of the first edition on purpose, because I thought of it as a broadly accessible introductory book and I considered surveys an advanced technique. And I didn’t want people to do surveys unless they really really knew what they were doing with them already. So I said, that’s not that’s not a topic for, for this type of book. But then, you know, six years pass. And there are so many tools and platforms for running surveys, and there’s so many terrible surveys out there and so many decisions being made based on them that I thought, okay, I’m going to have to put some information in there to provide some guidance and a lot of cautionary guidance about what it really takes to make a survey that you can feel confident in. It’s a lot of work, it’s more work to do a survey well, but the tools make it seem so easy. And, and so I think it’s really easy to make a lot of really, really terrible surveys, and it feels quicker to people. It feels like less effort. But if you’re doing it right in a way that you can actually have some confidence in, it’s it’s a lot more work. And so I thought it was necessary to include that in the second edition.

James Royal-Lawson
Yeah, I think you’re completely right surveys are very complicated to do very well and very easy to do quickly because of all the tools that you say. So in that sense, it’s very easy and cheap to get a whole load of lies or, or things that are biassed to fit in with your preconceptions.

Erika Hall
Yeah, and there’s no way to tell that you’re making a bad survey. If you’re having an interview with somebody, a research interview, you can tell if like, oh, this person doesn’t really match our target user. Oh, I think they’re making things up. Or they’re not really giving me what I need. Like you can you can tell if an interview is is going badly or if you’re not getting or if you’re doing some other technique, you’re reading other people’s studies, you’re observing people, whatever you can really tell right away, I’m not getting what I need. But with a survey, people will answer your questions. Like, even if you’re asking the wrong questions in the wrong way, and they totally don’t match anything in the real world, you’ll get answers back. And there won’t be anything in that data that will tell you that it’s a bad survey, like, like, if you write bad code, you’ll have bugs, you know, if you if you write some bad copy, you can have somebody read it and say, oh, I don’t understand what you’re trying to say here. But there’s nothing there’s no feedback mechanism in surveys to tell you that they’re bad or or not helpful or not reflective of reality. You only get proven wrong by you know, launching a product based on a bad survey or you do some other kind of research that contradicts it. So they’re very dangerous like surveys are so dangerous if you’re using them to make business decisions or investments based on.

James Royal-Lawson
I sometimes like to think of A/B testing as a surveys in a very similar way that your, your, the uses a proxy for research and actually, you’re just throwing them out there two variations as if it was a survey question and getting a sample back that says which one they they said yesterday,

Erika Hall
Right. Yeah, and it’s, and the other thing about A/B testing is you really have to make sure that you have statistical significance. And, and so anytime you’re doing anything like that, you’re really in the realm of some advanced math, like just because you’re, you have, you know, quantitative information, that doesn’t mean that you have statistical significance. And I think, again, that the tools that make this so easy make it look like you got a clear result when when you really didn’t. And so you really have to know what you’re doing with those things and know why you’re doing them. But a lot of times, you know, these tools end up getting used because the managers want a quantitative answer, or there’s discomfort with talking directly to people. I still hear this from so many people that, like in my business, I’m just I’m not allowed to talk to customers, or even people who are like customers, and they would just prefer it if we never talked to people. And how can you, how can you design something for people that you’re not willing to sit down with or even have a phone conversation with, and you expect those people to give you their money and time and attention, but you’re not willing to put yourself out there? That doesn’t work.

James Royal-Lawson
So both surveys and A/B testing are both ways that you can build a wall to prevent you from having that scary close contact with your customers.

Erika Hall
Exactly. And they with that sort of data, it’s much easier to manipulate it to support the thing you already want to do. Like if you talk to 20 people out in the world and those 20 people tell you that they have no interest in your product. Like there’s nothing you can do with that. But you can take survey data or or you can do some some sort of quantitative testing and you can make it seem like it supports the thing you want to build a lot easier.

Per Axbom
Yeah. And they’re not just easy to use all these tools, they give you all these pretty graphs with colours, but they couldn’t they couldn’t possibly lie lie to you because I mean, they’re pretty and they look so real. And it’s like you have these numbers and this bar chart is bigger than that one and so it’s it’s obviously correct.

James Royal-Lawson
Don’t you love the irony? These, these really highly polished UX, you know, the UX experience of these tools destroys their ability to give us insights into our UX work. It’s it’s wonderfully ironic.

Erika Hall
It really is. This is why I recommend everyone read Thinking Fast and Slow. I know it’s a pretty popular book, but it’s it’s by Daniel Kahneman and it’s about cognitive biases. And he talks about cognitive ease. And so your brain, our brains are always trying to trick us. And if you see something that’s well laid out, your brain will be telling you like, oh, just believe it. You know, it’s it’s very clean. It’s very pretty. Of course, it must be true. Like that is a really rough bias to overcome. Like, if if you see something like that, yeah, it’s just like you said, like, obviously, it’s such a beautiful graph. It must be true. That is your brain lying to you. And you it takes so much discipline, and so much, you know, people working together to fight their own brain. When they’re trying to learn real things about the world,

Per Axbom
I’m going to be buying the the second edition just based on that chapter because that’s something I’m gonna be throwing in my clients faces. Sometimes when they tell me, well, we’ve done this survey, so we have all the research, you can work on that and I’ll be going no, you really, really haven’t done any research.

Erika Hall
Fantastic. It’s it’s designed that I wrote the chapter for just that purpose, so that you could show it and say, look, read this, it’s, it’s only like 20 pages. There’s a whole story about centaurs in the middle of it to try to make it more entertaining, to really help people understand the consequences of, of using these tools that seem so easy.

Per Axbom
Thank you so much for teaching us.

James Royal-Lawson
Thank you very, very much.

Erika Hall
Thank you.

James Royal-Lawson
One of the points that we brought up during the interview was about organisation research, one of the chapters in the in the book, and it’s a fascinating topic and, I mean, we’ve, we talked about how a lot of the design work we do is about communication, if not all the work we do is actually about communication and making sure someone else understands your design or what you’re producing so they can produce it or and make it something that the user understands and so on. It’s all about communication through various mediums, and a part of our work that really is overlooked so often is understanding our organisations we work with, as Erika points out. But which made me you know, when reflecting on it now, it made me think back to two of our conversations reasonably recently. One of them of course, with with Kim Goodwin, when we were talking about decision systems, and in that interview, Kim basically gives us lots of things to think about regarding how we we make decisions in organisations or how the culture inside organisations impacts what we design and what we should be designing.

Per Axbom
Right. Exactly. Yes. And I mean, this is the point that Erika also makes with the with the research you you need to understand the business to be able to do good research.

James Royal-Lawson
Yeah. Yeah. Another another interview reflected back to when she was answering was our one about design thinking with Jeanne Liedtka. Oh, yeah. Erika actually points out that some of the problems around design thinking is its lack of understanding or appreciation of the complexities of the organisation itself. And I actually think Jeanne did a pretty good job of, of tying into the organisation better than I’ve, I’ve heard and I’ve seen and some of the things I’ve, I’ve read.

Per Axbom
And it’s true, because I mean, people also of course, debate what goes into design thinking but essentially, if you have a model where you have to find information, for sense making, then you can argue about of course what what information do you need to make sense of and the organisational business model and culture and everything needs to be a part of that.

James Royal-Lawson
Yeah. Again, if you don’t know what your business exists for, then everything else is irrelevant.

Per Axbom
So I really liked how Erika really went through the whole process of all the obstacles in research and sort of how to overcome them. Both your own biases, how to reach a level of confidence that you feel comfortable with. But also just acknowledging how everybody lies.

James Royal-Lawson
Yes, shattering some of the myths actually around design and research.

Per Axbom
As well exactly, yes. It really goes to show how complex it is though. And that’s something that a lot of people don’t appreciate how complex research can be, and how professional you have to be in not only doing the research but also in communicating the research to stakeholders and leadership. Because it can show things that people don’t want to hear.

James Royal-Lawson
I mean, I suppose we got like, you know, any, anyone can design. But at the same time anyone could create a bad design. Anyone can do research, but at the same time anyone can do bad research. And you need to have that self awareness of what you’re doing in order to to make it better.

Per Axbom
Yeah. And that point about people lying. I mean, sometimes people don’t even know that they’re lying. I have experienced from working with a bank and I was sitting at a branch an office actually with the, with the staff who actually taking on clients and I had previously done a survey asking them how much they use the internet to support them when they’re doing the talking with their clients, and they estimated something like four to five hours per day. And I did the observation studies sitting with them for a full day, and it was more like 30 minutes. So the experience of using something, of course can be hugely different from the actual reality Which means that asking people about these things just can give just such misleading results.

James Royal-Lawson
And the thing is that our brains are actually programmed to lie. We hit gaps and gaps in our knowledge and gaps in our stories, I guess. So if you if you’re faced with a direct question, then your brain is going to produce something and you’re going to spit it out. Even if you you might not consider a lie, but it’s not true.

Per Axbom
So we’re programmed to lie but we’re also programmed to misunderstand because I loved the point that you made about all that data that we’re we’re making it beautiful individual, and that actually makes us more it’s more likely that we misinterpret it or actually believe the data that is lying to us.

James Royal-Lawson
That particular thing fascinates me with how, you know we UX the hell out of some of these these tools to help us do UX work and end up tricking ourselves. It’s just it’s just a fantastic you know, complete loop of nonsense. But important, but the the critical thinking or the self reflection on the work you’re doing. I mean, the tools you’re using, you know, what, what are these tools actually doing for me? What are they actually saying? Do I understand the methods that are being incorporated and being used, deployed to present this data to me, and a lot of time, I don’t think we do,

Per Axbom
Which, which I that also makes me appreciate the point that she made about people coming from academia into this environment where they actually come from an environment where they can control their experiments more, but now they’re blindsided by the political environments, which is just uncontrollable and they don’t expect it in many ways.

James Royal-Lawson
And unexpected from many of them. Because the politics organisation is different to the the politics of research.

Per Axbom
Yeah. So just a reminder that if you want to see Erika speak in Stockholm alongside other industry voices such as Melissa Perri, Indi Young, Brad Frost and Margot Bloomstein. Use the discount code uxpodcast for 10% off your ticket purchase on frombusinesstobuttons.com.

James Royal-Lawson
And if you do that, at the same time, you’ll be helping support UX Podcast. And while we’re at it, something else to listen to. And we’ve already mentioned two episodes during the show that I think you should listen to. There’ll be in the show notes, but on top of that, why not go back to Episode 205. It’s a link show where I and Jonas Söderström discuss two articles that Erika had written about research. We we chat about that but also evil geniuses. Are we all evil geniuses? You should listen to that too Per because you weren’t in it.

Per Axbom
Oh really? Remember to keep moving.

James Royal-Lawson
See you on the other side.

[Music]

James Royal-Lawson
I was I was reading a book about helium.

Per Axbom
Okay.

James Royal-Lawson
I couldn’t put it down.

James Royal-Lawson
If If people only knew how much time we put into these rubbish jokes

Per Axbom
More than the rest of it.

James Royal-Lawson
Absolutely.

This is a transcript of a conversation between James Royal-Lawson, Per Axbom and Erika Hall recorded in December 2019 and published as Episode 229 of UX Podcast.