Adrienne: Hello, and welcome to Prod Pod.  We’re a part of This is HCD.  We’re bringing all human-centred practitioners real and honest conversations about product management.  My name is Adrienne Tan and I’m the co-founder of Brain Mates.  In this episode, I caught up with Nick Coster, a veteran product management practitioner and Gerry Scullion.  

We’re going to be talking about the dreaded NPS.  More importantly, what’s good about it, if any, as Gerry says, and what’s not good about it?  We also give practical advice for practitioners who are working in organisations who might be using it and working in teams who are so focused on it.  It’s a fun episode, so let’s just jump into the conversation.  

Gerry: Yes, my experience with NPS, it goes back probably about ten years.  I started to see it a lot more when I was working in organisations, especially in Australia.  I’m not a major fan.  I’ve never been a major fan of it, so I guess I’d be coming from the side of saying why we shouldn’t use it.  I don’t know what Nick’s perspective is, but I’ll pass it over to Nick and hear his thoughts on NPS.  

Nick: Sure.  Well, I’ve seen it being used a lot in the past.  Actually, for quite a long time, I thought, what the hell is NPS?  I didn’t actually know what it stood for, so it was being quite a mysterious metric that everyone was using.  It seemed to have, if the net promoter score when up, so that’s what it stands for, net promoter score.  If the net promoter score went up, then that seemed to mean good things for the business.  I could never quite understand how that score actually connected to positive revenue or positive growth for the organisation.  There didn’t seem to be a direct connection there.  I’ve always questioned or queried it in the past.  

Gerry: Yes.  I know from my perspective, if I was doing research, especially in the early days, and probably as UX and service design is maturing in Australia in 2011/2012, I would come back with some research.  The NPS score would contradict what the research was stating.  They were like, no, but the NPS has gone up.  I’m like, but I’ve got all of these problems here that haven’t been addressed.  

That’s where the relationship between me and NPS starts to… I’d like to say suffer but deteriorate is probably a better way of describing it.  That’s still there.  If they’re using NPS, it seems to have this trait of disrespect regards quality research and actual real customer problems and actual real insights.  Stuff that they need to fix right away.  

Adrienne: What about you, Nick, have you seen any correlation between any performance of great business outcomes versus high NPS scores, or vice versa?  

Nick: I think I do most of the training, so I’m in the classroom training people most of the time.  The experience I had is, a lot of people when we talk about metrics to measure the business, I come up with NPS as being on the great things to use to measure the business.  Then I’ll ask, well, how does that turn into money?  

Any metric, ultimately, has to have some influence on the revenue or the cost of the business to represent the sustainability of the business.  That’s usually where the conversation stops.  NPS is seen as an end point, not the starting point for a conversation around how things turn into either more profitability, more growth, more revenue, or something that actually represents the financial transition for the business.  

Adrienne: Do you think we’re being too harsh about NPS?

Gerry: No.  Straight away, no.  It’s the one thing that I guess I’m a little bit impatient.  If I have the same conversation twice in a day, I’m like, okay, I can get away with that, but I’ve been having this conversation for it seems like years.  I totally respect organisations that are using it.  They’re seeing some results from using it, but I think there are better ways of doing it.  That’s the role of service design and UX, project management, whatever you want to call it, there are better ways of doing this.   There are better ways to deliver value for the customer.  NPS, for me, just doesn’t do it.  It’s just too vague.  

Nick: I think NPS even when it’s being used to its best intent, which is to try and get some gauge of customer satisfaction, such that they would in an ideal world refer to another person or potential customer, tell a story, hey, that was an amazing experience.  You should use that too.  Practice that never happens.  Even if it’s used in that intent, it’s still a blunt instrument.  It’s not a fine-grained research tool to find out what’s actually happening.  There’s very little granularity to it.  There’s no insight, there’s only a numerical and a fairly simple numerical measure of this arbitrary customer satisfaction.  

Gerry: I mean, if it was an instrument, like if it was a tool, I don’t even think it would be a mallet, I think it would be a foam mallet, if that.  It wouldn’t hit anything and give you any damage, it wouldn’t give you any impact.  

Adrienne: Look, I’m not a fan of NPS either because I think that people put too much weight on NPS and therefore, at the detriment of other probably better measures of customer delight, customer satisfaction, and success of the business, but don’t you think that businesses have something to focus on, as opposed to absolutely no numbers whatsoever?  

Gerry: Something is better than nothing.  

Adrienne: Something is better than nothing.  

Nick: Like I said, when it’s being used well, it can be a good dashboard indicator to say: Something is right, or something is wrong.  Or something is generally improving, and other things are not improving.  That’s if it’s being used well.  If it’s not being used well, it’s just a vanity indicator and people can game it and used the language around the question of: Would you refer this to somebody else?  By the way, if you do, you might be in the draw for $100 voucher.  The way it’s framed can be terribly bastardised.  It becomes meaningless.  It becomes a vanity metric that people are abusing rather than using effectively.  

Adrienne: Have you seen it used well in any business?  

Gerry: I have seen a great presentation by Harriot Welkom.  I don’t know if she’s still at Medi Bank, but it was a couple of years ago I saw at a conference in Australia, where she was able to extract value from the NPS scores, but from recollection, it was down to an open text field, perhaps that was extracting some qualitative value, which in itself is not NPS in my world.  

The value or the bit that people tend to focus on is that rating scale of zero to ten or one to ten, I’m never too sure, but that’s the only one I said, do you know what?  There’s something good coming out of that, and that’s fine if you’re getting something that’s qualitative out of the process of using NPS.  Using the scale on its own, I just don’t see any value.  It’s made up.  

Adrienne: Yes, it’s almost like a signal to do further research.  

Nick: It is a signal.  

Adrienne: Yes, but you’ve got to take it as a signal.  You can’t just leave it.  

Nick: Its simplicity is, that it’s very easy to put into a survey.  It’s very easy to calculate.  There’s no deep science to the mathematics of it.  It’s very easy to stick into a survey, to put onto a checkbox form, and then ask someone to score from zero to ten what their experience was.  Then get a number, which you can then aggregate across a large volume and see an overall trend for a part of the process or for the overall product experience or company experience at large.  

Gerry: Yes.  

Nick: I think a lot of those numbers get blurred together, and so, is it the particular experience they just had that they’re ranking, is it the perception of the overall product or the company or the brand that they’re associated with?  I think all of those things get blurred together.  In some respects, they should because that’s part of the customer experience.  As a measurement tool, it becomes very difficult to work out what are you actually measuring when you do that part of the survey process?  

Gerry: Yes, I’ve tried to get people onto the This is HCD podcast one.  I was running the show on my own.  That sort of podcast, as opposed to the project management specific one.  I found it very difficult to get people to come on to talk about NPS.  I had a lot of people saying, “That would be really interesting.  What do you want to talk about?” I said, I want to talk about how bad it is.  

They would shirk away from the conversation a little bit.  I think the reason why businesses are attached to it, is because it’s a simple number.  They said it’s simple.  My response to that is, it’s not simple.  So, they’re looking for the easy way out, but using NPS, to really value customer experience and service design and UX, it’s not an easy job.  

You need dedicated people to be doing great research, to be bringing it back, to surface it into insight, and actionable insights at that, to be able to deliver value and solve those problems with customers.  NPS just does not do that.  There are no ways that you can change the conversation that it will happen.  As Nick said, it’s a blunt tool.  

Adrienne: It’s a signal, it’s a signal for you to do further research.  It’s a signal for you, but potentially, it might not be, it might be a great NPS score, yet, nothing positive is happening in the business.  

Nick: Yes, and a lot of the interactions you have, if things work well, then your satisfaction doesn’t really change, yes, it worked.  Like, if I go to the bathroom and I flush the toilet, I don’t give it an NPS score because it actually worked.  It just did what it had to do.   Most organisations seem to be throwing in NPS surveys at the most mundane points of their service design or service interaction.  To say, look, how was that for you?  Well, it did what it was meant to do.  

I’m not delighter, so the resulting NPS score is either they don’t leave one, or they don’t state to the survey, yes, I would absolutely recommend this to my friends and family because that was the most mundane experience I expected.  It doesn’t play out that way.  

Gerry: Yes, but products and services are not linear.  Okay, they’re radial and they’re totally fluid.  By placing NPS as sporadic locations, you’re not getting a read on anything of value.  For instance, if it was a bank and you’re calling up about your, I don’t know, you’re trying to do a direct debit with someone.  They gave me an NPS score, yet, my card hasn’t arrived in the post, and I’ve been waiting two weeks for it.  There are two different silos in the bank there, and the score is not going to be reflective on the overall experience, it’s going to be reflective on being able to setup that direct debit.  

Nick: I had an interesting experience some years ago at a car hire company.  As I turned up to the car hire to drop the car off, as I was stepping out of the car, they ask me, “So, how was your experience with the car?” They asked me to essentially rate it verbally and gave me an NPS score at the time.  It was fine, the car worked.  It was great.  Would you recommend this brand?  Sure.  Then walked into the office to hand over the keys and do all the paperwork, and everything went pear-shaped.  

They found a tiny ding on the car.  Then I had a terrible experience that wasn’t captured in the NPS score.  They very deliberately picked the point when people were at the happiest moment of the experience, which is, I arrived safely, it’s all good, to pick the point for the survey.  Not the point after which they may have incurred additional charges, where they be at their most miserable.  I tore up the score.  

Gerry: Yes, well, that’s the thing, like, it’s when it’s being placed will give them a false-positive.  It gives them this read that people automatically get behind.  You know, our NPS score has gone up, so what?  

Adrienne: I think the question is really, for NPS anyway, a really off question, because you’re asking humans to predict what they would do in the future.  I don’t think at that point after using a particular product or service, that you’ve really thought through, “Well, will I recommend this?” It’s such an odd question to ask people, because you don’t know.  You don’t know in the future who you’re talking to, what situation it might be.  You might be happy now, but you might not be happy in the future.  

Nick: If you think about how it plays out.  I don’t know how many NPS scores I fill in or complete once a week or once a month.  Let’s call it five a month, I’m fairly confident that I don’t rush out to my friend and say, “Do you know what?  Five of the products I want to talk to you about today.  These were amazing things.  I did this survey.  I have to talk to you about it, it’s going to be incredible.” It just doesn’t happen.  The volume of surveys that we perform, there’s absolutely no way that we actually run around promoting all of these services and products.  

Even if we scored them a ten/ten in a net promoter score.  From that perspective, it’s complete fantasy.  When we talk about this being a metric and a value metric for the business, that’s where I get really frustrated about it, because it simply doesn’t correlate to business return.  It doesn’t correlate to growth.  Customer satisfaction does because you may actually talk to customers about it to other people, but completing a survey about, would you refer it to somebody is a fanciful question that doesn’t mean anything.  

Gerry: Yes.  

Adrienne: We’ve talked about the problems about NPS, but what are some of the other alternative measurements?  How is it that we can measure the success of the services that we provide to our customers?  The products that we sell them and how satisfied our customers are at using these products and services?  

Nick: I think that when I’ve challenged people on net promoter score, where I’ve heard about it being used well is when they use it in a much more targeted way from different parts of a customer flow.  Now, the question I think is still wrong.  I think a better measure is to say, for this particular interaction, how satisfied were you with this thing you just did when you completed the task?  When you completed the outcome?  When you tried to do this particular thing?  Make it laser-focused on that outcome.  Not try to pretend that it actually turns into a customer referral at some point in the future.  That, to me, is the logic break in the system.  

Adrienne: I completely agree.  

Gerry: If there’s a sequential flow.  If you’re doing onboarding all the way on to exit, in terms of a customer journey and it’s been NPS at each stage, then you can see visibly that there’s a drop over time in a certain part of the journey?  Is that what you’re saying?  

Nick: Absolutely.  

Gerry: I guess there’s something in that.  

Adrienne: Yes, but he’s also saying that the question that’s posed in an NPS methodology is fraught with difficulties and potentially, we need to look at framing the question differently.  Asking the person, and it has to be a lagging indicator instead of a leading indicator, right, because it’s how satisfied are you with what you’ve just completed on our platform, on our service, on our product, as opposed to a leading indicator, which is, will you recommend this to somebody else?  Well, I don’t know, it’s going to be zero, because I might be satisfied but I might not necessarily recommend that.  

Nick: I think what you’re saying there talks to the fallacy of the net promoter score.  Its intent is to be a predictor of growth, which is, if my net promoter score is going up, then theoretically, my referrals are going up, my word-of-mouth and sales will increase.  

Adrienne: People love me.  

Nick: Therefore, it will be a driver of my business.  

Adrienne: Yes.  

Nick: In fact, if that assumption is actually not true, and can’t be validated, then all it is, is a customer satisfaction and lagging indicator, at best.  I think it’s better to call it out and say, for this particular registration flow, or for the call you were just on, for the operator you were just with, or for the action you just performed, how satisfied were you that you were able to complete your task, or that you had a good experience?  You can choose what sort of framing you need to for that.  I think they’re more useful questions.  They’re still numerical ones, so you don’t get the insight, you just get an indicator.  They’re still more targeted to the recent interaction the person is involved in.  

Adrienne: The target is useful because then you get your service designers and your UX designers, or experience designers to go out and understand more of a problem if the satisfaction level was lower than your intention.  

Gerry: Yes, maybe.  I just feel that any of the businesses that I’ve been in tend to lack the maturity to have good quality researchers to go out and do that work.  Also, for the business, to champion it.  There’s usually a disconnect between the c-level or the senior stakeholders and that qualitative research.  They seem to be a lot more attached and attracted to that NPS score.  That, in itself, is where the biggest problem is.  

It’s the disrespect for qualitative research and it’s one of the indicators for me when I’m trying to evaluate an organisation’s design maturity.  Where do they put the emphasis on the qualitative research and how are they going to be able to act on that?  Too often, they’re attached to that NPS score.  As we were saying with Nick, it’s because of the business KPIs that are attached.  

Adrienne: I think it’s the personal KPIs.  

Gerry: Personal KPIs to an NPS increase.  

Nick: I think that’s a really important point because I think the rise of NPS has been driven in many cases by the fact that a person’s professional KPI has been bound to this number.  

Adrienne: Yes.  

Nick: As soon as someone says, “Well, my KPI is…” well, then I have to move my net promoter score for my product from a 50 to a 60 or from a zero to a 3, or whatever it might be, or from a negative to a positive.  Depending on how much in the hole they are.  Then rather than actually focusing on customer experience, it’s how do I get someone to fill in the survey properly and give it a nine or a ten.  

Gerry: Yes, so gaming it.  

Nick: It totally games it.  Whole teams can be focused on trying to effectively deliberately or inadvertently gaming that number, but it doesn’t really reflect improvements in the product.  

Adrienne: You can see why people use it, you can see why people have it because it’s so hard to put proper measurement and KPIs in place for individuals and for businesses.  It’s extremely difficult.  How do you measure a good product manager?  How do you measure a good UX person?  How do you measure a good service designer?  Those are really difficult questions to answer.  

Gerry: Yes, and it’s a difficult problem to solve.  

Adrienne: Yes.  That whole NPS seems to be so simple.  The calculation like you say is simple, the premise is simple, and so, if they don’t have that, what is it that they should use and how do we calculate that?  

Gerry: If it seems to easy…

Adrienne: It is too easy.  

Gerry: It is too easy.  

Adrienne: Yes.  

Gerry: It’s like a sugary donut.  I’m looking at Adrienne because she’s brought sugary donuts today.  If you eat that every single day, you become fat and unhealthy and you will die.  That is what NPS is.  

Adrienne: Thanks, Gerry Scullion.  

Gerry: I think I might have nicked a metaphor from Jarred Spool there, but I don’t mind.  I’m sure he doesn’t mind.  That’s, to me, if it seems too easy, it is too easy.  

Nick: It is, it’s like a sugar pill, it’s not real medicine.  It’s a placebo metric.  That even if it’s used well, it’s actually giving you incorrect information.  Look, to be honest, if you see a net promoter score move from a low number to a high number, that’s not necessarily a bad thing, but it doesn’t really give you any positive insights and it doesn’t necessarily mean that the customer is satisfied.  

Adrienne: But if you see an NPS go from low to high and yet, you’re most important indicators for the health of a business, which in most cases of profitable businesses are around your revenue, your net profitability, if those don’t move, it doesn’t actually mean anything.  

Gerry: Yes.  

Adrienne: We’ve talked a lot about some of our actual experiences about NPS.  As practitioners, you go into businesses all the time, let’s say you went into a business that was super focused on using NPS and measuring recommendations essentially.  What will you do as a product manager, Nick Coster, and what will you do as a service designer, Gerry Scullion?

Nick: Well, the first thing I’d want to find out from the current use of NPS is, at what point are they trying to measure it and after what experience?   That sets the tone for what they think they’re trying to measure.  In some respects, I start to work my way backwards from that.  I then say, if it’s simply at the end point of any customer experience and they’re all being aggregated together, I’d probably cry quieting into my team because it’s total garbage.  It’s aggregates of aggregates of garbage.  

Then I’d say, well, let’s see which touch points we’re actually trying to measure and find out if there’s a better question we can ask at the end of those experiences that’s not the NPS question.  Like, how easy or difficult did you find this task to be?  Did you complete this task in a reasonable timeframe?  What’s the actual measure of success that we’re trying to drive at that particular interaction point?  Are we achieving the goal with that particular customer?  

Gerry: Yes.  

Adrienne: What about you, Gerry?  

Gerry: I mean, I’ve worked in lots of them, like banks, they love it.  Big organisations tend to love it, as well, because it makes the complex simple.  I would never turn around to them and say to them that I think they’re making a big mistake, but I’ll definitely help them come to the water a little bit and hopefully let them drink.  I try to understand what they’re doing with it and what insights are coming off the back of that NPS score.  I’d be asking the question of, all right, great, so the score went up in the last quarter, what did you do with that?  

How has that solved problems for the customers that have come out of the qualitative research, that hopefully the UX team have still got high spirits and they’re still going out there and they’re fighting the war, and they’re still bringing those insights back to the business?  I would be trying to identify the disparity between the two, because there will be, depending on what Nick said, where it’s been applied, where that NPS is being applied, there will be a disparity between the two worlds.  

Helping the stakeholders understand that there are problems out there, that they’re not getting solved, and those problems have been there for a period of time.  Now, it’s something that I’ve spoken about a little bit before, where I’ve got this methodology called the wall of pain.  I did it, I know, Adrienne, you’ve seen it at some of the organisations I’ve been in.  I put a dollar value in each one of those problems.  

I help the stakeholders see that there are these qualitative pieces that have come back into the business that have remained there for a period of time.  There’s a dollar value on them.  It’s very interesting to see business people change the conversation and move away from an NPS thinking into more of a qualitative thinking and taking those and acting on them.  

Adrienne: In a financial perspective, as well.  

Gerry: Absolutely.  

Adrienne: I think if you associate a dollar value to a pain, a customer pain, that makes it more meaningful, really.  I was going to ask, what if businesses simply just collect NPS scores and, yes, try and move them towards a more positive side, but don’t link it to anything more meaningful, anything more financially relevant.  How would you get them to do that?  

Nick: Well, I think as we were talking about this, I was thinking through one of the challenges here, because the opposite of NPS is just focusing on the money.  If we focus on the money on its own, again, then we lose sight of the customer and the customer experience, which I think we’re all in agreement.  That if you’re having a really positive customer experience, if you’re solving the customer’s problem, the revenue will follow, providing you haven’t spent too much money building this experience in the first place.  If the metric becomes purely financial, and it’s just about making money, then you can lose sight of the customer experience.  Again, you have these negative behaviours.  Again, the intent of net promoter score is to try and track customer behaviour, which is a good thing.  It’s just the wrong tool.  I think the link, I like the idea of the money wall, the pain wall.  

Gerry: The wall of pain, yes.  

Nick: Yes, where there is a deliberate connection between customer satisfaction or customer challenge, or the customer problem, and the cost or the opportunity financially that represents.  I haven’t seen anybody make an effort to correlate a net promoter score with a dollar figure.  I think that, to me, is another one of the missing links.  One of the things I would champion for is, if you’re going to say, it’s really important for us to get from a net promoter score of, again, 20 – 30, then there should be a discussion of, well, how much is that worth to you?  

Is it worth another million dollars to the business?  How much money will you spend to get from 10 – 20.  If you’re going to spend 20 million dollars to make a million dollars, it’s probably not that big of a deal.  Do it the way it is.  On the other hand, if you’re actually focusing on the real customer pain points, you can spend a little and make a lot because you’ve got a lot clearer focus on where the pain really is.  

Gerry: Yes.  

Adrienne: Any final last words on NPS for today?  I think it’s a topic that we can continue to explore in future episodes.  Do you have any last words to say?  Should we use it?  Should we not?  

Nick: I think that in many cases it’s so entrenched that to say, no, you can’t use it, is a limiting move.  However, I think that recognising that and recognising that it’s a bit of smoke and mirrors that keeps some executives happy and placid is one thing, but then go deeper.  Ask the right questions.  Sometimes keep the net promoter score in because you need to, because people’s financial goals at in touch to it, but then ask deeper questions to find out where the real pain points are and where the real customer experience and opportunities are.  

Gerry: Yes.  As Nick said, it’s going to be there, it’s not something that we’re hoping, like with an episode it’s just going to go away, and we’re going to go, who knew?  All we had to do was a podcast.  I would say that if it’s just the NPS, you’re doing it on your own, that’s the only metric that you’re getting back.  You’ve got a bigger problem there on your hands.  Make sure you’re doing other forms of research, especially qualitative research, ethnography.  They’re the key pieces where you want to get that really, really rich insights that are actually going to help change the experience.  You’re going to make things better.  If you’re not doing that, and you’re just relying on NPS, you know, I’m sorry, but you’re going out to war with a paper sword.  That’s what I say to any of the clients that I’ve ever worked with.  

Adrienne: Thank you so much for you time today.  

Gerry: No worries, thanks for having us.  

Adrienne: I’ve enjoyed this conversation.  

Nick: My pleasure.  

Adrienne: I think just as a farewell thought, that if you’re using NPS as a vanity metric.  I think the first thing that you need to do is try and find some way to make meaning out of that score.  Try and use it as a signal to go deeper and not simply as a measure to make you team and your organisation feel better.  Thank you so much for your time today.  

Gerry: No worries, thanks so much.  Thanks for having us.  

Nick: Thank you, Adrienne.  

Adrienne: Thank you for listening to Prod Pod.  If you want to learn more about the wonderful shows on This is HCD, we’ve got others, such as the Power of Ten with Andy Polaine, bringing design closer with Gerry Scullion, and Ethno Pod with Dr.  John Curran.  Please visit: thisishcd.com where you can sign up to our newsletter, join our Slack channel, connect with other human-centred practitioners around the world.  Thank you for listening and speak to you next time.  

End of Audio