10 / Evolution of the Product Manager Role


The product manager role has been around for decades, but its contributions have been generally overlooked and misunderstood.  No longer is that the case, according to the 2019 State of Product Leadership report, prepared by Pendo + Product Collective.

In this episode, hosts Sean and Joe speak with Pendo chief marketing officer Jake Sorofman about the recent reports and the continuing evolution of the product manager role. “It’s a role on the rise,” Jake says, “but also one in a state of transition. It’s only in the last 10 years that product management has really come into focus as a very strategic part of the business.”

Read our blog post

About Jake

Jake Sorofman is CMO of Pendo, a Raleigh, NC SaaS company that provides insights, guidance, and communication for digital product teams. Before Pendo, Jake was VP and chief of research at Gartner, Inc., where he focused on CMO topics and marketing trends. Prior to that, he spent 16 years in marketing leadership roles with venture-backed software companies.

l t 


Joe: [00:00:00] OK so Sean, we’re here today with Jake Sorofman, and he’s the Chief Marketing Officer over at Pendo and that is a tool we are hearing a lot about in the product space of companies starting to use to better serve their users.

Sean: [00:00:13] Super excited to have you here, Jake. We met at the product conference awhile back. Thanks for joining us today.

Jake: [00:00:18] My pleasure Sean. Good to be here.

Joe: [00:00:20] So Jake if you could just start off, you know typically how any podcast starts off, by introducing yourself. Tell us a little bit more about Pando and what your role is.

Jake: [00:00:28] You bet. So my name is Jake Sofofman; I’m CMO of Pando. I joined Pando about 18 months ago and before that I spent five years as a Gartner analyst. I was V.P. and Chief of Research of digital marketing and CMO practice at Gartner. And that was a bit of a left turn in my career. I had spent previously about 16 years with venture-backed software companies, so I returned to my roots and I’m having a lot of fun doing it.

Joe: [00:00:55] Awesome. So what we wanted to talk to you about today is, you’ve put out, Pendo and you obviously steering it, have put out this State of Product Leadership survey and really talking about, you know, product management, product leadership….

Jake: [00:01:09] Yep.

Joe: [00:01:10] Just everything having to do with it. And so we wanted to talk through what we saw on the results there. But why did you feel it was important or needed to kind of put out the survey?

Jake: [00:01:20] Yeah that’s a good question. I am a content marketer by philosophy and inclination and what I mean by that is I think it’s really important to start with what’s at stake for the audience you’re trying to reach, and I really tried to authentically deliver something of value to serve that community. And the community we serve is product management. Product managers use Pendo to better understand their users and to guide their users within their products. So we’ve made a pretty significant investment in premium content in the form of editorial content in our site, productcraft.com, and e-books, and primary research reports like this as a way to deliver something back to the community we serve. And that’s really what the spirit is behind State of Product Leadership. This is the second year we’ve run the survey and we’re sampling about 300 product managers in North America. And it’s always just very interesting. It tends to feed us all year long as we look to tell stories around sort of the data and the findings and the trends that we see in the product management community. You have to remember for Product Management as a role, while it’s been around for a long time, it’s often been really misunderstood and sort of neglected in some ways. And it’s only in the last, say 10 years, that product management has really come into focus as a very strategic part of the business. Of course this is arguable but this is my observation where companies are seeing it as a key lever of growth. And this is our way of trying to better dissect that role and help product managers themselves to more effectively grow their careers and execute the functions that they’re responsible for.

Joe: [00:02:59] Very cool. So I looked through the 2018 edition, I looked through the 2019 edition, and I have some questions for you here about some of the specific aspects of what changed over time and what was consistent, but just for you personally, what was the biggest surprise you noticed year over year?

Jake: [00:03:15] So last year I’d say that the narrative was of a role on the rise. Product Management is sort of fast becoming one of the new “it” roles. The way I like to frame it is, newly minted MBAs used to beat a path to Wall Street or hedge funds or management consulting, and now they want to be product managers at SAS companies in Silicon Valley. That feels new and different. So there’s a lot of heat and light being drawn to this discipline and a lot of demand and interest in product management. So we saw a lot of signs that this was a role that’s taking on more strategic importance, a role that is clearly in demand. That continues in the 2019 edition, but we also see a role that may be somewhat in transition as product managers sort of reconcile the fact that they’ve been given a lot of responsibility, but in many cases, they simply don’t have a ton of authority. It’s a discipline where you lead through influence not formal authority and that can lead to some degree of frustration. And we saw some of that frustration in the findings in the survey.

Joe: [00:04:24] Got it. That’s interesting comment about Wall Street and kind of those types of jobs, wanting to go towards product management. You know, something I hear a lot, something that my team asks me and our company in general and I talk to a lot about with other product managers is, “do I need to be technical? Like how technical do I need to be? Do I need to know how to write code, do I need to know how the server architecture works, like where’s the line?” What did the survey show in terms of product managers needing or wanting to be technical or not.

Jake: [00:04:50] So last year… This is always a very controversial question. It’s fairly polarizing. People take very strong positions for and against this. But last year we found that the majority of product managers were coming from a non-technical background. By and large they were coming from, you know, in undergraduate or graduate program studying business, they were coming out of marketing as their most recent last job, and this year we found that more often than not product managers have a technical degree of some sort. They have studied something more technical, either computer science, engineering, or the hard sciences; and less so business, liberal arts, et cetera. But we also find that their last job was was marketing, so that held true year over year, but we are seeing a bit more respondents tell us that they have more of a technical background. Now whether a product manager needs to be technical, if you ask me, I think it really depends. I think they need to be technical enough to have credibility with an engineering team, their core constituency. If they don’t have credibility with engineering they’re going to have a really tough time doing their job effectively. But whether they need to be technical beyond that I think depends on how technical the product is and how technical the buyer is. To the extent that it’s a technical product sold to a technical buyer, I’d argue that there is a stronger case for them to be truly technical themselves. They need to understand the domain.

Sean: [00:06:17] Cool. I was super interested to see, I think it was in 2018, this changed: in 2018 only 1 percent of product managers had a design in a creative background, but in 2019, it looked like there are a bunch more coming from UX and design. I wasn’t sure what to make of that. What do you think about that? Why do you think that occurred?

Jake: [00:06:34] Yeah. Go figure. It’s an interesting one for sure. The best I can make of that, is that, and we see this at Pendo ourselves, we have a good sized and very progressive product management team, many of whom came from UX backgrounds and they’ve moved from design into sort of bonafide product management. But it makes some sense if you think about it because design is so so critical to great products these days. Innovation is moving up the stack, like every layer gets commoditized from infrastructure to platform to the functionality itself. And now the innovation is often found in how that functionality is rendered and how it’s made available to users in a way that’s simple and inviting and delightful and intuitive. I think of it as almost like the Apple Effect. It’s like great product design from companies like Apple, or frankly Uber or Lyft or Amazon or any great companies, consumer companies, that have really figured out how to deliver simple and intuitive user experiences, are sort of putting the onus on the rest of us to do that much better. Because our users are arriving to our product with different expectations and these are expectations that are shaped not by their next best alternative, which is to say, other available solutions to that problem, but by their last best experience, and often those experiences are happening in their consumer life. So that’s a very long way of saying that I think UX and design are just absolutely critical to how we think about great products today.

Sean: [00:08:06] Sure. No doubt that’s rapidly changing in the market. The expectations of consumers are definitely getting more and more demanding around good design, right.

Jake: [00:08:15] Absolutely.

Sean: [00:08:16] The other thing I saw in 2018 was that most companies don’t have a dedicated product division or Chief Product Officer.

Jake: [00:08:24] Yeah. One of the one of the interesting findings over last year was that Chief Product Officers are on the rise. Last year I believe something like 6 or 7 percent of companies reported having a CPO or equivalent to sort of the senior most product leader reporting up to the CEO within their organization. And this year it was about three times that. And it seems like, you know, traditionally marketing has been the reporting line for product management. And still it is the dominant reporting line for product management. But increasingly we see product gaining a seat at the table and represented by an executive in the C Suite, which I think makes a lot of sense and frankly we see it as sort of the marker of a mature product organization is having that seat at the table and that sort of C-level equivalency within the executive team.

Joe: [00:09:17] So to that end, with the CPO reporting directly up into the CEO, do you think that having still such a strong presence within marketing is just because of tradition of, “hey we have digital products, we have software products, they should go to marketing.” Do you think that’s what’s causing that?

Jake: [00:09:32] Well I think marketing in many organizations owns the voice of the customer and really has a deep understanding of the markets they serve and the need states they’re solving for. So I do think that there’s some wisdom in having product management aligned to marketing if not reporting to marketing. Yeah, it makes some sense. I think it’s the more progressive variant than having product management roll up to engineering. Ultimately Product Management is an iterative and exploratory process of solving for problems over time and that requires deep insight into customer need and deep insight into markets. You don’t know the answer until you know the answer, and getting closer to that insight is what’s so critical. I think the reason that having a CPO is better still is because it gives product more influence and strategic decision-making within the company. And many companies are recognizing now that product is kind of the new battlefield. Like you can’t hide behind a false brand promise. You need to deliver an exceptional product and an exceptional product experience and without that you’re quickly discovered and stamped out. So getting it right really really matters.

Joe: [00:10:41] I mean, Sean, we talk about kind of the fourth dimension of competition. You know, you’ve got to be fast, you’ve got to have quality…

Sean: [00:10:49] When I was in business school, they taught you that there’s this thing called the trade off triangle, right. Quality, speed, and price you got to choose two, it can’t be all three.

Jake: [00:10:55] Right. Exactly.

Sean: [00:10:56] The reality is the internet and product and consumer expectations because of the Internet has changed all of that. Like if you’re not high quality everyone your customers has a bullhorn called the Internet, right.

Jake: [00:11:08] That’s exactly right.

Sean: [00:11:09] There’s nowhere to hide. So it’s like almost perfect transparency. We’ve all got these things in our pockets called smartphones and we can find solutions in a heartbeat. So those are off the table. You have to be high quality. You have to be fast. Are we all racing to zero as some would argue? You know Starbucks has changed the market and they charge five dollars for a cup of coffee. That was unheard of 10, 15 years ago, right.

Jake: [00:11:29] Right. Right.

Sean: [00:11:30] They’re not selling you a cup of coffee, they’re selling you an experience. And we all have to come to grips with that fact that we’re a product of the experience we produce. That’s it.

Jake: [00:11:38] Couldn’t agree more. Well said.

Sean: [00:11:40] How everything else adds up to that: the quality of our products and services the speed that we deliver them and the experience we provide, that’s basically how we are able to come to a fair price in the market. My opinion.

Jake: [00:11:52] Yeah. And I think it’s part of that evolution up the stack where the battlefield has shifted, sort of progressed up the stack and it’s progressed from, you know, features to the experience that you wrap around those features both in the design of the product that you’re delivering as well as the value added experience that you provide as an organization. Sometimes that’s within your application itself. That’s one of the interesting things that we found, and one of the things frankly propelling our growth as a company is that there are aspects of sales and marketing and support and training that are moving into the application because the experience itself is increasingly digitally led and self-service. It’s not wrapped around the product through other customer touch points, but it’s inside the product and that needs to be enabled in a really elegant way that’s aware of who you are as a user and where you are in the journey and what your specific need might be.

Joe: [00:12:46] Yeah I love the way you phrased that; it’s all moving into the product, because we’re seeing that as well with our clients. They’re thinking about how to move all these cross-channel experiences just into the product. Just move it all in there.

Jake: [00:12:56] Totally. Absolutely. 100 percent. So if I had to pick one part of the survey that I found to be the most interesting, it was that in 2018, the respondents mentioned that their roadmaps were being largely driven, like largely, by competitors. And this year it seemed to swing back and it’s again more about the customers. That was very good to see, from my opinion.

Jake: [00:13:16] It felt like progress.

Joe: [00:13:18] Yeah, yeah. So what do you think was, any reasoning behind that, do you think? Or what was causing that?

Jake: [00:13:23] It’s really hard to tell. Data is going to vary year over year across samples, even though we drew a very similar sample, but there’s always going to be some noise in the data. So it’s entirely hard to tell. But I think that where it belongs is where it landed and that’s that, you know, the customer is dictating of priorities. But I will caveat that, I think that that can become, you know, if you’re too beholden to explicit customer feedback, you can become somewhat whipsawed and sort of wagged by customers who aren’t necessarily thinking about the problem as deeply as you might be or may not have the objectivity to think about it as clearly as perhaps you can or perhaps a whole constellation of customers can.

Sean: [00:14:05] Or your team.

Jake: [00:14:06] Or your team, exactly.

Sean: [00:14:08] I was super excited to see, in 2018, you didn’t even ask. Like it wasn’t even one of the options to choose brainstorming or internal suggestions and requests.

Jake: [00:14:17] Right.

Sean: [00:14:17] And those are two options that showed up in 2019. I know in one of your articles I read you mentioned Steve Jobs’ distain for focus voice groups and customer voice as a negative thing, but I think he was onto something there, in that your customer only experiences, they have one single thread of experiences with you. and your product, right.

Jake: [00:14:35] That’s it exactly.

Sean: [00:14:37] But your people, your marketers that are out talking to lots of customers (hopefully), and your salespeople that are out talking to lots of customers (hopefully), your service people, your delivery people, your, you know… The people that are talking to lots and lots of customers every single day: they know where the skeletons are hidden and they know where your customers might find enjoy or where they might be frustrated. And if we can figure out.. We run these it’s one of things we do these one day workshops, that work through a bunch of processes that kind of suck that customer, that juice, out of their minds to come up with some ideas to really think about.. And and to create empathy for the customer right, because the more empathy you can create, the more care and concern for your customers you can create in your organization, the more powerful your organization becomes. So I was excited to see that show up in the 2019 results.

Jake: [00:15:20] I love how you put that. I am a big believer that, I think product management is as much or more about synthesis than it is analysis. And synthesis, there is a lot of art that goes into synthesis. It’s about pulling together lots of data streams and sources of feedback and inspiration and pattern matching and then adding your own subject matter expertise and your own instinct. And that’s really where great products come from. It’s not from being explicitly driven by customer feedback or explicitly driven, for that matter, by this rare visionary genius that’s represented by the Steve Jobs archetype of someone who can see around corners and better understand the customer need than they can. It’s a combination of all these things.

Sean: [00:16:07] And experimentation.

Jake: [00:16:09] For sure.

Sean: [00:16:10] Hey I got a question for you. So I went back and read a bunch of things that you’ve wrote in the past, so if I bring up something from your past articles and it embarrasses you I apologize.

Jake: [00:16:19] Oh no. No worries. All good.

Sean: [00:16:23] Just kidding. I read an article you wrote that claimed that you’re an introvert, so, funny that you’re in a CMO position and on this podcast.

Jake: [00:16:28] Yep yep. True story.

Sean: [00:16:32] And you also mentioned earlier about what makes a good product owner and a good product manager. I’d love to hear your thoughts. I’d like to give my community access to some of your thoughts on why you think introverts could be really good product managers because I thought there was a lot of great insight in that article.

Jake: [00:16:48] I appreciate it. So I’m trying to think back around my arguments I believe I argued in favor of introverts as the better product managers, and let me give this a shot. First of all, I’ll say that I’m sort of an extroverted introvert. I don’t think that every introvert looks alike and I think that the way I draw the distinction between introversion and extroversion is, where do you find energy? And my energy is found in quieter moments, more solitary moments. I’m happier probably on average reading and writing and thinking and at that sort of self reflection. It’s not to say that I don’t find energy otherwise, it’s just, it’s more of a continuum. But anyway, as it relates to the product management discipline, I think that those quiet moments, that ability to go deep and to focus and to reflect yields, at least in my personal experience, which is all I have, better insights because you give time and space to those thorny issues. You’re willing to give time and space to those thorny issues to sort them out and make sense of them and go deep and go into the dark places to better understand them. I’d also add, you know, you mentioned empathy. I think empathy is the cornerstone characteristic of a good product manager and without it you’re nowhere. Empathy is a willingness and ability and genuine aptitude or instinct to really recognize the pain and problem in your customer’s life, and even more than that, to want to solve for it. Great product managers are nearly obsessed with solving for a customer pain and wanting to make their life better in some small way.

Sean: [00:18:21] I love that. I’m a big, as Joe can tell you, I’m a big fan of empathy and empathy training and really understanding what empathy means. I’ve read a lot of stuff from Daniel Goleman. Social intelligence, emotional intelligence, I love that stuff. I think that there’s two key scales of empathy and there’s different types of empathy. One is this capacity for caring, and the two key attributes of a great product leader, I believe, are the capacity to care and the capacity to influence which is defined as emotional intelligence.

Jake: [00:18:49] Brilliant.

Sean: [00:18:50] When you tackle those two scales really, really well… And, the other part of great product leadership is not just having the capacity to care and having the capacity to influence but actually putting them in action and pulling your team up, getting your developers to care about your customers and the product, getting your QA people to care, getting your marketers to care. Right, to care about the people and the users of the product, not just the dollars and the revenue that it’s going to bring in.

Jake: [00:19:14] I love that.

Sean: [00:19:14] Just putting the right metrics in place around that is amazing.

Jake: [00:19:18] Yeah. I think to the extent that you can transfer that empathy across the organization that’s what leads to great companies. I really, really like that.

Joe: [00:19:27] All right, watch this transition. So speaking of that, the survey actually showed that product managers are becoming more and more tactically focused.

Jake: [00:19:36] Yeah.

Joe: [00:19:36] Being more in the weeds doing day to day things and they’re not really focusing on the vision much. But if you’re going to be able to communicate to your team and get them to understand the customer and how to empathize with them, you’ve got to help them with the vision. You know, “who are these users? What does success look like? What are we trying to get done here in the world?” So isn’t it a huge risk for our role in product management that product managers are spending less and less time on the vision and more on the tactical side?

Jake: [00:19:59] I don’t think so. First of all, I think product management requires a blend of both. Like most things, there isn’t an either/or. It’s and/and. But I think that division comes from the hard work of the doing and there is a lot of doing in product management. It can be a bit of a grind. There’s a lot of detail that needs to be managed. I think that the better product managers that I’ve worked with are willing to live in that detail and willing to suffer the pain of that detail, and in doing so, they create better products, they create more predictable product delivery, they create more sort of functional relationships with their engineering counterparts and the vision comes from the knowing. Like, when you’ve been able to live in that detail, because Product Management is a discipline of synthesis, it’s about pattern matching, there are insights that are revealed and the vision is sort of the sum of the parts of all that detail. I think the mistake is that when product managers look at this as something that is more like the jobs in archetype where you just show up and you sort of can see around corners and have this rare instinct without having to dig into that detail, that’s what leads to foolish decisions because you’re too far removed from the actual doing of the job itself. So I think it’s this weird combination of both where the vision comes from actually being in the in the weeds.

Joe: [00:21:20] Yeah. I think that’s going to help a lot of people to hear that. It’s a grind some days, like it really, just getting in there and being in the weeds for a while to get get it done. Get the job done.

Jake: [00:21:28] Totally.

Joe: [00:21:29] And you know, you talked about alignment a little bit there. Internally here, we’ve been talking a ton about, with our teams, what are the big trends, what are the bigger issues that we’re seeing in our jobs here? And more and more that our product team is working on is just aligning so many different stakeholders nowadays and making sure that they’re all on the same page, you know, one team, one dream here, trying to the product delivered. And we saw that a little bit in the survey too.

Jake: [00:21:53] Yeah absolutely. And one of the surprising findings perhaps is that product managers saw that sales, marketing, UX, design, have more important points of alignment than engineering. On the surface that might feel a little alarming, but as you dig into it, it’s also revealed that they feel pretty well aligned with engineering. I think they feel like there’s more work to be done in some of those other functions. Customer success came up as an area where there might be a bit of a gap. I think that’s a really interesting one. In a sense, product management and customer success are becoming kind of like the sales and marketing alignment. Like you need to get that right for the business to function. And that’s because, in a SAS-based world, you need to continue to renew loyalty. You need to continue to deliver value and delight. It isn’t one and done. And customer success is so close to the customer, so close to the renewal, and the feedback that they get and the early signals that they’re able to detect are so important to be brought back into the product management process to ensure that the product is getting better and better in conjunction with that sort of signal as it’s being revealed.

Sean: [00:23:06] Yeah. So the key measures that we have for our product success are what we call “trust, loyalty, and advocacy,” and we’ve actually come up with a way to define specifically, what does that mean? What does that customer journey look like? And when your customers are behaving as advocates, they do things for you, like they volunteer to be a beta tester or they will give you constructive feedback on how to make the product better.

Jake: [00:23:24] You bet.

Sean: [00:23:24] They’ll basically invest in your future and that is like the holy grail, when you have an ecosystem that’s helping you build a better product at the end of the day.

Jake: [00:23:31] Yes.

Sean: [00:23:32] You wrote an article, another one from the past, on, in support of really, of NPS. I’m more in the Jared Spool camp now.

Jake: [00:23:41] Oh boy, haha.

Sean: [00:23:43] I think NPS has a place for certain types of products and services where you have such a large population of people and you don’t have a better way to get access to customer behaviors. But in our world, we’re developing software products. So we can collect actual human behaviors and measure those. So why would we interrupt somebody with a survey that really doesn’t mean anything except their sentiment at a point time? It’s not a measurement of how they would actually behave. So anyway, I’d love for you to talk about that a little bit.

Jake: [00:24:12] Yeah I feel like I continue to offer hedge answers that are essentially that it’s all of the above, but I truly feel that NPS is a really important metric. But it in and of itself is insufficient. It’s a good measure, it’s a solid measure, of advocacy, of willingness to advocate on behalf of a brand. That reveals a level of satisfaction. It’s imperfect, but it’s widely adopted. It’s very very simple. It’s one number, one question rather, and people get it, you know, they’re familiar with it. So I think the fact that it is simple and standardized counts for something, but it alone doesn’t tell you all that much. In the absence of behavioral data, you only know the what, not the why, and that’s why… So for example, within Pendo, we do allow our customers to deliver NPS surveys in product, but we also give them deep insight into how users are engaging with that product. And when you combine both of those things, the qualitative and the quantitative, you can start to really derive pretty actionable insights and answer questions like, not only, “how does my user base feel about my product?” but, “why do they feel that way? Which features are driving delight and frustration?” So it’s really the intersection of both the behavioral analytics for understanding the root cause as well as more qualitative measures of sentiment. And the final thing I’d say on that is that NPS is one of a constellation of qualitative measures that you should look at. One that I particularly like is customer effort score, which is a measure of friction within a product. And it essentially asks, “to what extent was..,” and this is maybe not the most elegantly written question as I’m phrasing it, but, “to what extent does the product solve the problem that I was seeking to solve?” And ultimately I think that that’s a really really good measure of whether it’s been a satisfying experience because it speaks directly to the need and whether the product met that need. So NPS, customer effort score, CSAT, all of the above matter, and then most importantly, pairing that with behavioral analytics to understand the root cause behind the feeling.

Sean: [00:26:20] Cool, I love it. I got one last question for you and then I think Joe has a question and then we’ll wrap up here. But one of the things that Jared Spool talks about, independent NPS, didn’t want to scare you there, is about the big bag of money that some customers come along with, right. So you’ve got a product, you got a customer that comes along with a big bag of green money and they’ll take you off vision to build this feature set for this one customer. And you guys recently, just this month or early last month, put out a report called the feature adoption report that I love. You’ve quantified, you’ve figured out how to quantify, the cost of bloat.

Jake: [00:26:53] Yep.

Sean: [00:26:53] It’s cool. So I just want to give you a chance to talk about that and what you guys found in that report.

Jake: [00:26:58] Yeah absolutely. So what you’re saying is right on. A lot of the features, most of the features we’ve found that companies deliver are just never adopted by end users or they’re never adopted or rarely used. And, you know, when you think about it from the perspective of the value of engineering resources, you know, the scarcest talent resource in the planet, and they’re spending their time building features that end users just ignore. That doesn’t feel so good. So we decided we’d try to quantify that and because we’re measuring feature adoption and measuring product usage we were able to aggregate and anonymize a lot of data to get to a statistically valid and representative sample to understand, on average, what percentage of features are adopted. We normalized it too to sort of adjust it for the impact that Pendo has on driving feature adoption and what we derived from this analysis is that 80 percent of features are rarely or never used. And when you look at that through the lens of the economic impact of those features, estimating the R&D spend for an average SAS company of being somewhere in the neighborhood of, I believe 18 or 21 percent, you can extrapolate the economic impact of those features that are never adopted or rarely used. And it’s a very big number. It’s twenty nine point five billion dollars for SAS companies worldwide.

Sean: [00:28:21] That’s crazy. I’m going to use that statistic everywhere, by the way.

Joe: [00:28:25] Very useful.

Jake: [00:28:27] Awesome. Yeah, it’s gotten a lot of attention. It’s kind of an eye-popping number and I think kind of an interesting way to put a sharp point on that idea.

Joe: [00:28:37] It’s crazy how much waste there is in software, we try to throw people in the right direction…

Jake: [00:28:41] Absolutely.

Joe: [00:28:42] All right, so last question about the survey here. So if you’re gonna put your hand to your head and turn on your psychic abilities, what do you think the 2020 version of the survey is going to show?

Jake: [00:28:53] Oh wow. So one thing I should mention about 2019 is that we saw a pretty low NPS score. It had fallen fairly significantly year over year and this was the NPS score where we asked respondents whether they would recommend their profession to a friend. By and large, they said no. That wasn’t so great to hear. But I think that it’s probably, some of that dissatisfaction is wrapped up in frustrations of misalignment, the fact that they have a lot of responsibility and don’t always feel empowered, the grind that we’ve been talking about… I think that, you know, I’m hopeful that next year we’re going to see a better NPS score, more satisfaction with the profession. I think that we’re gonna see more product management teams reporting into CPOs. I think that, whereas this last year marketing was still the dominant reporting line, I think that it’s going to tip in favor of CPOs as the dominant reporting line. And I think with that, we’ll see more strategic involvement and driving growth for the business, I think we’ll see more acknowledgement for contribution, I think we’ll see more influence in business decisions, and accordingly, I think we’ll see happier PM’s. That’s my prediction.

Sean: [00:30:02] Cool and I hope we see more and more of the product feature set being driven by behavioral metrics.

Joe: [00:30:09] Oh yeah.

Sean: [00:30:10] Right. And more insights to come from the team.

Jake: [00:30:11] Let me add one more. This is the other prediction that I’m hoping comes true next year. One of the findings that we didn’t talk about is that product managers, more than any other metric, measure success on the basis of feature delivery, of shipping features. I believe that’s the wrong metric, and this ties into the conversation that we’re having about features that go unadopted. It needs to shift downstream to KPI’s associated with adoption, KPI’s associated with retention and associated with renewal. So the metrics that matter to the business are the metrics that matter to the PM. And today that’s a little out of alignment.

Sean: [00:30:48] Agreed. Alright, last question. What book would you recommend to our audience? What’s the number one book that you recommend?

Jake: [00:30:57] Oh man. So I should betray my cynicism about business books. I don’t love them. There are a few that I’ve read over the years that have changed my life and most that have bored me to tears. The ones that have changed my life; I’d say The Tipping Point, Crossing the Chasm, A Hundred Years Ago, The Innovator’s Dilemma. These are really important books. If you haven’t read them, you should. Now fast forward to today, I read a lot, but I read outside of my domain mostly. I look to other forms of narrative to find inspiration because I think that, frankly, creativity is, or one form of creativity is, taking known ideas from one context and applying them to another.

Sean: [00:31:37] Call that idea sex.

Jake: [00:31:39] Yeah I love that. That’s great. That’s fantastic. But coming back to business books, the one that I’ve read in the last I’d say year or so that I did love was Shoe Dog by Phil Knight, the founder of Nike. If for nothing else because, A) he’s a great writer and a really thoughtful person, but also because he put so much passion into building a brand and building a great company and I don’t think it fundamentally changed him as a person.

Joe: [00:32:02] That’s a good one. We haven’t had that one yet. Very cool.

Sean: [00:32:04] I did have somebody recommend that to me recently so that’ll be the next on my list.

Jake: [00:32:08] It’s really good, really good.

Sean: [00:32:10] All right. Is there anything you want to plug for Pendo or anything you’ve got coming up in the near future?

Jake: [00:32:15] I have a couple things. So Pando has an editorial site, we publish daily content, so we have a weekly community poll, we have a point-counterpoint debate that we run weekly, and we publish articles, point-of-view articles, best practice, at productcraft.com. We also have a conference coming up in San Francisco on May 9th. It’s called ProductCraft: The Conference. So we’re trying to break the model a bit and create different, sort of interesting, spaces, more different ways to deliver insight and learning that isn’t quite as sort of dry and tired as the traditional conference experience. If you go to the ProductCraft site you’ll see a way to learn more about that conference. It’s May 9th in San Francisco at the Palace of Fine Arts.

Joe: [00:32:58] And where can people get the survey as well if they want to look through it more?

Jake: [00:33:01] ProductCraft or Pendo.io.

Joe: [00:33:03] Sounds good. Well I thought this survey was a great idea. I went through it line-by-line, both years, just seeing what I disagreed with, what I agreed with, and I’m really pumped that you guys are putting it out and I hope to see the next version next year.

Jake: [00:33:16] Really appreciate it. Nice talking to you both.

Sean: [00:33:20] It’s really valuable work. It’s improving the practice and the craft of product development and it’s certainly been useful for our business. Thank you for your hard work on that stuff.

Jake: [00:33:28] Absolutely, our pleasure. I think we’re passionate about the same things so I’m glad we could make at least a small contribution.



09 / Finding the Right Metrics


How do we know our work is working? How do product designers know their work product is solving the problem it was intended to solve? That’s the kind of question that keeps us up at night.

“It’s an insidious question,” says Kate Rutter, designer, tech junkie, artist, and Principal at Intelleto. In this episode, hosts Sean and Joe chat with Kate about Metrics, but not just any metrics and not just those that only measure performance. Kate says the true power comes from our alignment around metrics as a very tangible element that people can get behind. “It gets really exciting when you…start to see metrics as human behaviors with your products stated in numerical terms.”

Read our blog post

About Kate

Kate is an entrepreneur+designer and Principal at Intelleto, where she creates visual explanations that make complex ideas simple, memorable and shareable. Kate pioneered the UX learning track at Tradecraft, co-founded Luxr.co, and was Senior Practitioner at UX consultancy Adaptive Path. She co-hosts the NSFW podcast What Is Wrong With UX with Laura Klein, tweets at @katerutter and blogs at intelleto.com.

l t f i  


Joe: [00:00:00] All right. Well on today’s episode we have Kate Rutter, and Kate, we’re going to have you introduce yourself. Today we are really going to talk and focus on metrics. What are they? What makes for a valuable one? How do you know if you’re using the right kinds? What kinds to avoid… What do you think, Sean?

Sean: [00:00:15] We’re big on metrics here at ITX. So we have our own kind of flavor for how we deploy them and we’re interested to learn more from you.

Joe: [00:00:23] Data, right?

Sean: [00:00:24] Data.

Joe: [00:00:25] Kate, hello.

Kate: [00:00:27] Hello. It’s such a pleasure to be here.

Joe: [00:00:29] We’re very happy to have you. Would you mind introducing yourself, where have you worked in the past? What are you doing now? What’s your role?

Kate: [00:00:36] Sure. So I have been a longtime UX generalist. I was director of technology for a non-profit fairly early in my career. And that was when this web thing was just starting to wake up and I saw a lot of potential there and so I started playing around with it. And since then, because I live in the Bay Area, I always thought I might leave and go somewhere else, but the world of work just kept getting more interesting and so I’ve followed that path of curiosity. A couple of things you should know is, I’m an opportunist, which means I often find myself working at the beginning of something that’s happening that’s kind of big, like the Internet, or at the time desktop publishing was even part of my background…

Joe: [00:01:20] Oh…

Kate: [00:01:20] Yeah. User experience, as a field, we evolved from hand-coded, you know, “look I can make this thing,” to, “what thing should we make and how does that affect people’s lives?” Designing more for impact. So what I have is a rich set of experiences and an extraordinary set of people that I learn from, that I’ve gotten to work with as clients and as as colleagues in a variety of ways. But I’ve never worked for a big company so I’m not sure if that’s in my path or not. I’m curious about that. But I did spend quite a few years at a company called Adaptive Path which was a consultancy really heralded for pioneering a lot of terrific thinking and user experience design space. And that was a life-changing experience and it really took me from thinking about this world of digital technologies as things and technologies that were interesting ways of kind of how the technology functions into really thinking of technology as always in service to people and how people can realize their dreams and find their aspirations and discover new things about efficiencies or being effective or connecting. And that, I think, is a life changing shift, but it was also a shift in the practice. And so I’ve been very lucky to be able to be part of that shift to the practice. So after my time at Adaptive Path, I co-founded a startup with two colleagues, Janice Fraser and Jason Fraser, a married couple whom I’d known for years. And this startup was called Luxr and it started out as a program to help early stage technology startup founders really integrate user experience and user-centered thinking into how they think about their products. And this was happening at the rise of Lean Startup as well. Janice and Eric Ries know each other. And there were all these different ways of maybe, not short-cutting, I don’t like that phrase, but really being more efficient and validating ideas before putting a lot of effort into the development or design of those ideas. And that was an important stake in the ground, I think, for our field as well. So we had a good couple-year run on Luxr, and then as many startups do, we gained foothold but not scale. And so then I really was excited about finding behavior change in our field through individual practitioner behavior. So many people I know focus on scale and organizational change and team change, but I really am in the seat next to you as a practitioner. I think that change comes when we change our individual behaviors and how we think about design and that that infectious kind of approach of design can infuse a team and then infuse a company. So since then I’ve been really working as a design educator.

Joe: [00:04:05] Interesting.

Kate: [00:04:07] Yeah.

Sean: [00:04:08] That’s great. You recently talked about, in some interview I read, about unicorn hunting. I think this plays along with the individual practitioners, and anyways, you want to talk about that a little bit?

Kate: [00:04:18] Sure. Yeah. So as part of where I spend my time, I teach at an institution called California College of the Arts. It’s a fabulous fine arts institution that about a decade ago started investing in interaction design as a practice. And they have a real focus on social impact there which I think is important especially for designers now. So I spend my time teaching as an adjunct professor in their interaction design department. And then I spend another part of my time with a colleague Laura Klein who is kind of a known name in the lean user experience world. And she and I have a podcast together where we drink and we pretty much bitch about what’s wrong with UX design, but also how to make products suck slightly less. And you know, we harvest all of our past experiences and our observations and then we debate about it. We share a lot of deep fundamental values but we really disagree on how those values come to play in human behavior and in teams. And so we fight about it and we made this podcast on it. And we did one on unicorn hunting specifically for teams or managers or hiring organizations that are looking for someone who can do it all, who can, you know, deliver pixel-perfect visual designs as well as validated code that can launch and be deployed. And it’s like, well, you know, the specialization of our field over the past couple of decades is really allowed us to do many many more magical things, so expecting one person to have that breadth of skill, there’s just not enough time for it. So we talked about how unfortunate that is when good people can’t find a good fit in an organization because of this unicorn myth that we have.

Sean: [00:05:54] Right, and it takes a balanced team with lots of different personalities and lots of diversity, this is my opinion anyway, and lots of different skill sets to really build great products.

Kate: [00:06:06] Yeah. That’s the exciting part, right, is we have so many people coming from different adjacent fields with their own experiences and expertise and the problems they want to solve and that’s amazing. And getting a group of people who are really different to work together is a total pain in the butt. It’s hard because you have to find that shared ground of why we’re here. And then, I mean, even if you can do that, which is its own challenge, you also need to find a place where compassion and empathy allow for an effective workplace where you can understand each other, listen to each other even if you disagree, use your dynamic of disagreement to make the work better but not to piss people off, but also to allow people to change over time. And that’s asking a lot. It’s asking us to be different humans than we were when we showed up, we did a job, we punched a clock, and then we left and went home to our quote “real life.” And so this is a challenge. But I do believe, I agree with you, that more breadth and more diversity of experience and diversity of perspective is more likely to help us solve the hard problems that our products are now in the position of solving.

Sean: [00:07:13] Which brings us back to goals and metrics. So I believe, here’s my theory about metrics, is that they’re only really useful if they align people. So if you have the right metrics and people are aligned around those metrics then the diversity becomes more valuable because we’re all going towards the same set of goals, right.

Kate: [00:07:32] Yeah I think that’s an astute statement. I I love listening to your podcast because the way you all phrase your own perspectives and hypotheses with your guests is always very succinct.

Sean: [00:07:43] Thank you.

Kate: [00:07:43] I’m like, “oh, I would like some of that succinctness.” Sadly I do not have that gift yet. One of the things though, about that alignment around metrics, is it’s an element and it’s a very tangible element that I think people can get behind but it gets really exciting when you as a team and as a company start to realize that metrics are behavior, human behaviors with your product, stated in numerical terms. And so by committing to a metric and aligning around a metric the hope is, and I think the deeper purpose, really aligning around the the intent of your product in your customers lives. And because words can be so facile and hard to wordsmith or hard to really lock into our goals and the why statements and the purpose statements, I think metrics can be a more specific and clarifying tool for teams that are trying to rally around similar goals.

Joe: [00:08:38] So as you know Sean, metrics are great to rally around. But if it’s not the right metric it can be dangerous. So Kate, what do you think makes for a valuable metric? Like how do you know where to even start with knowing what metrics to use or what not to use?

Kate: [00:08:51] I love that. You know, I’m going to back up a little bit and tell you how I came to this work because many of my colleagues would be like, “it’s odd that you’re talking about metrics,” and I wouldn’t disagree. But when I was at Luxr with my co-founders, one of the elements that was so important early startup teams was figuring out how they know if their work was working and if some experiments they were delivering or the product content they had was solving a problem. And we immediately looked to the world of measurements to help validate that and to figure out if was something there. And the world of metrics, you know, there’s a ton of smart people working on that. I mean metrics and business performance have been out there for a long time, right. So I don’t in any way profess that I have a big, broad picture of all of that expertise. I think that would be impossible. But one thing that it did help us do is realize that, I hope every designer and every person that works on a product that doesn’t have an effective metrics program in place, I hope the question that keeps them up at night is, “how do we know our work is working?” And that’s such an insidious question. And the more I heard it from the startup founders, the more I started to internalize it for our own work and it was a question that I had to start to reconcile with. Because delivering an interface or a design or a feature or even releasing a product was no longer sufficient. That wasn’t done. It was, “what has changed in the world and the use of our customers as a result of that release?” And so when we talk about the right metric, it’s kind of like saying, “well what’s the right personality to be successful?” There is a wide range of things that can work for you, but the right personality to be successful is the personality where you feel you’re the most you.

Joe: [00:10:31] Right.

Kate: [00:10:31] Right, like you feel you can truly be yourself and be that best version of yourself.

Sean: [00:10:35] I love that.

Joe: [00:10:35] And it’s personalized too.

Kate: [00:10:35] It is. And so here’s a quote that I love which is from Bokardo, who is Joshua Porter, and he says, “your metrics will be as unique as your business.” And that’s the kind of metrics that I’m talking about, especially for product success. So your company can have metrics related to sales or growth or market adoption or customers et cetera. But when we’re really getting down to the product level, when you phrase a metric, it should be specific enough that just by hearing the metric that you’re measuring, people should be able to tell what your product does. And so that’s kind of more the right metric. And unfortunately metrics are so tightly held that I don’t have a lot of like numerical, specific examples because they’re just not… That’s my next phase of research is finding the companies who will open up about that. People are very cagey about their metrics because they’re so tied to their business performance. But I was working with a team and their sales numbers were totally off the charts. They were doing great. They had a product line, consumer electronic, that was doing super super well. What they found when they started to look at it was that people were buying the thing but they were not using the thing. And all of a sudden there was this kind of flash of cold water, they’re like, “well okay, so let’s project how that might play out in the future, everybody buys this thing, nobody really uses it, and now it’s sitting around in your house unused, what do we think the endgame is going to be on that?” And it could be just a very quick kind of collapse of that business line, right. Because at some point people get smart. Like we pay for a gym membership because we’ve got all the best aspiration and goal to go every week to the gym, and after three months you’re like, “why am I paying for this, I haven’t been once.” And everybody cancels in March because they New Year’s resolution didn’t get them to the change in behavior. And that’s the kind of fragility that I want to help our teams avoid. So when you’re talking about the right metric, it would be the kind of metric that’s intrinsically connected to the purpose of your product and your customer’s life and in a way that’s instrumented through the product interface or behavior so that you can start to count the use of that product in that way and then hopefully improve it intentionally over time.

Joe: [00:12:40] Got it. And so, you know, I go to these home pages a lot of times for products just to learn about them or whatever and I see all these big numbers flashing in front of me that seem to be very impressive. And so back to your point there, how do you know when these metrics, and they’re called vanity metrics, they sound good, they look good, they make you think the companies being really successful, so you can maybe define a vanity metric for us? Because I loved the way you stated it before when we were talking in a prior time.

Kate: [00:13:05] So this comes from the Lean Startup management philosophy of, there’s vanity metrics and then there’s actionable metrics. And I credit Eric Reis with that. There’s also some really nice practitioners working in Lean Analytics that lean heavily on this separation of, kind of, church and state. A vanity metric is a metric that only ever goes up over time and it doesn’t change your behavior. Like you can’t do anything about this. So when I talk to startup founders, they’re notably proud often of top-of-funnel metrics that grow over time like, “we had over 10000 downloads,” or, “you know, our average time on our site is 30 minutes or 20 minutes,” whatever. Or, “we had 450 new updates this week.” And the thing about those numbers is they’re positive in that they help you feel good. And we as humans need to feel good because our work is hard. They can also influence stakeholders, they can influence investors, they can influence the public markets; so they have a real utility because people want to follow success. I mean they have to be honest, right. You can’t lie about your numbers, that’s fraud. But they do grow over time. What they they don’t help with as design and product teams is they don’t help us understand if our product is working for our user. So you might have a real peak in adoption and get ten thousand downloads, but how many people have taken that download, opened your product, actually started to use it, and then came back to use it again to where they become a habituated user? And those are the numbers that tend to be much less sunny and so people don’t like to talk about those as much, but as product teams we have to look in that mirror and see those real actionable metrics, things that will change how we affect our behaviors for our products.

Joe: [00:14:44] Yeah I think one of the telltale ways to spot a vanity metric is when it’s the total of something with no timeframe.

Kate: [00:14:49] Right! Which brings us to, so well if that’s not a vanity metric, then what’s an actionable metric?

Joe: [00:14:56] Right.

Kate: [00:14:56] You know, one of the things that I really do in my practice is take the knowledge that I think people are exploring out there, because there’s a lot of people working on metrics, and try and make it into something that an everyday practitioner can integrate into her or his practice. And so on this one, I’m going to use the definition of actual metrics that I learned from Lean Analytics which is a book that was published by Alistair Croll and Ben Yoskovitz; I’ll probably mention it a couple of times. It’s a fabulous book in that it does talk about the different stages of maturity for an organization or company or a product and help you identify some things that might be crucial at different time phases. But they were the most succinct I think in defining that actionable metric which has some certain characteristics. It’s clear and specific. It is normalized, and I can go into more detail about that. It’s comparative, so there’s time differences, you can say, “what was it before, what is it now, what’s changed?” It’s actionable meaning that the change in that number helps you and your product cohort make a difference in your product, actually do something about it. And then it changes your behavior, right, you actually want to do and need to do something about making that measure, that metric, move. And so those are the attributes of it and where I think we’ve been struggling is, especially UX designers, product people I see much more well-balanced in their metrics fluency. I think developers, and certainly data analysts and people for whom data is their creative material, they’re really well versed in this. But what I see is a real differential between designers being able to integrate metrics behind those goals that those metrics represent into how we shape our actual product behaviors, the features we decide to work on, the purposes of those features. That is what I see as a real gap. It’s like we have expertise on one part of the market and we have expertise on the other part of the market, but what we don’t have in our companies is this consistent team commitment to metrics that change our behavior and help our products improve. And that’s the gap I’m hoping to fill.

Sean: [00:16:50] I like how you said earlier about how your metrics should be about how you’re improving how your product is used. It’s not like just increasing usage or, like we said earlier about the vanity metrics, it’s really about understanding the purpose and fit of your product and understanding how we’re going to measure that it’s actually being used to productively meet that fit, if that sums it up.

Kate: [00:17:12] It does sum it up. There’s a very small, or I think it’s growing and a huge focus, point between the nature of the metrics that people start to capture. Common metrics would be often around adoption. A good framework, especially for startup founders, is Pirate Metrics by Dave McClure at 500 Startups. And so it’s this categorization, it’s almost just categories, of types of metrics that you might need as your product starts to evolve, or if you’ve got a product already in the market, that you might try to better understand at a deeper level. And pirate metrics is: acquisition, activation, retention, referral, and revenue. And there’s been a lot of good play about that, you can Google it, it’s a great term. And the reason it’s called Pirate Metrics is the first letter of each of those spells AARRR.

Sean: [00:17:57] Aarrr!

Kate: [00:17:57] It’s kind of this pseudo mnemonic thing. But of all of those, I mean to grow a product you need to increase acquisition, of course. To hook people in or to ensure that they’re connected for usage, you need activation. The real thing that I think drives overall product in long term company success is really retention. How do we keep people? How do we ensure that what we’re delivering them is meaningful and valuable so that they will stick around? And then when that’s a positive cycle, you hope that there will be referrals. And then, I mean unless it’s a hobby, you gotta have some revenue and you want that to kick in. So really it’s that retention line. And looking across multiple frameworks for metrics, the one that most of them have is retention. And that has been where I’ve seen a whole heightened attention on retention because that’s the element that demonstrates customer ongoing loyalty, success, trust loyalty advocacy to use your own loyalty ladder that I know that you use as part of your business.

Joe: [00:18:58] It’s the regulars at the bar.

Kate: [00:19:01] Right, yeah.

Sean: [00:19:01] So the loyalty latter is very similar to the Pirate Model in terms of flow except it’s from the user’s perspective, like how is the user exhibiting behaviors that we would indicate as trust, loyalty, or advocacy.

Kate: [00:19:13] I love that. I think that’s a terrific categorization and place to rally around.

Sean: [00:19:17] Yeah and in the long run if you don’t tie that back to your ultimate metric, which is profit and revenue, then you don’t have a business.

Kate: [00:19:24] Right. And that brings up the other point, which is something that, again I focus a lot of my messages on design teams, but that also means that they need to work in well with product teams and with broader business units. But the business metrics can be very distinct from the product metrics. This is one of the examples that just irks me, but I’m not going to change the world in a day and I hope there are other smart people working on it, but the bestseller list. Like that is such a publisher business metric, right. But we’ve condoned this best seller, like, “oh I’m on the bestseller list. What’s the bestseller?” as this proxy for readership, right. We assume that if people buy it, then they will read it. And what would I really love to see instead is like the most read list or the most read and lent. That, I think, would be a much more meaningful metric for authorship. Maybe not so for publishers who are in the business of sales, but for authorship or for people who are convening ideas that are important and meaningful, I think readership and lending-ship is a much more astute behavioral actionable metric. But we don’t measure those because we are so tuned towards a business metric that we haven’t really developed the same level of sophistication around usage. But now with our products and our digital products, we not only have the opportunity to do that, I think we have a mandate too.

Joe: [00:20:41] So we’re talking about metrics alot, just to make it a little more tangible for the audience, you know, we talk a lot about big companies like Facebook and Twitter and Snapchat. When they do their quarterly earnings calls, you can get some insight into the metrics they care about and a lot of times you’ll hear about like MAU, DAU, it daily active users, for example, and it’s really one of those metrics where you’re like, “yeah I get why people measure it but is it really the best metric?” So can you talk a little bit about an example, whichever one you want, of what’s a good metric versus a great metric. How do you take a metric and actually make it much better? Because something’s better than nothing in a lot of cases.

Kate: [00:21:17] Right, for sure.

Joe: [00:21:18] But then how do you make it those best ones, like you were saying before about making it normalized and all the other factors.

Kate: [00:21:23] Right. So to go back to Josh Porter, a great metric is going to be one that is unique to your business. But I actually have a step-by-step continuum that teams can use to start from basic or mundane, unhelpful metrics into something that’s truly awesome. And to do that, there’s this concept of, “well what is our product do for people?” And again, this doesn’t necessarily work up to the business metrics as effectively as it does to work down towards more detailed things, like, you have a product, you have features, you have interactions. And so this level of questioning really helps forward building the right thing for the right reasons. And at Luxr we created this term of a key use. So what can someone do with your product that they can’t do without it? And unless you understand that, you don’t have the clarity to be able to identify a metric that would be meaningful for your product to measure. So that’s your first step is to have a hypothesis about what it is that that product, that feature, that interaction does for your customer that helps them complete something, experience something, or have a specific behavior. And interestingly enough, we call it key use, but this level of thinking is all over the place. There’s a concept of core action which has been popularized by Josh Elman at Greylock Partners. There’s critical event which some of the more analytics platforms are starting to use. But there’s this concept through all of those that there’s this thing that people need to be doing in your product that you’ve gotta measure, and that’s the first step. So let’s say, let’s walk through, continuum, for an example. So the example that I use which is pulled out from a variety of the startups I worked with at Luxr is a consumer mobile app for task sharing: task management, sharing it, seeing things completed. And the key use for that is to share a task and to confirm that something was done. And it’s super simple. It takes a while to kind of get to that level of simplicity. So a metric that would be unhelpful for that product would be, like, sign ups. First of all, it’s a category and it’s just, like, saying people might have given it a try but it doesn’t actually provide any utility or use to them. A vanity metric would be total number of registered users because that’s only ever going to go up over time, it’s not comparative across time periods, it’s not normalized so it’s a pure number; it might be 50, it might be a thousand, it might be a million; but it doesn’t tell you anything about the behavior. Starting to get good would have some of the elements of the actual metrics. So the percent of new users per week. Now that would be an acquisition metric, right you’re trying to grow revenue users per week, but at least it’s a percentage so of your entire user base, what percentage are new, and we’re going to measure that on a weekly basis. And that adds in normalized and it adds in comparable over time. Something better than that would be the percentage of users who sign in or who interact with the product three or more times a day, per week. So now you’ve added a specific behavior that your product needs to instrument or event and you’ve got to capture. You want to know the percentage of users we’re really using your product quite a bit within this time span of a day. And then you’re gonna measure that weekly. Now all of this, as you can tell, gets exponentially harder just to keep track of the numbers, much less the timing that the events would fire and how you visualize that which is its own challenge. But we’re getting closer. Even though a lot of products have habitual use, people use them multiple times a day. So that still doesn’t tell us what’s unique about our business. So something that would be awesome would be the percentage of users who share a task, three or more times a day, per week. So if you can capture the numbers at that level then you can start to say, “how do we move that number up so that we can get closer to 100 percent of our users sharing a task three or four times a day?” Now it might be that users don’t have that many tasks to share. So are there other things that they could share that would really involve them with your product and help them get more utility out of it? That again is a user research question and user research is the place where all the best metrics come from because they’re unique to that task. So that would be a continuum, and most teams start out with like, sign ups or total number of users, and that’s a fine place to start. You have to start somewhere. But when you really are able to nail that core action, sometimes on a per feature basis, and measure it, and start to adjust it and change interactions so that that number moves over time, that’s an extraordinary skill. That’s kind of the Holy Grail.

Joe: [00:25:38] That was a great example. Thank you. And Sean, I think we’ve probably had every guest talk about this so far, about user research, we should probably just rename the podcast to Do User Research.

Kate: [00:25:47] That’s funny. You know, when Laura and I talk, the thing that we always end up with is task flows because I actually think that that is one of the elements that makes the designers so much different and better than others. Like task flows is it, and of all time our podcast on task flows is like our number one download. So yeah, you could redo the user research.

Joe: [00:26:06] It’s off brand, Sean I’m sorry.

Sean: [00:26:07] No, no worries. I agree. So one last question then we’ll begin to wrap it up here. Short term versus long term metrics. Is that something that you’ve thought a lot about? So as we’re talking a lot about technical sort of product metrics like how people are using it, but over the long run is there something different you might measure or think about measuring?

Kate: [00:26:26] I haven’t thought about it in those terms but there was a really interesting insight that came out of work that I did of when I was at Adaptive Path. I was on a project with Jesse James Garrett who’s a fabulous practitioner. He wrote The Elements of User Experience. And we were working with a team that was looking at where they wanted the roadmap to go. I have a lot of opinions about road maps, but that’s off the topic so I won’t go into them, but projecting like, what would allow this product to grow in healthy ways that would really benefit the customer, and of course enhance the business possibilities? And Jesse came up with this fascinating thing, he called it the More statement. And for that specific product which was Gumtree, so it’s similar to like Craigslist for the UK. The More statement was that overall for the product and business to be healthy they needed more people making more trades or exchanges for more money with more other people more often. That’s a lot of mores, but what each of those allowed us to do as this workshop team was take those more statements and say, “OK more people, what kind of more people? Who are your people?” And making more trades, “is there a cap on the number of trades or exchanges or sales people would make? How do we up that? How do we help them think about Gumtree in a way that allows them to think, ‘oh this is the first place I go anytime I need to get rid of something or acquire something.'” And for more money, their platform had some basis of the amount of money, but when you get into big-ticket items like cars or boats or big things, it didn’t feel like that was the place to do a high level exchange like that. So how could they evolve into allowing those types of exchanges or trades? And were people mostly exchanging with people in their neighborhood or were they doing things at a longer distance, so how could they make exchanges with more people? And then how could people do it more frequently? And sometimes that’s enough of just doing some outreach and notifications to tell people your product’s still there if it’s not something that has a habituated daily use. And so each of those little more statements kind of created this population of how could growth or invention could happen in our product that we could capture more of the market or better serve our market and then start up what you might consider that loyalty ladder of then trust, loyalty, and advocacy? And that was an interesting way to get at that long term thinking and to almost, in some ways, predict what kinds of metrics you might want to capture over the long haul as a comparative from now to then.

Joe: [00:28:46] And that’s where Moore’s Law came from.

Kate: [00:28:49] Well said, well said.

Joe: [00:28:51] Just kidding. That’s a joke, Google it if you need to.

Kate: [00:28:53] That’s right.

Joe: [00:28:54] So, you know, with metrics it’s all about having the data to have the metric and what we see happening, which is on everyone’s mind is things like machine learning and AI. How do you, or do you, see that playing a role in how metrics that companies care about or that they should care about see that evolving in that discussion?

Kate: [00:29:14] I do and I’m not quite sure how. I’ve been dabbling a little bit with the research around machine learning. I know there’s huge amounts of opportunity there. I’d say I’m a novice in that, but eager, and it’s on that threshold. So it’s that next level of thing that I want to explore. I think right now where we can look at is the rapidly changing landscape of analytics packages. At the minimum, you know, there’s Google Analytics, but at a much more sophisticated level they’re really starting to be more attuned towards specific product events and how we capture events and being more integrated almost with a development flow of a product. So I’ve been poking around a little bit with amplitude, and I don’t advocate any one platform. I do know that if you just take Google Analytics out of the box like it’s all vanity all the time, right.

Joe: [00:29:56] You’re right.

Kate: [00:29:57] It can do anything. But without knowing what it can do, you don’t really get the utility out of it. But amplitude has a really nice white paper and they define that kind of key user correction as this critical event and then look at how that can be instrumented into your overall thinking to enhance retention. And I think they’re putting some nice thought leadership in it. Obviously it grows their market so it’s a good investment. But for people who are in a team where the quantitative expertise is already high, I think looking at those analytics packages, they’re all starting to say the same thing that UX teams and product teams have been saying, which is, “our product has to work for the people and when we measure how it works for the people our products can improve much more dramatically,” and then the sales are the outcome of that. So that’s kind of the virtuous cycle I’d like to see kick in.

Sean: [00:30:44] You know, along those lines, I like to say that the right metrics can unleash creativity in all of your unicorns that you have working on your team.

Kate: [00:30:52] Oh hells yeah. Yeah. I do quite a bit of workshops, also with Laura Klein, and she has this fabulous thinking method. She calls it, “walk a metric up and walk it back down.” This is a common situation, I’m sure you all have been in it before, which is, someone who is highly influential, it might be a client if you do agency or consulting work, it might be a product person or a business person in your own company, and they come and they say, “I want this thing.” Like, “I want this feature.” Jared Spool calls it, “someone’s going to give us a big bag of money if we build this thing.” And the question is is what does that thing represent. And it’s not that people are stupid, like we ask for solutions because they’re specific and they’re knowable and that’s how we think as humans. But what it is is it’s an opportunity for any team to walk that question or that request for a feature through a set of inquiries and then come back with other new ideas that might better solve the problem that feature represents. And we all need help and guidance with this. So from the feature, like if you say, “we want product recommendations,” or “we want video for our e-commerce site.” Whatever it is, you say, “well what is the outcome, like when we have that thing how well our product be different? Well if it’s product recommendations, we might have more sales per customer.” It’s like, “OK so if we have more sales per customer, how might we measure that in quantitative terms?” And there’s probably going to be discussion about that, but you might come up with something like, “well there’d be an increase in the average order size, OK, so in order to increase the average order size what would that look like in our product in a very specific visual way?” And the behavior of the human is that they’re going to check out with more items in the cart. So if you’re really looking at that behavior, then you say, “how could we, in our product, help people checkout with more items in the cart?” And it might be that the feature that would best be worth exploring or experimenting with it doesn’t take nearly the effort or the focus that creating brand new product recommendations does, because there is a cost to that feature development: technical cost interface cost complexity cost for your users. So maybe it’s a free shipping threshold, maybe it’s an add-on product, maybe it’s 10 percent off a second or third or fourth item, maybe there’s some business logic that you can invest in that doesn’t take a development effort that would satisfy that behavior and therefore move that metric. And that’s the type of thinking that I think all of us on teams need to get very well versed in without being a pain in the ass about it. Like, without pushing back and saying, “well that’s just a feature, why do you need that?” Like we don’t deserve to be shirty, we deserve to have open and empathetic inquiry into the real purpose that people say when we speak in solutions.

Sean: [00:33:32] Great advice. All right, last question I promise, and you can use one of the books that you’ve recommended earlier, but we always look for, like, the book that you’re reading now or you’re most likely to recommend or that you have recommended or given as a gift. Just kind of collecting data on people in our space, what you’re reading and what you’re up to so we can share that with our audience.

Kate: [00:33:52] Sure, I’d be happy to. I love me the book so I usually have a pretty full list of them that I’m reading. My favorite one on this topic is Lean Analytics. There are a few other quantitatively-focused books, most of them are about measuring usability, which is distinct from measuring the use or retention, and those might be helpful for specialty teams, but for a generalist mindset about how numbers can amplify and affect your work I think Lean Analytics by Alistair Croll and Benjamin Yoskovitz is a must buy. I give it to all the teams that I work with. I use it myself. I think that it’s great. The second book, which I would love to pump up a little bit, but full disclosure I was involved in its production, is a book by Laura Klein, Build Better Products, and for a generalist team that’s looking to really focus on growth and doing the right kinds of work instead of getting caught up in the trappings of interface details or, “is it pretty?” I think it’s a very powerful book. I provided the illustrations for it. Laura doesn’t sketch so I had a hand in that, but the reason that I participated is because I feel so strongly that she’s got a terrific take on how product teams can be more effective.

Sean: [00:35:01] All right. Well thank you for joining us. And thanks for participating the ITX UX Conference too, you were fantastic, we got great reviews for the conference.

Kate: [00:35:08] It was a fabulous event.

Sean: [00:35:09] Thank you.

Kate: [00:35:10] You all know how to throw a good event. I hope you do that one again because people should go.

Sean: [00:35:14] A little plug for the ARTISANworks in Rochester too, the venue was amazing, right. A neat place.

Joe: [00:35:18] Yeah.

Kate: [00:35:19] You know, having an event with a bunch of people for whom creative problem solving in a very creative visually enticing place was a real nice fit.

Joe: [00:35:30] Yeah it was nice. So for anyone who wants to follow your work, follow you, where can they find you? Anything you want to plug?

Kate: [00:35:36] Sure. I have a personal site, my little corner of the Internet, at intelleto I N T E L L E T O dot com. It’s a word that was coined by Michelangelo about the inherent intelligence of art in a material, so I really respond to that. And then I’m on Twitter right @KateRutter.

Joe: [00:35:53] Very cool. All right. Well this was a great discussion about metrics. We hope it was helpful. And thank you so much for joining us.

Kate: [00:36:00] It’s been a pleasure. Thank you.



08 / Planning & Prioritizing Product Roadmaps


Have you ever wondered what exactly it takes to create great software products? Those who spend even a little time in this space learn quickly that there is no wizard behind the curtain.

In this episode, hosts Sean and Joe speak with Rohini Pandhi, currently on the product team at Square, about her experiences developing and implementing a product roadmap – the path that connects a customer’s problems with a solution that drives their business forward. It’s not magic, Rohini says. There is no special sauce or magic potion. It’s a combination of talented, creative, hard-working people grinding through priorities, making sure they never lose sight of the customers’ destination along the journey.

Read our blog post

About Rohini

Rohini Pandhi is currently on the product team at Square. She has experiences building new products for startups, developing existing product lines for growth stage companies, and creating new innovation strategies for larger corporations. In her free time, Rohini also advises, invests, and advocates for tech startups, especially those with underrepresented entrepreneurs through a nonprofit she co-founded called Transparent Collective.

lm t 


Joe: [00:00:00] OK. So Sean, next episode, here we are. And today we’re going to be speaking with Rohini Pandhi and she’s a product manager at Square, and I met Rohini, we met back at the industry conference in Cleveland in September and we were sitting at a roundtable and Rohini started talking about different ways that, at Square, she manages her team’s road map and how they prioritize. And I found it very interesting and asked her if she’d join us today. So Rohini, hello.

Rohini: [00:00:27] Hello, thank you for having me on.

Joe: [00:00:29] So just to get us going, would you mind introducing yourself giving us some of your background, your role, how you got to be where you are today?

Rohini: [00:00:37] Sure. Sure. So I started off as a computer engineer who decided after graduation to not be in front of a computer programming anymore and so I went in to kind of like a technical consulting role after school. And I did that for a few years, became a road warrior, kind of got to have an expense account and travel around the country which was pretty fun. After that, I realized that I could either continue on that technical route or try something different. So I went back to school and did grad school to get my MBA and then realized there was a nice, cool, new role called Product Management that was the intersection between my technical and business sides. So I just kind of stumbled upon product as a career and fell in love with it. I worked at a lot of startups on the West Coast and then went to Square just a few years ago. And I work on the Square invoices product line for the company.

Sean: [00:01:36] Very cool. A quick question for you, so where did you go to get your MBA?

Rohini: [00:01:40] I went to the University of Chicago. So on the south side.

Sean: [00:01:44] Awesome.

Rohini: [00:01:44] Yeah.

Sean: [00:01:45] I went to the Simon School, University of Rochester….

Rohini: [00:01:48] Cool.

Sean: [00:01:49] …to get my MBA, and I’m curious to know what you think about that in the context of product development. Do you think the MBA was helpful? And if so, how do you think it was helpful?

Rohini: [00:01:59] Sure. I get a lot of questions from people who ask if they should be getting their MBA. And I think that’s such a, it’s a case by case question. It’s an expensive education, for sure. For me, it was really worthwhile because it helped me to break down the engineering mindset I had and think about things and solving problems in a different way. And so a lot of the frameworks I still use day-to-day. The frameworks that I learned in school I still use in our strategic planning or our competitive landscape comparisons and things like that. And so I think it’s a great way to get hands on experience, but it’s not for everyone.

Sean: [00:02:37] Oh, I agree and it can be a very expensive proposition.

Rohini: [00:02:40] Exactly.

Sean: [00:02:42] But how do you say no to anybody who’s looking for more education, right.

Rohini: [00:02:44] Yeah. I always say that if I won the lottery, I would just go back to school. I would just learn as much as I could on different topics. That would be so much fun.

Sean: [00:02:54] For me, I think that there’s a lot of quant tools in the MBA program, at least at the Simon School. So there’s a lot of statistical tools that I think were valuable, and that whole sort of approach to business was valuable.

Rohini: [00:03:07] Yeah. We had a great program that was more on the experiential learning at Chicago. And so you built your own business or you did competitions on something that you built yourself, and it just kind of demystified a bit of what entrepreneurship meant for me. And I think that entrepreneurial side is really important as you kind of take on product, especially early stage products.

Sean: [00:03:30] I couldn’t agree more with that. I do think having a little bit of knowledge around the language of business and accounting all those sorts of things can’t do anything but help you build better products. And to know how the business is thinking, like to know that at the end of the day, we always have to tie our success back to an ROI of some sort and profit.

Rohini: [00:03:50] Right, yes, it was great. I mean, I think, kind of like what you were saying, Sean, it was the first time for me to even see a balance sheet or an income statement. That was like the first time I’ve ever even looked at those things. And so to be able to speak that language now was incredibly helpful.

Sean: [00:04:06] Cool, so let’s jump into the meat.

Joe: [00:04:07] Yes. So today’s episode is really, you know, we’ve had some episodes around like vision and you know, how to think about products at a high level. Today is really about getting in the weeds. It’s very tactical, you know, what do we do with roadmaps? Just to start off with that, I think, you know, a lot of companies they define roadmap in a different way. So just for you guys, how do you differentiate between what a roadmap is and then a release plan or a project plan?

Rohini: [00:04:34] Yeah. I kind of think of all of these artifacts as different levels of detail, kind of sitting and zooming out. So for us at Square, or at least on our team, on invoices, we have the highest level, which is kind of our strategic plans, that include like the vision, the one- to three-year horizon outlook of what we’re going to go accomplish, our competitive and market landscapes and things like that, our customers jobs to be done at a high level. And so we have that plan. It almost looks like a business plan, but it’s our strategic view of where we want to take this product over the next few years. And then a level deeper we have our quarterly, it could also be multi-quarter, goals. We use OKR’s as the way to kind of outline that level of detail. And then we have our actual roadmap and that includes all of our projects that are in flight or in the backlog. So the roadmap would have line items for each workstrain that’s going on right now and then link out to even more details like our eng or jira tickets, our launch plans, things like that, that offer even more detail if want to get into the weeds. But most of our cross-functional team kind of looks at things at that roadmap level and when we expect timelines and milestones to be hit.

Joe: [00:05:57] Very cool, so at the company level they’re setting the vision strategically for the product. That’s a bird’s eye view, and then you get further and further down and you’re getting more and more specific. That’s how you’re looking at that.

Rohini: [00:06:07] Exactly. And really when you say company level, that is actually our team, our product team. We’re the ones making those goals and aims for everyone.

Joe: [00:06:16] Got it.

Rohini: [00:06:17] It really helps kind of tie the team and make sure that we’re all held accountable for the things that we’ve just said we are going to go accomplish. It’s not like some consultant that came in and said here’s what you should do for your business. You kind of are very attached to the goals that you set for yourself. And so it’s intentional and it’s by design to do it that way.

Joe: [00:06:35] Yeah, I mean so you’re essentially getting a bunch of autonomy and then you’re also accountable, though, at the end of the day.

Rohini: [00:06:40] Exactly. Exactly.

Joe: [00:06:41] So a lot of different roadmaps are out there in the world, you know, even companies like Facebook and whatnot, they put a roadmap out for the public to see. But like you were saying, it’s very very high level. How do you structure your roadmaps? Do you have timeframes on them? Is it certain chunks. Like how do you actually structure yours?

Rohini: [00:06:59] Yeah, I’ve seen a few different styles and I think everyone should kind of use the style that works for their company and their culture. I believe I even have a actual like Photoshop version of ours with some details cut out in a recent blog post that I put out on road mapping. So if anyone’s interested, it’s on Medium. You can just search for me and you’ll hopefully see it. But we use just a simple Gantt chart that we’ve built in Google Sheets, and it does have timelines by week. So I don’t get granular into the day because that seems too precise, but we do need week-by-week because we have a lot of cross-functional partners that rely on these release plans and large dates so that they can coordinate things from the marketing communications, when engineering starts rolling things out we want to put our support articles together, and so we want everyone who’s either an account manager or anyone who’s customer-facing, like account management or support or whoever else, to be able to see that and say, “oh this is why somebody has reached out to me because they’ve started seeing this feature that’s been released.”

Sean: [00:08:02] That’s great. You mentioned OKR’s in your first answer. So objectives key results, kind of like a KPI, key performance indicator, of some sort.

Rohini: [00:08:10] Right.

Sean: [00:08:11] So I like the fact that you’re tying that, and you also mentioned jobs to be done, which we’re big Clayton Christensen fans here as well.

Rohini: [00:08:18] Cool, yeah.

Sean: [00:08:18] So you tie the jobs to be done, and your OKR’s, I assume, are associated with your getting those jobs done for your customers. So I’d like to just pull on that OKR thing a little bit. So give me some examples of what you’re using, what you’re measuring, to know that you’re getting these things done.

Rohini: [00:08:34] Yeah. I mean, I’ll be completely transparent with you guys. I think that we aren’t doing the best job at our OKR level. I’m going to put together a proposal, in fact, like this week, to change our process for 2019 kind of going forward. Because we have a few things, that like you said, are evergreen. They are key jobs that our customers hire us for and we should continue driving those and making sure that we have the right kind of engagement and retention at those levels. But then there are certain amounts of risk taking that we should also be comfortable with. And I don’t want our OKR’s to get too stagnant and us getting complacent. So I also want to put in some objectives that are not just aggressive but something new that we haven’t really dug into and making sure that those are still accounted for and part and parcel of what we go do per quarter. Just allowing us a little bit more bandwidth to make those extra bets.

Sean: [00:09:28] Well I didn’t want to derail us too much from the roadmap conversation, but you mentioned that, so I wanted to pull on that for a little.

Rohini: [00:09:33] Yeah, hopefully I’ll have a better answer for you in terms of our process next year.

Sean: [00:09:38] No, I like where you’re heading. That’s really neat.

Joe: [00:09:40] Just to set up the next couple questions, do you use the concept of epics in your roadmaps?

Rohini: [00:09:45] So we use a little bit of both. So we have the Google Sheets, which has the high-level projects. And then within Jira, in our ticketing system, we have epics, and so each epic kind of lines up to a work stream in our Google Sheets roadmap. And so we just pretty much pull these things from our backlog in order to put them into the roadmap and then we break those down into individual tickets from an epic into Jira.

Joe: [00:10:09] Got it. And then do you size the epic somehow as a first step once it gets in there?

Rohini: [00:10:15] Yeah. We look at a few different things. We work in six week cycles just to break the quarter up a little bit more in order to have a bit of a tighter turning radius. And one of those weeks is pretty much set just for planning and our engineering planning. And so we look at the scope of work, we break it down into bigger tickets, and then we do a level of effort estimation that is really led by engineering to be able to help us say, “OK this piece is going to take this amount of time or this number of sizing,” and then we do that for all of the projects that are in there to say, “OK this can actually most likely get done within six weeks, or, “this is really a twelve week project that needs to be either rescoped or reprioritized.”.

Joe: [00:10:58] Just to keep digging here for the listeners… So t-shirt sizes? Do you use that, or how do you actually put a number or size on those?

Rohini: [00:11:05] So we actually first start off with t-shirt sizing and then we go into more details. So we use t-shirt sizing for our backlog and that includes a level of effort estimation. The other pieces of our backlog in terms of a priority framework, we use Adam Nash’s model which he kind of has a great blog article, again for anyone that’s interested it’s called the three buckets of prioritization. I’ve used that in the past and we use it the invoices team as well where we look at customer feedback as one of the buckets, metrics movers as a second bucket, and then what I usually refer to as foundational work is the third bucket. So it’s something that may not move a metric right away. No one’s asking for it just yet, but in order to get to where you want to be in two to three years you need to have this foundational or visionary work kind of completed. And so those are the three buckets. We added an extra one called level of effort. And so what we do is we prioritize against those three buckets and then we also look at the prioritization by level of effort. And we use t-shirt sizing for all of that. Again, because we’re all analytical nerds, we like to have, instead of small medium large, we go from extra small, which is a one, to extra large, which is a five. We put it in a spreadsheet and we actually start scoring things and having an average kind of pop out of that scoring system.

Sean: [00:12:25] Great. That sounds like, then you’re keeping some sort of measures too around the size of your backlog so you can control the flow of the requirements.

Rohini: [00:12:32] Exactly. Exactly. And so there’s always going to have to be a tradeoff. When you say something’s actually going to go into the que for building and so we look at it by that score a level of effort and how impactful it’ll be. And then we make a decision based on that.

Sean: [00:12:48] OK. So your impact measures are sort of, what bucket is in? Is it foundational, is it metric mover, is it? OK.

Rohini: [00:12:53] Exactly.

Sean: [00:12:54] Interesting.

Rohini: [00:12:54] Yeah.

Sean: [00:12:55] So we have this concept at I TX called the minimum viable backlog.

Rohini: [00:12:59] OK.

Sean: [00:13:00] We know we need to keep the backlog at least this full to keep the team flow super healthy, right.

Rohini: [00:13:06] Yeah. Gotcha.

Sean: [00:13:07] And then we have an optimal backlog, and then we’ve got our long-term backlog, which in theory should continue to grow forever, right. If we have great products, we know that there’s always something you could refactor, something you can improve, or…

Rohini: [00:13:17] I like that. That’s cool.

Sean: [00:13:18] Thank you. Do you guys measure the size of your backlog? Is that a factor?

Rohini: [00:13:23] We don’t, and the way that we’ve gotten around measuring our backlog is that our six week cycles are planned for well ahead of time. And so we know, kind of in that six weeks, how many folks we have that are going to be kind of working on backend server stuff, front end, mobile, we kind of know our capacity planning at that point, and then we say, “OK here are the big boulders that we want to put into the six weeks and here are maybe the rock-level improvements that we want to see,” everything else that’s kind of like pebbles and so forth are moved into individual sprints or are kind of cherry-picked by the engineers themselves because they either find something frustrating or they want to improve some specific part of their code base that they own. And so we tend to not put too many pebbles into the planning and if there are a bunch of, I’m going to keep going with this analogy, but if there are a bunch of pebbles, we’ll actually start grouping them into themes of work and making them about a rock size.

Rohini: [00:14:22] Interesting. That’s neat. Do you guys measure the quality of the backlog in any way, and if you do, what tools do you use for that?

Rohini: [00:14:28] What do you mean by the quality of the backlog?

Sean: [00:14:30] So, how much rework has to get done based upon the poor requirements, for example, or…

Rohini: [00:14:36] Oh, I see, I see.

Sean: [00:14:38] …defective user stories versus actual, you know, not in terms of software bugs. In the nature of software development, sometimes the user stories themselves can be defective.

Rohini: [00:14:46] True, true. Yes.

Sean: [00:14:48] Do you have any sort of a measurement system for that?

Rohini: [00:14:50] We don’t measure it in an analytical way, but what we do is when we go through our backlogs, we have folks from the design leads, the engineering leads, and the product leads all sitting in a room together on a regular basis. Every two weeks we actually go through the backlog, and we put anybody who has actually inputted an idea or has heard an idea from other cross-functional partners has to put their name behind this piece in the backlog that’s either the user story, use case, or whatever the definition of that project is. And so if we all sit in a room and we’re like, “what is this?” and that owner of whoever inputted it into the backlog cannot describe it in a full way, then that person has to go back to the drawing board and say, “in order to submit something like this I need to know all of these other requirements.” It doesn’t mean that a full user story at that point needs to be spec’ed out, but we have to understand what the problem at hand is. And so if we don’t understand the problem or we disagree on the problem it’s probably going to get a low score anyways. And in order for someone to champion it, they really have to be quite clear about what that backlog meaning is.

Sean: [00:16:00] Makes sense.

Joe: [00:16:01] So kind of along that line, with the model, do you see that certain requests that fit into one of the three buckets tend to win out more? Like do the customer requests supersede any of the other kind of requests?

Rohini: [00:16:13] Yeah, so actually go into even more detail on our model, what we do is we say that the customer requests could be a couple different things. So they could be things that we’re hearing from the support team, and so that would be like a, I guess a sub-bullet inside the customer bucket, or it could be things that we’re hearing from our community forum. And so both of these could be really important inputs for us. Either, like, our support threads are just getting way out of hand and the support team can’t manage everything, so for that quarter, for instance, let’s say our support requests are highest priority, we just need to decrease those. Then we’ll put that as a higher weight into the average that’s being calculated because we really want to push that metric down for the quarter. And a lot of this goes back again to the OKR’s, everything kind of feeds in and out of itself of what we’re trying to accomplish. Similarly with metrics movement, maybe our risk loss is like really important for us that quarter and so we will prioritize and make sure that we call out, “would it have any material impact on risk loss?” If not, it’ll get a lower score with all else being equal. Like, we’ve waited risk as being a big deal for us that quarter and so we’ll actually put weights around things. You kind of see the outcome as based on what weights you’ve all agreed to for either the quarter, or the year, or however you want to arrange it and collaborate with your own team. But it’s never like customer success will always win or the customer bucket will always win because you’ve already weighted it. Unless you weighted it where it’s like 90 percent customer bucket and 10 percent for everything else, that part should already be determined before you kind of go into this operational process.

Joe: [00:17:52] Got it. Got it. Makes sense. See you mentioned a meeting where it’s like the engineering leads, design leads, product leads. So just to kind of piggyback that, ballpark how many people work at Square or how many teams do you have now?

Rohini: [00:18:04] Oh, we have, I think we’re close to 3000 employees worldwide. But the invoices team is a lot smaller than that. I’m not sure if I’m allowed to say, I don’t know.

Joe: [00:18:14] No, that’s okay. So my question is basically, we look at a lot of big companies or enterprises or as companies grow they have to adjust with that to keep their teams aligned with what the vision is for the product and keeping them motivated and aware. So a typical kind of meeting or ceremony you hear them start to adopt is like a scrum of scrums. So my question is like, how do you all stay tied together with what you’re all each working on because I’m sure there’s times when the problems you’re solving or the requirements are going to kind of intersect and you have to work together or look out for that.

Rohini: [00:18:44] Yeah I think we do kind of use a scrum of scrums idea. So on the roadmap, we have a call out for the folks that are responsible from: there’s one person responsible from the engineering, design, and product side of the house. Sometimes we have more responsible parties on the engineering side because you need someone for each platform, like web mobile and server. But we have someone that’s kind of like supposed to be on top of every project from each discipline. And so those project teams tend to get together more frequently to either have stand up or sprint planning in and of themselves. And then we also have across all projects that are happening for this product planning weeks and retros in the classical sprint-style meetings for everyone. But usually it’s only the direct responsible parties that are speaking on behalf of the entire team because otherwise they would just get too long of meeting, too many updates, there’d be just so much going on. But we try to kind of create these smaller teams that then feed into the larger one, and then even within our product teams we then report out so that other product teams kind of know what’s happening, and then our VP’s C-level, C-suite folks can also see and connect the dots at their level.

Joe: [00:20:03] Very interesting. Can you speak a little bit more to that planning week and what that looks like and then the output of how you report out just in general what you have going on.

Rohini: [00:20:13] Yeah absolutely. So that planning week, it’s really focused for engineering. It’s just focused time where they’re not trying to build but also plan. And so it creates a little bit of a break in between our cycles where we say, “OK for this week you know the work that’s going to be happening, we all kind of understand what the customer problem is, we understand why we need to be doing it and why it’s a priority at this moment.” And then at that point, the product piece has been defined, or the business case has been defined, and then design has usually by that point also done several versions of exploratory work and have some mocks that are pretty close to complete. And all of that has already been work that’s been done with engineering involved. So even from the definition phase to the exploratory phase, engineering has always been involved, so that we can make sure that the stuff that we are talking about isn’t completely out of left field. But then at the point when there is a planning week, that means eng has a lot of the definitions and now we need to go into, “OK, what’s going to happen week over week, and what do we know versus what do we not know how to do just yet?” And so any work could be really just it’s a simple document where we just say, “OK, here’s the work, here’s what we need to get to, what are the things that we need to build in order to get to that point?” And the discussion is led by the engineering lead with inputs from the entire team. We rotate who the lead is, and we say, “OK, for this project, so and so is really interested in being the lead, so therefore they’re really just the point person that I can go to with any questions, that design can go to with any questions, and everyone just reporting status and updates to them.” And so they will go through and say, “OK for server work, here the end points we need to make sure we have, for the mobile team, when can we expect an Android version versus an iOS version?” And then they break down that problem into even more granular details, and week over week here’s what we should expect. If something isn’t a green light in a particular week that means it’s going to be moving into that next week. And then I can look at their breakdown document and say, “in week three, oh crap we’re really behind because everything’s red instead of green.” And I can either go into the Jira tickets to learn more about what just happened, or I can go talk to that engineer lead and say, “OK what are some of the tradeoffs, where can we either cut out requirements, change the scope, or what do we need to do to change the timeline?”.

Joe: [00:22:33] Awesome.

Rohini: [00:22:33] That was a really long answer to your question but I hope it covered everything.

Joe: [00:22:37] I think it’s great information because people are doing their work day to day all the time and they sometimes don’t get to hear other perspectives of how other companies work whether they’re in Silicon Valley or not. So it’s really good to just hear about how the day to day happens, like how does this software actually get made. It’s really not magic. There’s no special sauce. It’s just grinding through and prioritizing and making sure that you’re on top of everything.

Rohini: [00:22:59] Yeah and I think what we’ve learned is, like you said there’s no special sauce in this, and a lot of it is just operational processes that either will work for us or won’t. And then we’ll throw them out if they don’t and we’ll try something else. We try not to get too heavy-handed on the process, but communication, it’s an n-squared problem, right, with n being the number of people you have on the team.

Sean: [00:23:19] Of course.

Rohini: [00:23:20] It’s just exponential and we need to make sure that we’re all kind of aligned. And so this works for us, but I love talking to other product folks on what’s worked for them in the past and what hasn’t, just to keep our processes as lean as possible but also affording the structure with some of the benefits of creativity within that structure.

Sean: [00:23:40] Yeah I have an opinion around that. So process itself, it’s a word that generally means restrictive, right. Like you have to follow this process. Actually in software, if we’ve learned nothing else over the last few decades, it’s that we need to give teams a certain degree of flexibility to produce the best possible products. We know that. So it’s better if we frame what we would normally call a process as a best practice.

Rohini: [00:24:02] Love that.

Sean: [00:24:02] These are the best practices as we know them today, that doesn’t mean we’re not open to changing them, but this is what’s helped us provide predictable results over the years. These are the best practices that we know, but we should always be open to adjusting them and experimenting with those just like we do with the features of our software products, right.

Rohini: [00:24:20] Yeah, I love that. I think the words, matter language matters, and so just even changing the framework. Like even when you said that, in my mind, it just changed how I thought about the ways that we do our process and then just changing it to best practices means rules that are allowed to be broken and they should be if something changes.

Sean: [00:24:37] Right, it doesn’t mean you shouldn’t be accountable for that. I mean if you’re going to make a change to a best practice, you should have a good reason and it should be treated as an experiment.

Rohini: [00:24:44] Yes. Yeah that’s very cool. All right.

Sean: [00:24:47] Next question we have for you is around motivation, and the product that you’re building is an invoicing product.

Rohini: [00:24:54] It is.

Sean: [00:24:54] It’s not so sexy, no offense. So how do you get the team motivated to produce the best possible product?

Rohini: [00:25:03] You know, I guess I’m pretty fortunate in that the team that I work with is very passionate about this. I think that it doesn’t come with a certain degree of a product mindset in all of the disciplines. And what I mean by that is like, having some user empathy or customer empathy around what we do. So for the folks that are already pretty jazzed about you know providing payments in an invoicing format, I love bringing them along in customer interviews, in onsite visits, that kind of stuff; because it just puts a face or a story to the problem. And it still affects me even now after having done however many thousands of hours of interviews over the years. Just when I talk to someone and they are kind of holding back tears on a phone call where they tell me that we offered them capital financing from Square at the very moment that they weren’t able to continue their business, and because of our financing, it extended their cash flow for another 30 or 60 days when their invoice was getting paid. And that was just like awe-inspiring to see how many small businesses and customers we have who just rely on this type of suite of products that all of Square has in order to make a living, but then also achieve a lot of the professional dreams and goals that they even set for themselves where they’re starting something on their own or they’re trying to build something for their community. And it’s just a really fun aspect to be a part of that story with them.

Joe: [00:26:33] I think that’s so important, what you just mentioned about telling stories to your team because the first thing someone might think when they hear invoicing is, “oh, you’re just sending receipts,” but really you’re allowing them to conduct business and conduct it well and it’s taking something that’s typically a nightmare of a process and making it easy. And that’s so important to give your teams, and this is for all of the product managers out there, newer to it or been doing it awhile. Don’t forget to do that. Do it regularly, like they should be hearing and seeing all that. Like you said, take them along with you sometimes. That’s amazing.

Sean: [00:27:06] That was the best answer ever.

Sean: [00:27:11] Hahah, oh good.

Sean: [00:27:11] And it supports this ongoing theory that I have and that we push here, in that any compelling vision has got to connect the people that are achieving the vision to the people that we’re solving problems for. You hit the nail on the head and I think taking your people to meet with real customers and making sure that they really understand the problems that they’re solving for other people, that’s what underlies all motivation. There’s a lot of science to prove that, and we talk about that stuff a lot on the podcast, but you just hit the nail on the head.

Rohini: [00:27:40] Yeah it’s eye opening. I mean for all of us, too. I still love those customer conversations to this day. It’s kind of the best thing about being a product manager is having those on a regular basis. And then also having the team realize after two or three conversations that there are definite patterns that already start emerging of, “oh this would be something cool to do” versus, “we absolutely need these types of things.” And for us, the thing that I constantly hear is about faster payments and your cash flow piece, right. It’s, “if I don’t get this money like immediately or yesterday, I don’t have enough to fund the next project that I want to go do, and if I don’t have that…” Cash flow just becomes this recurring nightmare for our sellers, and so kind of realizing that we’re not just an invoicing product but the payments piece is so crucial. And how we look at payments, it’s not just if it is a pretty looking invoice, but all of the key pieces right underneath that. I think Square it does a really good job of being a little bit like an iceberg where you still see the top piece of that iceberg and it’s very simple and clean it to the point and the designs are really well done. I think our product engineering and product design team are just incredible at what they do because they take all that complexity that lies underneath the water and just hide that from a lot of our customers that don’t need to know, how does a payment actually happen? And it hides a lot of the financing tools that we need in order to be able to offer this type of loans to our customers. And so the ability to take something really complex and messy and tough to handle and make that really beautiful, simple, elegant, and remarkable at the end is a lot of work. And if you can highlight to your team why that piece is so important, why hiding it is very important, why people care about the key pieces of your value props and how to make those even better, that becomes your differentiator and your source of lifelong customers.

Sean: [00:29:25] Again you hit the nail on the head. Just a little side note, I’ve been a Square customer since the beginning of time.

Rohini: [00:29:32] Oh no way. OK, cool.

Sean: [00:29:32] So I’ve got a little side hustle. I started buying and selling antique jewelry as a very young guy at like 16 years old. The first credit card I took was from Square and I’ve been a huge fan ever since. I was such a big fan that in my little local antique community I got a ton of people to sign up very early on.

Rohini: [00:29:51] So cool.

Sean: [00:29:51] And again, I’ve always believed that Square has found success because of their focus on the user experience and putting their customers first. You guys certainly have lived that set of values.

Rohini: [00:30:02] Oh that’s great. I’m so happy to hear that. That’s fantastic. The product side of me wants to hear all of your feedback now but we don’t have time for all of that on this podcast, but at some point I will be reaching out to hear what’s good, what’s bad, what’s working, what’s not.

Sean: [00:30:16] To be honest it’s a side hustle and I don’t do it a whole lot anymore. If I do set up at shows, it’s only for charities, it’s for nonprofits and I bring my kids along.

Rohini: [00:30:25] Oh cool.

Sean: [00:30:26] When my son was literally 5 years old he could process a credit card and take payment for a ring. So…

Rohini: [00:30:33] That’s great. That’s awesome. We should have him on some of the commercials.

Sean: [00:30:39] He’s 18 now, so…

Rohini: [00:30:41] Oh, OK.

Joe: [00:30:43] All right, so we’re talking about tying your teams to your customers a lot. So I have to ask, do you use user personas in any way, shape, or form. And if so, how deep do you go using them, meaning, you know, some people use them in their road maps to prioritize, all the way down to obviously as a persona name in their stories.

Rohini: [00:31:03] We don’t use the traditional sense of customer personas because we are, as a company, kind of focused on jobs to be done and so we have jobs-based language that kind of breaks things down into more detail. We don’t say like, “twenty-five percent of our roadmap needs to fit this job,” but it’s more of either how important or how impactful is it on kind of our three buckets checklist based on all of the different criteria that we have listed out. But we will say, like, “the reason that we’re doing Project X is because it helps our customers accomplish this, and the customers that need this hire Square in order to do X Y and Z.” And so we do use a lot of the jobs language. We also use, I sometimes do deep dives into week-long interviews of just hiring jobs or just our firing jobs. The reasons that people choose Square invoices, the reason people leave Square invoices, and really understanding, “what kind of buckets are there of different jobs that we satisfy or we fail at for our customers?” And so we reorganize things. It’s not just bottoms up of, “here are all the tickets, here’s the backlog, here’s the roadmap.” It’s also looking at things and challenging ourselves from top down, saying, “OK, what’s out there in the competitive landscape, what jobs do we satisfy versus a PayPal or a Quickbooks? What jobs do those tools satisfy and why would you choose one versus the other?” And really understanding that core need. And then also the acquisition piece. It’s both for product and it also definitely helps product marketing.

Joe: [00:32:42] Yeah a lot of the roadmaps that we’re trying to push now don’t include features, they include problems to solve.

Rohini: [00:32:47] Yes. Yeah. So that’s what we’re tending to get drawn towards and move towards.

Joe: [00:32:51] So you mentioned the iceberg example before, and we use that analogy a lot here too. So I’m curious, when you’re working on any kind of a product at Square, how do you factor in the non-functional requirements, the NFR’s, such as security, performance, tech debt, compliance? How do you build those into your roadmap and backlog ultimately?

Rohini: [00:33:14] Those usually go into our PRD’s, our product requirements documents, to make sure that things are kind of at least accounted for or unknown to our teams that care about the security and compliance pieces. So we will actually kind of outline for our legal, or compliance, or our risk team, “hey, we’re gonna go do this work, we’re going to build it in this way.” That’s kind of a known pattern around Square, and we want to just make sure that their eyes are on it so that they know that this is happening. If they have any issues with the design or how we’re going to go build it, it’s known way before the project gets kicked off. But we don’t necessarily have a roadmap item that says, “make sure all of the tech debt that we’ve accumulated for all of compliance is done by this date,” unless there’s some sort of like, new GDPR-type of law that’s in place and there is an actual date. We try to build all of that into each product work stream that we work on. We don’t want any holes in our security with any new feature, we don’t want any legal issues with features that don’t get accounted for for six months, or something like that. So we try to put those part-and-parcel with the work that we do.

Sean: [00:34:26] That makes perfect sense. Those are non-negotiables.

Rohini: [00:34:28] Exactly.

Sean: [00:34:29] Last question. We ask all of our guests. What is one book that you found powerful enough that you recommended to your team or to other people in this industry?

Rohini: [00:34:39] So OK. The one that I really like and I’ve gotten as a gift. It’s a really small, short book but it’s so fantastic. It’s called Steal Like An Artist, Austin Kleon, K L E O N, it’s just a great way to remember how to break some rules and kind of get out of the box thinking, like we were talking about, how process can help, but then also, you know, breaking some of the paradigms and working like an artist in the things that you do day-to-day. Again, it’s very short, very small, it’s actually a small, physically small book, but I really enjoy it, It’s just fun to read and I often find myself kind of going back to it and trying to remember some of the quotes that he puts in the book just to kind of push my own thinking. I mean it’s not really like a product or business book so I am also starting to read Principles by Ray Dalio and just finished up Sapiens by Yuval Noah Harari. Both of those are just like wonderful, so far on Principles, just a great framework that Ray kind of puts together in his past background, but they’re both like 700 page books and I love reading real books. I don’t have a Kindle version.

Sean: [00:35:54] I’m the same way.

Rohini: [00:35:54] I know. There’s something really nice about the tactile version of reading.

Sean: [00:35:58] And I just read a statistic, every form of media is on rapid decline, right, from obviously music and video all that stuff is dying, it’s like everything’s being streamed but books are still expected to continue to grow at 1 to 2 percent forever into the future and they’re still pretty solid.

Rohini: [00:36:16] Oh, I love it. Yeah. I really enjoy having an actual, like that’s the first hardware that I’ve had. So it’s just really nice to have a book. But I will warn the listeners here, those are like really big ones. And I have accidentally fallen asleep with one and had to wake up because it was hurting, I was on my chest and stopping my breathing. So maybe get the Kindle version or the audio version, but they’re both fantastic books.

Sean: [00:36:43] Cool. Well I definitely like the first one that you mentioned, the idea of stealing from artists. I think it was, what was that guy’s name? Bernard of Chartres, I think. “We are all standing on the shoulders of giants,” right. All of our discoveries, there’s really nothing new, we’re putting things together in new combinations, but we’re all standing on the shoulders of giants.

Rohini: [00:37:03] Exactly, yes. And it’s really about that. There’s very few and rare true innovations. It’s about how do you reapply some known things. One of my professors from my MBA once said, “well stolen is half done,” and I still use that to this day. It’s so true. Well stolen is half done.

Joe: [00:37:23] That’s a good one.

Sean: [00:37:26] Awesome.

Joe: [00:37:27] Well, Rohini thank you for joining us today. It was amazing to hear and get a glimpse into how Square works and how you specifically work with your team and manage roadmaps, and backlogs, and motivation. So thank you so much.

Rohini: [00:37:39] Thank you both very much. I appreciate it.

Sean: [00:37:42] Super impressed. Very good interview, thank you so much for your time, and taking time out of your day to join us for this.

Joe: [00:37:46] So you mentioned Medium. Where else people find you if they want to connect?

Rohini: [00:37:51] I’m also on Twitter. It’s @RohiniP. And I’m sure, I’m on Linkedin, I’m the only Rohini that’s at Square. So you should be able to find me.

Sean: [00:38:01] Outstanding.

Joe: [00:38:02] Alrighty, well thank you again so much.

Rohini: [00:38:04] Awesome, thank you.

Joe: [00:38:06] All right. We’ll talk to you later.

Rohini: [00:38:07] OK.


07 / Resisting Experience Rot


“Why, daddy,” says the curious toddler. “Why?”

No parent has yet found a way to escape this exasperating line of questioning. We should be grateful. For as frustrating as the interrogation becomes, its purity of purpose cannot be denied. Focus on the why. The honest, unrelenting desire to understand requires us to rethink, reconsider, and clarify motives. In a similar way, UX designers and product managers can employ this technique to resist the impacts of “experience rot.”

In this episode, hosts Sean and Joe speak with UX design authority Jared M. Spool about the experience rot paradox, a phenomenon that arises when product owners perceive new features in a way that reinforces the “if one is good, more is better” philosophy.

Read our blog post

About Jared

Jared M. Spool is a Maker of Awesomeness at Center Centre/UIE. Center Centre is the school he started with Leslie Jensen-Inman to create industry-ready User Experience Designers. UIE is Center Centre’s professional development arm, dedicated to understanding what it takes for organizations to produce competitively great products and services. In the 39 years he’s been in the tech field, he’s worked with hundreds of organizations, written two books, published hundreds of articles and podcasts, and tours the world speaking to audiences everywhere. When he can, he does his laundry in Andover, Massachusetts.



Joe: [00:00:29] All right so on today’s episode, we have Jared Spool, and I know a lot of our audience is probably familiar with Jared, we are. We know your work very well. We follow it. But for those who are not, can you please introduce yourself, talk about where you work, what you do, what’s your role overall in the world?

Jared: [00:00:48] Well I’m an internet sensation and teen heartthrob. I basically research what companies need to do to deliver better products and services and I help those companies and the designers within them become better at delivering products and services.

Joe: [00:01:14] Very cool. So you know, we’ve talked to you a lot recently and you’ve written and spoken on many many topics but one that’s kind of near and dear to our hearts lately with the work we do with our clients and all the conversations we hear about and have is this concept of experience rot. Can you define what that is?

Jared: [00:01:32] Sure. It has to do with what happens when you indiscriminately add features, right. There is this notion that a product with more features is better than a product with less features and that’s sort of reinforced by the product management behavior of sort of defining roadmaps in terms of features. So everything in the organization is sort of rewarded by features. And then the other sort of thing that happens is, you do some sort of competitive analysis and almost always that’s, “well what features do they have that we don’t have? And so now, okay, now we have a bigger list of features.”.

Jared: [00:02:14] So we start adding all these features and we add all these features, and over time, at first it’s a good thing, right. You know, we’re making the product fuller, we’re making it more robust where it can do more things, and people are like, “yeah, we should buy the thing that does more things.” But over time, you sort of hit this point where every new feature you’re adding starts to make the product more complex. And not just complex from a user experience perspective, well from a user interface perspective, of “well, there are more menu options and there are more settings in the program,” but all those things also increase complexity internally because every new feature is a new piece of code that has to work with the old pieces of code.

Jared: [00:03:03] And you get to this point where the design is you’re adding features that someone has put a big bag of money on the table and said, “you know if you build this one feature that I know you know only we need and nobody else needs, we will give you this big bag of money.” And it’s very tempting to say, “well yeah, sure, and we’ll just put an option in the settings that lets people turn that feature around or turn it off and most people have it off and that’s fine.” But what’s now happened is we’ve created this massive piece of complexity. And if I just want to add a new feature, I have to make sure it doesn’t break all the other features, and if the other features are controlled by settings, I then have to try all the different settings to get those features to turn on and then to turn off.

Jared: [00:03:56] And I was with a team and they had just hired a designer, this was a few years back, and they had never had a mobile version of their app, and they hired a designer and he was great. And he did this brilliant design for this mobile app. And the developers coded it up really fast because they were really motivated by it. They wanted to get this thing out. So they sat down, and in less than a week they had this thing running. And everybody was really excited. And then it got turned over to their QA team, and the QA team did with the QA team does, and they started turning on options. And it turned out there were 14 options in the overall ecosystem that affected this mobile app. And every time they turned one of those on, the design broke, because the designer, who was new to the company, didn’t know about those 14 options. Some of them were legacy for a long time and there was only one or two customers who used them, but the app would break for those one or two customers and their users. And when you we did the math, it actually comes out to 14 factorial. And 14 factorial means there were eighty-seven million combinations that QA had to test, right. So not only do we have this complexity of having really hard navigation and really hard settings and all this stuff, but now we have code complexity. And code gets so complex that people don’t want to touch it because they’re afraid of breaking it. And now you’ve lost your competitive advantage. All the competitive advantage you’ve had from adding all these features is now negated by the fact that all these features have made the product really complex and you can’t add code easily and your competitors can add things because they don’t have this. That’s experience rot, it’s when features increase, complexity increases, and the experience goes down.

Sean: [00:06:06] That’s great. In our world, we refer to technical debt. Experience rot’s kind of like experience debt, like you create these things you’re going to have to pay for in the future and they add up.

Jared: [00:06:19] Yeah.

Sean: [00:06:20] I like how you use the factorial as a mathematic structure. That’s exactly how they add up, right.

Jared: [00:06:25] Yeah, yeah. I mean they don’t add up, they increase exponentially. And that’s the problem. You walk into this thing, and the problem with exponential growth is it doesn’t seem like a lot at the beginning but suddenly it’s overwhelming. And if you’re not prepared for that, if you’re not watching for that, you wake up one morning and you’ve just got crazy and you don’t know how to control it anymore.

Sean: [00:07:03] Hahahaha, you’ve got crazy, I like that. So I have a question for you along those lines. So one of the big problems we often run into in our world is, there’s always that executive, let’s just call him the executive, who is super competent, right, and wields some sort of power in an organization. It can happen in clients, it can happen within our own company, and they always seem to know exactly what the user wants. So you try to argue, “well, let’s go get some data on that.” But it’s always a challenging personality. I’m just curious, how do you handle people like that when that when that occurs?

Jared: [00:07:42] Yeah. So the secret of this is in the pre-work, right. In pre-wiring the data. In my experience, if you get to the point where someone doesn’t have the data they need but they believe they have the data they need, it’s really hard to demonstrate to them that they actually don’t have the data they need. You’re better off showing them upfront what it is. So identifying who those potential folks could be who will at some point emerge as being the self-proclaimed expert on the user, and start by either showing them that they don’t know as much as they think they do, or in fact, proving that they are in fact a self-proclaimed expert. So that means building the research in earlier than when they come back and say, “well this is what the user needs.” And so this means structuring the project work so that the first thing you do is research. And I’ve seen this done a whole bunch of different ways and probably the most successful is to actually break the project work into phases and have a concrete discovery phase where everything is treated like a hypothesis.

Joe: [00:08:58] Yes, awesome.

Jared: [00:08:58] And it’s like, “well everything we believe about the user in this problem we’re going to go out and really explore the problem and focus on the problem, not the solution, but the problem.” If we’re thinking of solutions, we are only thinking of solutions as ways to generate our knowledge of the problem. So it’s like, “well, we think people need to have a way to export data. Well, let’s go look at all the places where they export data and figure out what they do with it and figure out what triggers that so that we can make sure that we’ve got the bases covered for this export data feature that we actually believe people want because every so often someone asks for it.” And you know, maybe someone’s come in and put a big bag of money on the table and said, you know, “if we had a way to export our data we’d be great.”.

[00:09:43] But without knowing how to export the data or what export the data for, the odds of us doing a good job are really bad. So we use that as leverage to do the research and say, “we really need to understand the problem that triggers the need for exporting data. Because it turns out, maybe the reason they’re exporting data is because they don’t realize we have that functionality in our product already, they’re thinking they have to move it into another product, when in fact we already do what they’re going to do in Excel, for example. Or maybe they’re doing something in Excel that our products should do, or maybe they’re exporting it into another app and we shouldn’t use an intermediate data file format that requires the saving of a file and then the loading of a file, we should just connect to their database in the other app and directly share data and let things move back and forth so that if they make a change in the other app, it changes in our system too,” right.

Jared: [00:10:43] So there are all these different things that, maybe exporting data isn’t the right solution, but one of the problems is that customers in general, humans in general, when we ask something of somebody else we tend to ask in terms of solutions, not problems. We don’t say, “hey I really need to be able to do X is there a way I can do X,” except when we say, “X is the solution I need.” Right. So we don’t talk in terms of the problem, we don’t talk in the sense of, “I somehow have to get this information to all my customers. How do I get this information our customers? Well I’m going to bring it into Excel, put it there, then I’m going to create a template in Word, and then I’m going to do a mail merge in Word, and then I’m going to import that mail merge in Word into email, and then I’m gonna send it to my customers.” Well that’s a nice solution, but that’s not the best solution for your your problem probably. So what we want to do is really start the project by identifying the problems and then making sure that all those potential people who are going to be self-proclaimed experts down the road are getting all that information. We are pre-wiring them with, “hey guess what we learned about the problem, this is the problem customers have, and this is the problem customers have, and we’re exploring this solution and this solution and this solution,” and that point, if they have an idea, it’s within the framework of the research you’ve already done or it’s out of their framework, but you can say, “well hey we should extend our research to figure out if this would actually solve the problem, let’s do that before we invest a lot in it and then suddenly realize it doesn’t solve the problem.”

Sean: [00:12:29] Got it.

Jared: [00:12:29] I don’t know if that helps you with your client.

Sean: [00:12:30] No, that helps, that helps. One of the quotes that I wrote down in my notes when you spoke at ITX UX was, “great designers don’t fall in love with their solutions, great designers fall in love with the problems that they solve.”

Jared: [00:12:44] Exactly. Exactly. You actually said it better than I think I said it. The reality is that we are enamored by solutions because solutions feel like forward movement. If you’re just talking about the problem, people get frustrated, “we’re always just talking about the problem, we need to move on, we need to get to a solution,” right. And we resist this desire to spend more time really understanding the problem. But that always, always pays off. And if we can start to see solutions as just a better way to understand the problem. You know, “I proposed the solution that we have a one click file export, but then I want to go back and watch people use it. And what I may find out is it really didn’t make things much easier. OK. And it created these other problems, like they don’t know where the file got exported to, you know it’s just export and it says, OK, you’re done. It’s like, where did it put it?” So we have to understand. And this is why processes like Lean UX and design sprints and all these sort of up-front problem understanding techniques have really come into fashion in the last couple of years because we’ve gotten really good at once we understand the problem knowing what to do. But we find out that we still fail at delivering great results, not because we didn’t build the thing right, but because we weren’t building the right thing.

Joe: [00:14:35] That really hits home for us a lot. You know I’m thinking about what you’ve been talking about with companies being so tempted to be in a race to add more features and as many features as their competitors or add the features that clients will give them money for. So is this a culture problem for companies, or how can they be comfortable with saying no more? To keep a simple product.

Jared: [00:14:59] Well, I mean it is a culture problem in that the best definition I ever heard of culture is, it’s the things we keep saying we do until we actually do them. And so it’s a culture problem in that we seem to think it’s acceptable to take on work that we should have said no to that we didn’t understand. And so you have to you have to establish a principle and then you have to follow the principle that says, “we’re only going to commit to a solution when we feel like we actually understand that the solution will make life better for somebody, and here’s how we’re gonna tell that.” And as with any principle you then have to iterate over it to understand, “well what does know that the solution is going to make things better? How do we tell that? What are our techniques for doing that? What are the rules we use?”.

Jared: [00:15:58] You know, it’s sort of like how laws work in that do you have a law that says, “every person who is arrested must be read their rights,” and then you spend 40 years defining every word in that sentence as to, “what does it mean to have been read their rights? What do rights mean? You know, who are the people we’re talking about? Are there some people in some situations where that doesn’t apply?” And you sort of really hone in on when does the principle apply and when does the principle not apply. You know, if you have real principles, you’re doing that all the time. Otherwise what you have is just lip service of things you put on the wall. So if you’re going to have a principle that says, “we’re not to commit to a solution until we know for sure that that solution is going to make the users’ world better.” Then we have to start taking that apart and understanding how do we know that, what do we mean by solution, what do we mean by better, what do we mean by users? And really sort of nail that down. When do we apply this principle and when do we not apply this approach. And it’s a shift, right. Because when companies are very young or organizations are very young or teams are very young, they’re sort of in this anything-for-a-buck stage where it’s like, “hey someone’s gonna pay us for this, we should just do it, right, they obviously find value in it, they’re offering to pay us, let’s just make it happen.”.

Jared: [00:17:38] And in the early stages, that’s what you have to do to figure out what you are and who you are. But at some point, you’re smarter than the people who are asking you, and you’re like, “yeah you really don’t want me to do that.” We don’t go into our car mechanic and say… well, I mean, some people actually do, right. I’ve sat in a car mechanic’s office waiting for my car to be fixed listening to the conversations, and people come in and they say, “I really want this tire on my car.” And the mechanic who’s an expert in tires, it’s a tire shop that I go to, will say, “actually that’s not the right tire for your car.” “No it’s a good price. It’s really good…” And the person came in with a solution and it’s like yeah this is not going to end well, right. This is the wrong tire for your car. You’re gonna spend more money and potentially put somebody’s life at risk, you know. And a really good mechanic would say, “yeah I can’t put that tire on your car.” They may lose the customer over it, but at the end of the day they’re not going to do something that’s gonna put the customer at risk or give them a bad deal. Somebody else might and they know full well that they could walk into a shop down the street and that person will be just as happy to put whatever tire on the car that the customer asked for, but I go to the one where they say no. And then I walk in and I say, “hey, I don’t know what’s wrong with my car. Help me figure it out.” And they say, “actually there’s not a whole lot wrong with your car. It’s just this little rattling thing and we put a tie clip on it because it’s a 10 year old car and it doesn’t rattle anymore.” I’m like, “excellent, thanks.”

Joe: [00:19:15] Yeah, it builds trust and reputation.

Jared: [00:19:17] Right.

Joe: [00:19:18] Yeah. So I’ve kind of heard one way to avoid experience rot is collect a lot of feedback, have a lot of ideas, a lot of problems to solve in your road map so you have to be more selective. Do you think that’s a true statement or an accurate way to look at it as well?

Jared: [00:19:35] Yeah, so a big way to feed to to fix experience rot is to shift from features to solving problems, right. Because every feature theoretically solves a problem. If you don’t know what problem it’s solving, A, why are you building it? B, how will you build it well? So you need to understand what problem it’s solving. I mean people want to export data. OK. No one wakes up in the morning and says, “finally, it’s the day I’ve been waiting my entire life, I’m going to export data,” right. That just doesn’t happen. So what we want is to understand, why do they want to export data? Right. Of course, they should be able get data out of the app, there is no doubt about it. But what are they going to do with it when they get it out of the app? Is it just a backup? Because maybe we shouldn’t call it export data. Maybe we should call it backup data. Is it something that’s getting used in another app because that app has features we don’t have or has features we have but nobody knows about?

Jared: [00:20:43] So we need to understand what the actual root cause problem that’s being solved is. And so if we restate export data as “port this into the invoicing software so people can invoice against it.” That helps us really quite a lot. But that’s still sort of a solution, let’s state it in the problem, which is customers need to invoice when the project’s done. OK. So that’s an interesting problem, and then we can extend it and say, “well maybe the real problem is that this is billable work and we weren’t treating it as billable work. So we need to start treating the work that people do in our app as billable because people are trying to invoice against it and other things like that.” And so now that’s really interesting because maybe that changes the whole framing of how we present things in the app like which moments are billable and which ones aren’t, can we give them control over what is billable and what isn’t, what gets what gets put into there, are there things like taxes that need to be applied, is there some sort of service charge, are there add on charges? Right. And so now we can explore the problem better and we’ve moved away from exporting data which inevitably will be imported into a billing system to something much bigger and richer for the customer. And maybe they don’t need to go into the billing system at all, maybe we just need to flag that an invoice has been sent, tell the billing software that’s been done, and give them a copy of it. And as far as they’re concerned, everything is happy with the bookkeeping system and they never had to fire it up. So it’s understanding the problem being solved as we put it on the roadmap that allows us to have this real detailed sense as to, well what does it really mean to produce an outcome that really delights customers and isn’t just reactive: “I need a way to export the data.”

Joe: [00:23:04] Yeah, one thing that’s cool about the problem-based roadmap rather than the feature-based one is this is something, you know, for listeners, you can go back today, right now, look at your road map and figure out, “is this problem-based or feature-based?” And start changing that essentially right away.

Jared: [00:23:18] Yeah, yeah. And the big thing that you’ll find is that half the things on there, or some percentage of the things on there, you actually are having trouble figuring out what problem is solved. Somebody asked for it and you didn’t really inquire, you just said, “oh that’s a good idea,” and you put it on the thing and then you walked away from it and now you’re coming back to it and you’re like, “I’m suddenly realizing I actually have no idea why the customer wants this.” Or maybe you’ve heard it from four different customers and they all ask for exporting data and you’re assuming they’re asking for the same thing. But if you went and talked to each of those four customers you’d find out that they’re actually trying to solve completely different problems with it and really need completely different solutions.

Sean: [00:24:02] That’s great. All right so I’m going to step back a little bit back to your mechanic story and your answer to Joe’s question about saying no. I’m going to try to sum it up in a shorter sentence. So what you’re saying around feature decisions is the more data that you have and the better you are communicating it, the easier it is to say no. So you always have to go back to that research question and the better you are at really understanding the problems, the easier it is to say, “no, this is what we’re gonna do and this is why.”

Jared: [00:24:35] Yeah. And I think on top of that you need a basic principle, which is, “we need to protect ourselves from future experience rot,” which is basically saying, “we need to make sure that we don’t exponentially increase the complexity of what we’re building.”.

Sean: [00:24:54] Right. And to pull on a couple of other things that you said that we definitely espouse here in our work and that is that you have to have a vision for your product that is very explicit about how you’re solving problems for the humans that are using your products. Very explicit. And the better you know what those problems are, the better you can come up with the solutions to solve those problems. And the more robust your vision is, the easier it is to get your team aligned around solving those problems and that vision and the easier it is to build better products because you’re all aligned, committed, and confident that you’re going to get there.

Jared: [00:25:34] Right. Let’s look at the example of an airline app. So this is an app, produced by an airline for their travelers to book flights, to get their boarding pass, to find out where the restaurants are in the airport, to track their baggage, and they’re always coming up with new features and they’re adding it in. So you know, now when I check my bag, it’s added to my itinerary record. Suddenly I get a notification that says, “your bag has been put on the plane. Your bag has been taken off the plane. Your bag is in a city that you’re in.” And so that’s all great and that’s useful, that’s that’s a really useful feature. And you keep adding these features. And it’s like, “oh, well we should add features about, you know, working with our partner airlines so that when you travel on one of the other airlines in the network, well, you can do it through our app instead of their app, but their data is a little different than our data, so we have to change the way we present the data.” So you’ve got all this sort of more and more added complexity. And one of the things that they often do is they just create new menu options. So you bring up the app, and the first thing it does is say “what do you want to do? Do you want your boarding pass? Do you want to check in for another flight? Do you want change your seat? Do you want to find food? Do you want to check your bag?” And it just gives you this list of things. And that sounds great except that, you know, “I’m in the TSA line and I’m running late for my flight and I’m fumbling for my license and my phone. I need my boarding pass on my phone, and now I have to bring up the app, and the app comes up, it and takes a while to start. Then I have to bring up the screen and I realize that I don’t have the boarding pass loaded into the wallet because that’s an extra step and I forgot to do that step. And so now I have to fire up the app and, you know, I’ve got a line of people behind me and I’m running late for my flight and I have to find the right menu and I have to get the right flight. I have to click on the boarding pass for this flight. And then it comes up.” Or you know, we have the technology. It could know I’m standing at TSA because that’s the Wi-Fi transmitter I’m most connected to at that moment or the cell tower that I’m talking to, and it could know at that moment that that I haven’t checked in to TSA because I haven’t gone through that process, and it could just bring the boarding pass right up. In fact, while I’m in the airport, why isn’t the boarding pass on the home screen” That’s the type of thinking, but you don’t want the boarding pass on the home screen all the time. We only want it when I’m at the airport. And the device knows I’m at the airport. It can see that I’m at the airport. So the app can see that I’m at the airport. So why is it not just giving me my boarding pass until I’ve used it to actually get on the flight? And then it should come up with something different like how I connect to the Wi-Fi, or how long the flight will be, or the status of the flight, right. And if we thought about the whole experience we would actually change how the app reacts based on that without me having to explicitly look for the menu option that gets me there. And that immediately reduces complexity because it’s anticipating my needs.

Sean: [00:29:13] Right, weaving it all together in context.

Jared: [00:29:19] But to get that right, I have to do a boatload of research.

Sean: [00:29:22] Right.

[00:29:23] I mean you cannot just get your way into that because you will find yourself in a context that doesn’t make sense. And so you need to really understand what that passenger’s travel journey is like to make sure that you’re always giving them the options that anticipate their needs at exactly the right moment.

Sean: [00:29:46] Brilliant. Alright, so weaving this all together. One last sort of subject here and then I know Joe’s got a question or two for you. Let’s go back to that confident powerful executive who’s telling you what you need to build into your product, right. So I believe that, along your lines, when you’re setting up your product vision, it’s got to be solving problems for the people that are using the products. And if it’s not, everything else along the way gets broken. And we constantly are up against this mindset. We call it the fixed mindset. If you follow Carol Dweck’s work at all, it’s aligned with some of her thinking around how people evolve. But this mindset of: “we have this money, we’ve got to get these features out the door, here’s a list of features and we’ve got to get them out the door by this date.” And when that’s the vision that the whole team is marching to, “we just got to get the product out the door by this date,” everything sort of breaks down because we’re not focused on the problems that we’re solving. Do you have any examples of companies or teams that have managed that whole process well? Because at the end of the day the executive is doing that because he’s probably made some commitments himself, and at the end of the day our teams might be afraid of rubbing them the wrong way or losing the work and it’s this balance in the real world that we have to figure out how to meet.

Jared: [00:31:10] Yeah, so again, the first thing we need to realize is: that executive’s on the team. Right. We often want to position the team and the executive the sort of separate entities…

Sean: [00:31:22] Good point.

Jared: [00:31:23] …and say that the executive is not on the team and now the team has to respond to what the executive said. What we need to realize is, that executive’s on the team because that executive is influencing the work of the team. And anybody who influences the work of the team is on the team. And so we need to treat them as being part of the team which means that the executive needs to see themselves as part of the team. They need to be thinking if they’re making a commitment somewhere else that, “I’m taking this back to the team and we’re going to talk about this,” and that we is because the executive is part of the team. And, you know, some of that’s cultural, as we’ve talked about before, it’s just making sure that that we keep telling the executive that they’re part of the team. “You are part of our team because you are bringing us insight into what customers need. And we’re also learning insight from other sources. So we need to keep everybody up to date on all the insight we’re getting. So we need to include you as part of the team. Now you’re a very busy person. You can’t come to all the same meetings that we go to. So what’s the best way for us, as members of the team with you, to make sure you have all the insights you need so that when you’re making commitments, you’re making commitments that are going to be the things that we can execute effectively as our team representative and that we can make you look good, because, you know, part of our role as a team member is to make everybody on the team shine.”

Jared: [00:32:59] So we need to treat the executive in that regard. And that means that the executive needs to be fully aware of how we work and understand that we work best when we are really understanding the problem. And so that means teaching the difference between a solution and a problem which, for a lot of people, it’s a hard thing to identify the solution and the problem. Because the first thing you do is you start gravitating to bigger solutions. “I need to export data” becomes “I need to get this into the billing system.” That’s still a solution. You haven’t told me why we needed to get into the billing system. Right. So, you know, people play the game of five Y’s, or other types of root cost analysis to sort of get to, “oh wait a second, this is the thing that triggers this, the project is done and we need to signal that to the customer so they know to pay us for the project because that’s the way the project is set up.” So OK that’s really important to understand that there’s this moment in time where we have to signal that. And now we need to work in and through that direction. So that’s what we’re looking for, right. That’s the moment we’re trying to get to, and understanding that that executive is a member of the team is critical to that. So that’s one piece of it. The other piece that’s I think really essential here is realizing that you do your best work and the customers are most happy when you’re solving problems. And so solutions inevitably have to come. But problems are really the focus at the early stages of the work. And then they become the focus all the way through. And so getting all the language around problems, right. And we see this all the time.

Sean: [00:35:04] Right.

Jared: [00:35:05] Here’s an example: I was working with a team that had a research project called video sales, and its entire purpose was to figure out how to use video on their e-commerce site. So they contacted me and they were like, “so what are the best practices for using video on the e-commerce site?” And I’m like, “well there are lots of best practices around using video, there are lots of best practices for e-commerce, what specifically are you talking about? And they’re like well, “we want to integrate video into our e-commerce site.” I’m like, “OK why?” “Well we were hoping you could tell us.” But how would I know? “Have you observed problems that users are having?” “No. But you know, there’s all this video technology out there and we can do things like personalize videos. So when you bring up a product, it could actually address you by name because it gets that name out of a database somehow and inserts it into video. And we want to figure out how to use that.” I’m like, “well why do you need to use that?” “Well we were hoping you would tell us.” I’m like, “I don’t have an answer to this. Have you done user research?” “Yes.” “Has has any of that user research taught you that this is something that you need to do?” “No, that’s why we’re asking you.” And I come to learn that the project name is the interactive video team and if they don’t produce an interactive video, they fail. So they’re off to produce an interactive video even if it’s just a solution in search of a problem. So the way we name things, when we name things the mobile team, or we name things the data export team or the data export project, suddenly that project’s a failure if we don’t export data, even though exporting the data may be the absolutely wrong way to solve the problem. So we have to be very careful when we even create our projects and name them, and create our teams and name them, because we may be doing ourselves a disservice because we create this this moment of failure that is inevitable because we named the thing the wrong thing.

Joe: [00:37:38] Great example. Why, why are you asking me?

Jared: [00:37:41] Exactly. It was so crazy. I did not understand this conversation at all.

Joe: [00:37:50] Well just to close us out on experience rot and then we’ll just ask our final standard question we ask everyone, but, is there like a company and/or a product that you think does a really great job avoiding experience rot or managing experience rot?

Jared: [00:38:07] Yes. Well I think, I mean we saw it in the early days of the iPhone and the iPod. Apple was very good at keeping the functionality minimal, and then they sort of fell into the trap and now, you know, there are so many things that the phone can do that people don’t know that it can do, and it suffers from complexity rot like everybody else. I think the folks at at Basecamp are fantastic at this. Jason Fried and DHH and the folks who run that operation there have been very explicit over the years of keeping spirit to a point where they go to an extreme. They don’t just automatically change products on people, they create a new version of their product (they’re now on their third version) and they leave the other two versions running. And if you don’t have a reason to switch, you don’t switch to the new version. They don’t add any new features to it, they only keep it upgraded for maintainability and security. So Versions 1 and 2 are still out there, and your data is still there, and you can still access it. And they’re happy with that. And they completely rewrite the code from the ground up for Version 2, and then for Version 3, and it’s a different UI and it works differently because they’ve realized that the product needs to be something different. But if you don’t want to switch, you just stay with the old version. But the new customers all come in on the new version. And that’s worked really well for them.

Jared: [00:39:57] And I think there are lots of examples of things like that. Intuit, for example, when they created there, they realized that people were using the Quicken product to manage their rental properties. You know, they had an apartment, or they had a house with a couple of apartments, and they were trying to manage all the expenses and classify them because you have to report them on your taxes differently, and when you go to sell the property, you have to be able to pull out all the improvements you’ve made to the property because that adds to the basis. And so people are doing this with Quicken, which it wasn’t created for, and they were asking for solutions. “Can I classify data this way?” And they finally went out and studied people asking for those things, and found out that they were hacking the product to do this thing it wasn’t intended to do, and instead of just supporting that in the product, they just created a whole new product. They created Quicken for rental management, and it’s one of their best sellers. It’s a high growth product for them. It has a small niche market, but now they can actually tailor it to the actual problems of managing rentals.

Jared: [00:41:18] And I think if you take on solutions like that and you’re very creative about it, you can make sure that you in any given product you don’t have experience rot because you’re actually growing your business by creating separate products, where you can control the complexity, that all work together. And you know, Quicken has a core code base but then they have forks off of that code base that go for the main product for the rental property product. And so, you know, security and basic integrity stuff go in the core product, but things related to tracking and classifying data go into each of the sub products. You know, it’s a bit of work, but it makes it much easier for them to make changes and to grow each product.

Joe: [00:42:08] That was a very cool example with Basecamp. I think that’s a really novel way to approach it. So to sort of close this out, the question we ask everyone: is there a favorite book you have at the moment or that you like to gift out related to product, UX, innovation, any of the likes?

Jared: [00:42:26] Well that’s a good question. Which would be my favorite? Well I have I have so many books that I really like. The ones that I tend to focus on that I refer people to the most are around team building. Marcus Buckingham’s First, Break All the Rules book, which is from the Gallup Organization, and Lou Adler’s Hire With Your Head, which is a book about hiring which is essential for building great teams. Both of those books are really essential for building great teams. So I very much like those books. Kevin Hoffman’s new book on meeting design is fantastic. Scott Berkun has a, it’s not a new book, but it’s it’s a fantastic book on innovation and I’m trying to remember the name of it. If only I had a little box that could tell me the name of any book…

Joe: [00:43:37] Somebody could make a buck off that.

[00:43:39] Oh yes. So Scott Berkun has a book called The Myths of Innovation and anybody who’s interested in innovation needs to read that book because it communicates right off the bat that innovation is not about invention, it’s about adding value and that the way you get to innovation is by doing research. You just spend a lot of time studying problems. There isn’t a team out there that I’ve ever met who, if they really understood the problem, they couldn’t come up with 15 viable solutions to solve that problem right off the bat. People who are trying to innovate who struggle at innovating just don’t understand the problems they’re solving. So that’s sort of the core message of Scott’s book, and it’s brilliant.

Joe: [00:44:30] Those are great, we haven’t heard those ones yet. Thank you.

Sean: [00:44:33] That’s great. Well you never disappoint, Jared, this is a great interview. We really appreciate you spending time with us. Thanks for joining us today.

Jared: [00:44:44] Thank you. This is this has been fun, and now since I never disappoint, that’s something to achieve. I will add that to my bucket list.

Sean: [00:44:53] I’m going to end with one of my favorite all time conference quotes that I heard you actually utter these words as you closed out the ITX conference in Rochester. Heroin is delightful.

Jared: [00:45:16] Heroin is delightful.

Joe: [00:45:16] Alright, so that’s it for today. Thanks for listening. And we’re not going to just talk to talk, we’re gonna walk the walk, so we would love if you go into your podcast products and leave us a review. Sean and I guarantee, we are committed to reading absolutely every piece of feedback we get there, and not only that but you’re helping other listeners. By getting that feedback in there, it helps us move up the search rankings so that other people can find the episodes. So thank you everyone.


06 / Launching a Healthcare Product


Time, technology, and innovative thinkers have revolutionized what the healthcare system stands for today. Regardless of these achievements, though, the entire system remains very health-centric, focusing only on what happens when something goes wrong and a patient must seek out help. Imagine a framework that flipped this concept on its head, putting the patient at the center versus the health event. Imagine a system that gives users the tools and resources they need to create a healthy lifestyle that’s based on their wellness needs, motivations, and interests. Ultimately, a healthcare system that’s not just approachable, but promotes a better, more healthy existence for the entire population.

In this episode, we sit down with University of Rochester’s School of Nursing team, Renu Singh and Brian Harrington to get a behind-the-scenes look at a cutting-edge healthcare product they brought to market for University of Rochester employees, YoURHealth. Our hosts Sean and Joe not only look at how this power team brought this beautiful concept to life but also their process for continuously innovating the framework to provide care to the ever-changing needs of the different organizations it now serves.

Read our blog post

About Renu

Renu Singh is the CEO at the University of Rochester Medicine’s Center for Employee Wellness, as well as the Senior Associate Dean of Finance and Operations at the School of Nursing. Renu Singh spearheaded the development and implementation of the Center for Employee Wellness for U of R. As Associate Dean of Operations, Singh oversees the school’s technology, personnel, and facilities management. This year, Singh was bestowed the Women of Excellence Award by the Rochester Business Journal due to her career accomplishments, community involvement, leadership, and commitment to mentoring.

l t

About Brian

Brian is the director of Information Technology Services and Building Operations for the University of Rochester School of Nursing where he leads teams and projects to deliver high-quality integration of technology with the School’s academic, research, and practice missions. With over 20 years of technical expertise in healthcare environments, including leading the technical efforts of the flagship UR Medicine Employee Wellness program, he has also contributed to technical success with the University, Kodak, and Xerox. Brian has an MBA from the Simon Business School currently is an executive board member for Goodwill of the Finger Lakes and ABVI.

l t


Joe: [00:00:00] OK, so we are here with Renu and Brian from the University of Rochester. And we want to talk to them about the application that they’ve been deploying and launching and selling, essentially, at this point. Because what we wanted to talk about today was really the behind the scenes of project management and actually getting it into market and all the fun and challenges that that comes with, so Renu and Brian thanks for joining us today.

Renu: [00:00:24] Absolutely. This is going to be fun to talk about.

Sean: [00:00:27] Super excited to have you guys on. Can you tell us a little bit about your business and your roles for our audience just to bring them up to speed?

Renu: [00:00:34] Sure, absolutely. You know, I can start us off. Our business, we actually developed this business as part of the University of Rochester and we developed a product that’s part of the UR Medicine product which means it’s basically a clinical product. And what we do is we provide wellness assessment and wellness intervention programs to the community. We are a program that contracts directly with employers, and at a population level, we are engaging individuals to really take control of their health. And we also help coach individuals with chronic diseases to live a healthy and productive life.

Sean: [00:01:21] Great description. And what’s your role, Renu, with the organization?

Renu: [00:01:26] I am the CEO of UR medicine and Play Wellness program and I’ve been involved from the very early days, which was really not that long ago, but it has been involvement that really started with developing a technical platform in partnership with a wonderful tech company that’s in the area. And in that platform we’ve continued to further develop to essentially fulfill the demands of the workplace. Now, we’ve actually implemented our program in thirty nine organizations today.

Sean: [00:02:07] Holy cow, that’s amazing.

Renu: [00:02:09] Isn’t that.

Sean: [00:02:10] And do you happen to know how many lives you touch through this program?

Renu: [00:02:16] So the lives that we touch really is close to probably about forty five thousand.

Sean: [00:02:22] Wow.

Renu: [00:02:22] And yes, it has grown substantially. And by touch, you know, industries really define touch very uniquely and independent to what they do when it comes to wellness. But from a touch perspective, our platform has grown significantly, the number of lives we cover has grown significantly, and the programs that we’ve touched individual lives with is probably about thirty thousand.

Sean: [00:02:51] Wow. Brian, you mind jumping in and telling us a little bit about what you do for the organization?

Brian: [00:02:58] I’m the director of information technology services for the School of Nursing and the employee wellness program. As Renu indicated, started a few years back and I was immediately part of that kind of getting the technology going that would help the clients get their biometric information, their personal health assessment information, and as well as linking that information through to where the nurses and coaches could have access to it in order to provide you know the quality support that they do from a clinical perspective. So the technology kind of is the glue that holds everything together and makes everything work.

Renu: [00:03:44] And in fact, what Brian didn’t tell you is he also served in the early days as my translator when we really moved a clinical product and built the technology infrastructure and the platform to support that. And that required a real crash course in understanding technology and the language around it, so Brian was also my translator.

Joe: [00:04:10] That’s cool. So can you talk a little bit about how the idea initially kind of came up and how it was brainstormed. How did you, you know, UR is a very big organization. How did you kind of get the approval and the funding and get everyone’s buy in to do this?

Renu: [00:04:25] So it was an interesting process and I think that I can honestly say that we were in some ways fortunate that it was a concept of ours in early days, even earlier it was Dean Pat Shiverton in fact that probably about 15 years ago showed an interest in wellness and that was before the wellness industry really took off. And and she saw that the role of nurses was really instrumental in being able to coach and to educate individuals on their health and wellness.

Renu: [00:05:04] And so the interest was there early. However we didn’t have the infrastructure, we really were not able to be mobilized at that time. The University of Rochester, then I think they had about twenty four, twenty five thousand employees, and they were interested in moving in the direction of wellness because it was something relatively new in the marketplace at the time. And they contracted with a national vendor, I think they’re located somewhere in the Midwest, and that vendor though they had a beautiful concept, they were really unable to engage employees at the University. And the impact that the organization made really didn’t justify the cost of the program itself.

Renu: [00:05:52] We, recognizing that, worked with the University of Rochester’s human resource department and and finance office to make sure that they didn’t drop a concept that was beautiful but instead looked at programming and how it was most appropriate to deliver, you know, the program that we wanted to deliver. And so with a lot of discussion really around the clinical framework that really is important to successful outcomes for wellness programs was what we knew best. And that’s what we knew how to do and what we needed to do in order to be able to provide that was to build the technical infrastructure to support it. You know, there were many platforms that were available in the marketplace that were not designed around the clinical framework that we understood to be very important for us. And so the University said, “listen, if you think you know how to do it, we believe you. We see the science behind what you’re talking about and the programs that you’re wanting to develop. We will help you within reason.” And so what they did was they invested in us by actually allowing us to contract with them. So we had a guaranteed contract with the largest employer in Rochester, which was nice, but we also had to demonstrate great value and build our program to support the value that we that we proposed to them.

Joe: [00:07:29] Got it. So kind of piggybacking that, you know, we have product managers who may be working at a startup and that’s more entrepreneurship, versus other product managers who are working within a larger organization and launching a new business line, some people call that intrepreneurship, so in your opinion, where are the differences in how to approach launching the big new product line at a big company versus what a startup can do more nimbly.

Renu: [00:07:55] So you know, that’s an interesting question. I would say from an internal perspective there are some advantages that are inherent. You have an organization that is supporting your growth and helping you to really achieve your metrics and they’re partnering with you. So in a sense, it’s a little bit different than a customer relationship. It is a bit more of a partnership. What I will tell you is when you are providing a service to any organization, internal or external, partnership only lasts as long as you deliver. And so you’re still very tied to exceeding metrics. And we built the program knowing that if we could do what we conceptualized, then we would want to be entrepreneurial in the sense that we want to offer our programs to, at the time we were we were thinking just in the region. And so about two years ago we started to do that and our growth has just been very rapid and well received.

Sean: [00:09:08] That’s amazing. So I’m going to take a little side step here and ask sort of a different question. So I have this belief, this personal belief, that the health care system has evolved to where it’s at because it was the right thing to do in a past time, but things have changed. Like we now have access to people’s information and their attention through digital products like mobile phones and things of this nature, that we can actually take a completely different approach to health care now. And a lot of the complaints you hear, no matter where you go, about the health care system, I believe revolve around the fact that it’s very health event centric.

Renu: [00:09:49] Mhm.

Sean: [00:09:49] Like we have built the entire system to be focused on the health event. Like something goes wrong and that’s when you engage the medical system, right. And you guys have completely flipped that on its head. You said, “let’s not do that.” Even from our earliest conversations we had with you when you were thinking about building this product, we had some of these conversations. “Let’s put the patient at the center of our universe and figure out what infrastructure and what tools do we have to put in place to make sure that wellness is at the center of their universe, and how do they engage with our organization to remain well so that we can build this sort of framework and platform for a better, more healthy existence in the context of the world,” right. And it’s kind of a brilliant model. And we know you’ve been pretty successful in deploying it, thirty nine organizations, forty five thousand lives, that’s amazing.

Sean: [00:10:47] My next question is really, how do you measure success? And I want you to answer the question in two ways. One is, obviously outcomes are a clear measure of success in the long run. If you get better outcomes, healthcare costs are reduced, everybody wins. But I want you to also answer it from the perspective of the software product. How do you know that the software product itself is is achieving its goals in the context of your greater goal which is better outcomes?

Renu: [00:11:14] Actually that is a great question. First I just want to acknowledge that your assessment of our health care system and the way that we’re looking to be different and still be tied in to the delivery of health care is spot on. I will tell you is you know you’re absolutely right. Outcomes, clinical outcomes are you know the easiest way to say, “yes you’re successful.”.

Renu: [00:11:41] But another outcome which is very much tied to what we’ve developed in our platform is engagement. And engagement is a critical metric in wellness because we are looking to engage people before they have an event. So Sean, you talked about the health industry really being focused, you know, triggered by a health event. We’re really wanting people to get engaged because they are wanting to make a difference in their lives and not have a clinical event. So that really entails a very different way of reaching people, and reaching people where they’re at. So it’s the synthesis of information, of data, that we are collecting. Our portal, our infrastructure is allowing us to collect information and target individuals not only on what their health and wellness needs are, but what their interests are, what their motivation is, and we’re able to engage people in an entirely different way. So our portal allows us to model our engagement strategies in a way that really is different in healthcare today. Today, if you go to any traditional health care model is going to be patient is not feeling well and there’s something going on, they’re going to initiate the engagement. Our focus is entirely opposite and that’s what our platform has helped us to do.

Sean: [00:13:17] That’s a fantastic answer. So if I could sum it up, on the software side you’re measuring engagement and how often people come back and use it, which in our world we call that like a loyalty factor. Like they’re using it because you’re getting value out of it. And if you drive that that will ultimately drive outcomes. I love it. So I got a quick question for Brian. Going back to the very beginning of time when we thinking about different ways to get the product deployed in a market and used. You guys used some financial incentives early on, if I recall. And for us, it’s like, we believe that there is this thing called the loyalty ladder. You first got to get people to trust the product, to start to use it, and then you can work on engagement and loyalty and then ultimately we want to achieve this thing called advocacy, right, where they’re telling each other you got to use this platform right, you got to use the biometric tool. So Bri, I remember in the early days we were using financial incentives. What other tools did we use to try to earn trust from our users?

Brian: [00:14:21] So yeah, in the beginning there was the financial incentive, and that’s actually still present, but going beyond that, you know making sure that the system does what people want it to do, that it engages them in a way they want to be engaged. As Renu indicated, some of the things that work toward engagement, we have questions about motivation to change, you know, maybe you’re not motivated to quit one activity but you are motivated to quit another, whether it’s to lose weight or eat better whatever. We have that information and we’re able to kind of reach out to the people in the area you know that they’re interested in. So through the various kinds of data that we collect, we are better able to customize the way we communicate with the individual clients.

Brian: [00:15:05] And you know, you mentioned before sort of the event kind of driven health system. We’re always very careful to not call the people patients, right, they’re clients. And just that attitude that we’re talking about healthy people who maybe want to fix some area of their life and want to help to coach that sort of thing. They don’t realize, you know, everybody knows eat better and exercise more. But the details of that are a lot more tricky, right. What can I do that fits my lifestyle? And the coaching that the nurses provide helps make that happen helps find a way that is right for the individual and all the data that we have in the system is what helps make that successful.

Renu: [00:15:46] I would add to that Brian, just in addition to that, I will share that many of the organizations that we’ve signed with do not offer incentives. So though the University continues to, we work with organizations that do not offer incentives. So you know to add to what Brian was stating, I think that though we now also work with organizations that do not offer financial incentives, what we’re finding is the resources that are available to them that we can put online and organized in a manner to push content directly to individuals based on their motivation and based on their needs and their interest really engages people. So we have just as high of an engagement rate in organizations that offer little to no incentives as we do in organizations that have incentives. And I think that’s an indication of the usefulness of our infrastructure, of our portal.

Sean: [00:16:51] Absolutely. I was going to say that was a great answer, and I think your product, your service really, it’s more of a service than a product, is really an interesting and cool mix of real human services, the coaching that you mentioned Brian, which which builds trust early in the process. That’s a key that’s a key trust factor for you to get people using the system and get them engaged.There’s a human to human element that’s goes beyond just the software product and I think that’s really, really important.

Brian: [00:17:22] I’m just gonna jump in with, a lot of the things too is, we’re dealing with health information, and you know, trust is particularly important there, right. You have to trust that the organization is going to care for that information in a proper manner. And so we’re always doing security assessments on the system with a security group that’s internal to the University but not part of our organization to be sure that we’re handling the information correctly. And when we took over for the organization that was here previously, there was a lot of feeling from people that they didn’t handle things correctly (the outside organization before we came in). And you know it was a big thing that we wanted to do this right, to approach people in a way that felt comfortable, didn’t feel like you’re kind of invading their space, and it’s sort of a challenge to kind of balance that and come in in a way that’s helpful, and you know, meeting what they would want but not seeming, you know, kind of spooky or however you want to word that, you know.

Sean: [00:18:24] So piggybacking that, just to kind of talk about the behind the scenes again, there’s a lot of regulations that you need to comply with, whether it’s HIPAA or ADA or other ones like that. Could you give a brief overview of the types of regulations that you need to deal with and what they kind of each entail?

Brian: [00:18:40] Sure. So HIPAA obviously is one that we deal with in multiple areas here in the medical center but specifically with our product, right, there is health information in the system that we use to, as we talked about before, to better engage people. But so along with that comes a lot of, you know, security testing, documenting process, and having everything clearly understood and meeting the different requirements for HIPAA. And the medical center works with a program called High Trust which kind of tries to roll up a lot of the things that you would need to be compliant with. And so by meeting the High Trust standards, you know you’re going to meet HIPAA, FISMA, all these other things that are out there, FERPA, all the different letter acronyms. And so that’s been our goal and we work with them closely to ensure everything that we do is compliant with all that stuff. ADA is a little different, it’s more around the accessibility of the site. And we always, you know, everything we do from student facing as well as for products like this, we always make sure that we’re ADA compliant, things work well that way. And of course working with ITX got us started off with that right from the beginning. You guys were really on top of that, even when we first launched, having that kind of stuff built in from the ground up.

Sean: [00:19:58] Thanks for the plug.

Joe: [00:20:02] So how much, just to kind of talk and reality for a minute, how much did those regulations play a role in your decision making around how the user experience played out, or different product decisions you had to make?

Brian: [00:20:15] I would say that, you know, it’s it’s always in our head, right. Because we understand the importance of securing data. But at the same time, when we’re thinking about the user, the client experience, that’s at the front of our mind right. How do we make this the best experience for our clients that we can make it? And then once we understand that, how do we make that work within the guidelines of keeping everything secure and safe? Because you know, security is one of those things you have to have but you don’t want it to be in the way, right. It needs to be kind of seamless and that’s the challenge is to have a product that’s appealing to the client that they want to come use, that they realize the benefit from it, but yet also behind the scenes making sure everything is safe and secure.

Renu: [00:20:59] So Brian if you’d like, I can start on that, and then I would love to have you join in, one of the things that I think has been really helpful for us is that that we designed a platform that we could have grow with us. And so while we looked at our clinical processes and measured our outcomes and at times adapted our operations or our programs to really reflect outcomes that we wanted to mirror or to continue to improve on, our portal and the technology I guess was minimal enough to allow us to adapt our processes accordingly.

Renu: [00:21:50] And I will give you an example that to me is a very wonderful and differentiating factor supported by our portal that allowed us in wellness to exceed national averages of a high benchmark for engagement. And that was our process that we deliver our health assessments in are done one on one using clinical specialists. And we found that that was a wonderful time to engage individuals into programs, because as you know, I think it’s probably something that you would logically think of, but it’s not something that was built into the process of what is typically done for wellness. And what we did is, we learned that if you have a face to face interaction, that is the time to really engage individuals into programs that are designed specifically for them to meet their needs. And what what we were able to do is to really align our platform to support an approach that worked very well for us with engaging individuals into programs that were completely appropriate for their needs. And the way that we did that was we took the clinical interface on our portal and we created dashboard measures for our providers.

Renu: [00:23:18] So in real time when we are meeting with clients and assessing their biometric values and looking at their lifestyle factors that could contribute to their health and wellness, we were able to provide that information to the nurse right there on her screen and also in addition provide where they are most motivated to change and what do they need most. And so our providers were able to utilize that information and engage individuals on the spot. And I will tell you we had a 354% increase in our engagement, and the national average for engagement in disease management programs, for example is on the high side, if someone says that they have 10 or 11 percent they are doing a bang up job. Ours is 18 percent and that was really supported by the infrastructure. If we didn’t have that interface for the clinicians we wouldn’t have the engagement.

Joe: [00:24:24] Got it, yeah. Perfect answer. So the next question I would love to hear both of your opinions on. So thinking about the product and as the product was built and rolled out, how did the product drive changes on the business side? Like how did the business have to adapt with their processes, their approach to accommodate the software?

Brian: [00:24:47] You know, you kind of said, you know, “how has the business adapted with the software?” I think that’s the benefit of having our own development team, having you know, started out working with ITX and then created our own team that’s keeping the product moving forward is that really the business informs the software. As Renu kind of indicated, one of the key differences with us is that we’re clinically informed. We’re evidence based practice informed, you know, so nurses are involved in all of the process, whether it’s our nurse coaches, whether it’s people like Renu who are seeing things, looking at different numbers and saying, “oh we should be doing this this way,” we then are able to take our software and enhance it in ways that support what the business is looking for, what we’re trying to make happen, whether it’s engagement or any of the other areas that we’ve kind of slowly modified. And it’s great because, you know, we can try things, right. For our care management programs we said, “well what would it benefit us if we used, like Fitbit data. And so we know quickly wrote a module, got Fitbit data in, and were able to see how does that help us? Is it useful to the nurses in trying to coach people and help people move forward? And so we’re able to be flexible and try things and see what works and make the software really kind of match business needs.

Sean: [00:26:08] Outstanding. So the work you’re doing, it’s profound on many many levels. With so many users now and so much longitudinal data, because you’ve been doing this for a few years, have you published any research yet? Have you guys, I know you’ve had some data analytics people because they’ve had concerns right from the beginning about making sure we’re collecting data in a way that we can make it useful. Have you guys published anything yet in terms of how you’re able to move the needle on human behaviors with the platform?

Renu: [00:26:40] So what we’ve been able to do, we have a couple of studies that were published on metrics that we measured through the program. But we have more to do because our outcomes are phenomenal and the data has been very useful from that perspective. We can literally measure improvements or change in behavior because of the data that we collect on a routine basis and we can look at changes over time with individuals. That’s phenomenal. We recently had three presentations at the National Wellness Meeting in 2018, it was in early June, and the presentations will eventually become publications. So we’re maybe a little early on that, but the presentations were really developed through the findings that we’ve had with our collection of data.

Joe: [00:27:42] So we’ve talked about the growth of companies signed up, the number of users. So is it a fair statement to say that this is one of those products like Amazon where anyone can be a user?

Renu: [00:27:53] So you know it’s interesting. From a technical perspective, we can see anyone anywhere. Our program design right now is more of a regional design. But it’s interesting that you mention that because we are looking at a national presence. So we’re considering that just given our outcomes and the programs that we have designed clinically that are supported through our portal have really demonstrated themselves to be beyond what we’re seeing in the wellness industry in that space. So there is interest and there is good rationale to take the program nationally. I think, you know, if we did, our platform is probably the most prepared portion. And what we really have to do is look at how we can provide the high touched clinical interventions using in-person and telemedicine. And so we’ve already started with telemedicine for our disease management and our coaching programs. And now we’re looking at different ways to also provide our biometric screening and coaching that occurs early on to bring in engagement utilizing additional technology and a point of care testing.

Joe: [00:29:23] Got it. So considering that you could have someone who’s very young versus very old and all kinds of different health backgrounds using this, plus the idea of going national, how does that play into decision making around the product and how you build it?

Renu: [00:29:41] Well so that is I think where we’re at right now. And we’re evaluating what the strengths are locally in the market and what we would need to do from a national perspective. And I think that our product actually is well suited. We are very much like, it in terms of the resources, the access, and the ability to apply our wellness platform on a national level. The platform itself is ready. It’s there. It’s very comparable to other national organizations that have moved from kind of that plug and play wellness platform to more of a clinically integrated platform. We provide a lot of resources, online modules, and more of what I consider to be low touch but high impact programs on our platform already. What we need to do is to further integrate our clinical high touch solutions with a national presence. So the platform is there. I’m not sure that I’m answering your question, but…

Joe: [00:30:58] That’s good. The platform is ready. I mean, I think Brian, you and your team have probably considered that as you think through the database and various ways you’re going to architect the system. You know, I remember when when we worked on the product years ago, we had to consider if someone was pregnant and all these kind of different flows that a user could go through. So, you know, figuring out how the those different flows could come into play because they want to apply to everyone and you know and taking people down the right path for them, essentially, and the personalization you talked about.

Renu: [00:31:29] I see, and what I will share is that because our platform is really built to be customized for each organization, we would continue to do that on a national level. What is I think helpful for us in this domain is we’re working with a population level. So we’re already working with forty thousand individuals, for example, and as you move across the country, there are pockets, there are areas where the needs may be a little bit different, but from a clinical perspective, they’re all within the range of what we would expect and anticipate and what we’ve designed for it.

Joe: [00:32:11] Got it.

Brian: [00:32:12] I can add a little from a from a technical perspective. The platform is ready in the sense of the process, but as we grow, we’re looking at, you know what do we have? So right now we have fully redundant systems load balanced, and separate sites, and you know, Tier 1 data centers kind of thing. But at what point, if we have a bunch of customers on the West Coast, do we go to a cloud service and get some support there so that we can have a broader reach from a service perspective? And little things too, just looking at most of our time, all of our times right now, because we’re regional, we don’t really pay attention to time zones. As we start thinking now, we’re like, “okay let’s make sure that we’re thinking across time zones so that if we’re scheduling on the West Coast we don’t end up not showing up for three hours, or whatever,” that sort of thing. Thinking about stuff like that just from a data structure perspective as well as from a system architecture and what kind of support we need from a server perspective to meet growing needs. And that’s the kind of stuff that we’re also constantly looking at on the back end. You know, what’s our server utilization? It is the as system responsive as we want it to be? And that sort of thing.

Joe: [00:33:22] So just to talk in the weeds for a minute because I think this is important for product managers and teams, you know, operating behind the scenes here. What kind of cadence do you have where business is talking to technology giving feedback and then technology is giving feedback to business? As well about you know how do we deploy the West Coast and what that would look like?

Brian: [00:33:42] Cadence kind of makes me think about sort of a frequency that you’re kind of going after and it’s really pretty constant. You know, we are part of a bigger organization, but this group is almost like a startup within our organization. And so, you know, my team meets very frequently with the operations team, with the clinical team, and so there’s a constant sort of feedback. We’re always looking at, you know, “what do we need to adjust here? What we need to adjust there?” And Renu as the leader for the whole organization is really great about keeping us informed, making sure we know, “OK, this is what we’re looking at now, how do we get in this direction? We want to go national, what we have to do?

Joe: [00:34:26] Communicating the vision to the technology team.

Brian: [00:34:30] Exactly yeah. And to everybody. She’s a fabulous leader. So that’s really helpful for us, right. Because it’s not like something is a surprise. You always are kind of knowing what we’re looking at, what’s coming, and you can start thinking in that way so we’re ready to meet the challenge that comes up next.

Sean: [00:34:49] All right. Kind of one last question for you here around how you collect voice of the customer and how you get feedback from your user community. A lot of folks have moved towards this thing called Net Promoter Score. Have you guys heard of that? NPS?

Brian: [00:35:08] No.

Sean: [00:35:08] OK, I won’t talk about that then. But no, so how do you gather, like what you are going to build into the software product next and how do you get a pulse on what your users are using or not using, and how do you prioritize features into your software development investments?

Brian: [00:35:30] Okay. We do a lot of surveying through different survey tools, so that gives us an idea of satisfaction and what’s working, but a lot of our information really comes from the interactions, you know, the clinical team interacting with people, saying, you know, “this is what we need.” You know it gets talked about, it gets kind of brought up as, you know, “this is something that we think would be beneficial.” Operationally we look at it, you know, “does this makes sense to do?” So it’s really, right now it’s pretty interactive. I think what we probably need to be looking to do is better ways to capture that digitally from the system. And that’s an area that we can start looking to grow in.

Renu: [00:36:12] I think I would add to that you know, the difference between our approach I would say and the approach of what I now call first generation wellness programs is, our way of reaching people is about giving them the information that is important for them in a way that they understand it and in a place that they can find it. And so much of that is driven by the intelligence of our system really looking at driving content to the right people and making certain that the content we think is most important for them and they will be most engaged with is in front of them. And a lot of that is based a clinical need as opposed to what’s most popular to look at from a “how many clicks did you get on such and such page?” perspective.

Renu: [00:37:16] And one of the things, I absolutely agree with what Brian said, one of the ways that we are able to do that is we have a high touch program where individuals are very vocal about what they’re looking for and what areas they they could improve in if they had better resources and what kind of resources those would be. And so yes, absolutely, from a technical perspective there are always new innovative ways to make sure that you’re able to to target individuals from that perspective. And so we’ll continue to grow along those lines.

Joe: [00:37:59] Okay so the one question we like to make sure we get in with all of our guests is, I guess in this case is there any book that comes to mind that really speaks to helping bring a product to market, or you know, getting a product successfully into other organizations? What do you take inspiration from?

Renu: [00:38:20] So you know, that’s a good question. We’ve read so much on what it is to build programming that that is engaging. And I think, you know, I’m trying to think about what what I go back to, and I have to I have to tell you, I’m a little bit stumped because I can’t think of what I have actually gone back to and referred to several times. I may have to get back with you on that.

Brian: [00:38:54] So it’s not so much something I read recently, but Daniel Pink’s book Drive really is something that I read during this process of building and growing this whole platform and business. And just the sort of concepts that he brings up around what drives, what motivates, people to do things? In this case, wellness. How do you help somebody develop the drive to to choose a better lifestyle to live better and healthier? And some of the stuff in that book, it’s a great read and it really felt like it was really applied to what we were doing.

Sean: [00:39:31] There’s another plug for the University of Rochester because Drive is actually based on a lot of Ed Deci’s work. He was, still is, a social scientist at the University of Rochester who has worked his entire life on human motivation and Self Determination Theory.

Brian: [00:39:47] Absolutely.

Sean: [00:39:49] It all comes back around to the U of R at the end of the day.

Sean: [00:39:57] So thanks again for joining us, you guys. It’s it’s been a fabulous podcast. I think. We’ll see what happens, what our audience says I’m sure they’ll have some great questions and we’ll have some great follow on material for this. But we’ve been a big fan watching you guys evolve this product over the last many years. And again, I appreciate both of you guys taking the time out of your day to join us.

Joe: [00:40:19] Thanks so much.

Brian: [00:40:20] Thank you very much. It’s been a great conversation and we always enjoy working with you guys.

Renu: [00:40:25] It was such a pleasure, and I have to tell you, our experience in working with ITX and developing this product, from a person who had very little technical background, it was absolutely a pleasure to see how technology can really enhance a product and help you deliver what you’re hoping to deliver.

Sean: [00:40:46] Well thank you for that plug. I appreciate it. It was a it was a great adventure working with you guys so appreciate that.

Renu: [00:40:53] Thank you.


05 / Human Experience in Products


As companies implement more technology and automation into their products and services, it’s all too common for them to forget that customers and users are human beings. We should always be thinking of the human side of technology and software, considering how our users feel, think, and how to best communicate with them, to give us an advantage in how to better serve them. In the end, what we’re really selling is experiences.

In this episode, Sean and Joe talk with Kate O’Neill about her experiences with the internet and how she uses human-centered design to improve an individual’s experiences with products. We look at tactics for introducing more human-focused experiences and how to measure their success.

Read our blog post

About Kate

Kate O’Neill, “tech humanist” is founder and CEO of KO Insights, an award-winning thought leadership and advisory firm helping companies, organizations, and cities make future-aligned meaningful decisions based on human behavior and data. Author of 3 books including PIXELS AND PLACE: Connecting Human Experience Across Physical and Digital Spaces, Kate speaks regularly at industry conferences and private events, providing keynotes, participating in panel discussions, and leading creative brainstorming workshops for groups of all sizes. Her expertise has been featured in CNN Money, TIME, Forbes. USA Today, Men’s Journal, the BBC, and other national and international media.

l t i


Sean: [00:04:55] All right, so cool. So today we have a really great guest, Kate O’Neill, the underscore Kate O’Neill. So Kate, thanks for joining us.

Kate: [00:05:08] Thanks for having me. Thanks for the definite article with the underscore. I really appreciate that.

Joe: [00:05:16] Yeah, it’s the audio emphasis. So cool. So we met you and got to know you through a recent ITX conference we ran, but you’ve really been in the tech space for quite a while and have a couple books out now that we’ll talk about, but could you just introduce yourself for the audience and let them know how you got started, where you are at now?

Kate: [00:05:34] Yeah, so I definitely have been in the tech space for quite awhile. It’s been 20 some years or whatever. I got into it almost by accident because I was a linguist by education. I was a German major heading up the language laboratory at the University of Illinois at Chicago when I saw the graphical web for the first time, and so that dates me right away. But it blew my mind, and I just thought this is, this is everything. This is going to change the world, this is just phenomenal outstanding. So I just, I learned that I could actually build a Web site for the language laboratory, and so I just got crazy with it, I got excited and did that. It turned out to have been very early on in the sort of departmental Web site process, perhaps even the first departmental website at the University, and it got noticed by a guy in Toshiba in California and eventually they recruited me to come out and build their intranet for them which turned out to be the first departmental internet that Toshiba had.

Kate: [00:06:35] So I got on this role of doing these kind of firsts. And it was all just based on just curiosity and passion and seeing the possibility of what technology could do in these contexts, you know, to sort of create connections for people and opportunities for people to engage with brands and with departments and all the different entities that were trying to reach their their constituency. So I did that for Toshiba and then fast forward a few startups, because I did a bunch of startups, and I ended up at Netflix in one of the first hundred people there and created the first content management role for them which was pretty awesome. It was a great place to be and I loved my time there. So that’s that, and then I got to see the early days of Netflix, like a fighting all-out bloody battle with Blockbuster, and winning. So that was great.

Kate: [00:07:34] And then fast forward a bunch more years, I started my own agency, [meta]marketer, doing analytics and strategy, and got to work with cool companies like Adobe, Symantec, the Grand Ole Opry…this is great stuff. And now I do professional speaking and I’m an author, as you mentioned. I advise companies, mostly Fortune 100, but also high growth startups, all kinds of companies, really, on how to keep the focus on the human even during big digital transformation efforts. So how to make their business more successful and scalable through the digital technologies while also providing what I hope people embrace as more meaningful human experiences. So that’s the long, long version of this story. But there’s a lot of fun twists and turns out there.

Joe: [00:08:26] No, it’s great, very cool. And I think there’s one Blockbuster left, hanging on.

Kate: [00:08:33] Is it in Oregon?

Joe: [00:08:37] Is it? I can’t remember, I thought Alaska, somewhere out there though.

Kate: [00:08:40] I think it might be in Oregon, and I think, Sean, you’re in Oregon right now.

Sean: [00:08:45] I am in Portland, so I’ll have to go look for that today.

Kate: [00:08:47] Sean that’s your mission.

Sean: [00:08:49] Just for nostalgic purposes.

Joe: [00:08:53] So your new book is called Tech Humanist. And you’ve had a book before, Pixels and Place. What kind of inspired you to write Tech Humanist though?

Kate: [00:09:05] Yeah, so when I wrote Pixels and Place, it was meant to be a focus on this whole connected space of physical and digital experiences and how things like the Internet of Things and wearables and sensors and beacons, and all that type of technology is really merging the experiences of online and offline and digital and physical, and how there really doesn’t seem to be as much of a distinction as there ever was. And I touched on automation, and I touched on artificial intelligence in doing that work, but as the next kind of year unfolded, during my keynotes and other interviews and opportunities to discuss it, I found that one of the things I kept coming back to was that the world is increasingly automated as well, and there’s increasingly an emphasis on how businesses use technology to scale what they’re trying to do. And a lot of what’s happening is not necessarily done with a mind toward how that’s going to play out when it actually reaches this kind of tipping-point of scale where the world around us is almost entirely automated. And what does that look like for being a human in a world that’s almost untried entirely driven by machines?

Kate: [00:10:19] So I think business can really use technology and be very successful with it. And I think as humans who run businesses, it’s in our best interest to think holistically about how to make that technology serve the business objective while also serving the objectives of humanity in a very integrated way. So that was the gist. I wrote a manifesto, as one does, in mid 2017, that was called the Tech Humanist Manifesto, and the response to it was just so overwhelmingly good, I just felt so encouraged that other people were kind of thinking the same thing, that feedback was, you know, “you articulated exactly what I have been trying to figure out how to say,” and it was just so so exciting to think that there was almost like a movement waiting to happen. So I took that momentum and did the work, put the research together, and put this book together.

Kate: [00:11:14] So my hope is that what it does is create a model, a methodology, kind of a framework that business leaders can use to think really well about the strategy that they’re using, create is great, you know, profit driven experiences, or sorr, or profit maximizing experiences, that are good for the business, that that make everybody successful and happy within the business, but also that are creating these meaningful experiences for all the humans that are inside and outside the company interacting with those experiences.

Sean: [00:11:48] So I would argue that your timing for this kind of thought leadership and thinking is perfect.

Kate: [00:11:54] It’s great to hear.

Sean: [00:11:56] In theory, when you think about the evolution of how humanity has used technology to serve itself right, how we’re using it, you use the word digital transformation disruption. There’s a ton of words being thrown around the business world over the last five to ten years around what this actually means, but it is all about changing the way businesses operate to focus more on solving real problems for real humans in real context, which is what your thinking is all about. Stop me if I’m wrong.

Kate: [00:12:26] No, no that’s that’s a great articulation. Can you just come around with me and do that explanation everywhere we go?

Sean: [00:12:33] Sure.

Kate: [00:12:34] I think that’s a great opening for me to clarify too that I think digital transformation is this funny term, too. Right, like we’ve been talking about it on some level for maybe a decade or so. You know, it’s kind of surfaced here in there and the CIO spaces, and you know, kind of I.T. infrastructure spaces. Everybody knows that there is more and more need to digitize the operations of the business, the logistics, to kind of connect the dots on the supply chain, make sure there’s visibility and transparency through the supply chain and through the enterprise. And there’s incredible efficiencies that come from that.

Kate: [00:13:13] And then the opportunities that happen, you mentioned disruption too. The opportunities that happen when you think about platform disruption and the app driven service economy. The what can happen with using digital tools and data to try to surface opportunities that exist in marketplaces. There’s just a ton of rich opportunity there, both to make money and to make human lives more convenient and to make some interesting quality-of-life improvements for for some human beings. But I think where we end up too, one on one hand, one of the problems is I don’t think that some of those solutions necessarily look at the whole picture of humanity and aren’t necessarily thinking about all of the humans who are affected by some of those disruptions.

Kate: [00:14:02] And the other side of it is that digital transformation is almost like a misnomer, right. Digital is kind of missing the point. It’s really this data that’s driving this transformation, the underlying data that connects all these different pieces of the operation. And it’s the data that comes primarily from human interactions with those systems. So it’s us. It’s our human data, it’s our behaviors, and it’s our preferences and our purchases and our movements through space and all kinds of our behavior that’s generating that data that business is harvesting to make its decisions, to craft its algorithms to drive these things forward. And the important thing about that is to respect the real humanity that’s part of that data. So to create these experiences that acknowledge, and pay tribute to, the rich humanity that that data represents. So even when we talk about digital transformation, we’re really talking about data transformation, and fundamentally, we’re talking about human data that’s part of that. So every discussion about business, digital transformation, I feel needs to have a kind of core understanding about how much humanity is part of that discussion. And I think it’s very easy to abstract away from that in those discussions. So I just want to infuse that humanistic thinking in the boardroom at the moment of those decisions.

Sean: [00:15:34] That’s great. I think another way to frame that might be to say that it’s not really a digital transformation it’s more like an experience revolution, like we’re really…

Kate: [00:15:42] Oh man you’re good at this!

Sean: [00:15:45] Really, thanks. The transformation we’re seeing is around the focus on how we’re solving these problems in very technical and meaningful ways for people and that’s what we have to figure out. So our podcast here is really about building software products to interact with humans in solving those problems. So we want to try to get a little more tactical with you and get some ideas from you that might spark some of our listeners, our community, to build better human products. So do you have any specific advice for tactics, things that we might experiment with, or ways of thinking that will help us build more tactical human connections into software?

Kate: [00:16:31] Yeah I think there’s a number of specifics, or tactics, that I can suggest. And one way that I frame this is, in Pixels and Place actually, I introduced the integrated human experience design methodology, which my friend Jeffrey Zeldman lovingly calls IHED. I was talking a lot about the concept of integration as a whole is talking about adaptability and kind of learning from your iterative process, a whole bunch of other constructs that are part of that.

Kate: [00:17:08] But the thing that I think is the most tangible sort of specific takeaway that product thinkers can think about is how to find the meaningful metaphors of the brand and of the interaction and then how to make the experiences that are part of the product and part of the the extension of the product, all the interactions and micro interactions in and around that product, how can it resonate with those metaphors? How can it dimensionalize those metaphors? So, you know, even like a micro copy that happens within interactions, or onboarding that happens to a product or to an app, are really critically important in establishing that there’s this sense that there’s a shared understanding between the company or the brand or the product and the user of the product, the human who is the user of the product, right, that there’s some sort of shared understanding. Because a shared understanding is the key here, I always go back to this model of communication, that communication kind of fundamentally exists in three parts. It’s what the speaker is trying to communicate, the message itself, and then what the listener receives. And anywhere that there is an overlap or a shared understanding between those three parts and that’s where meaning exists.

Kate: [00:18:32] So I feel like that’s a very transferable model to thinking about product to thinking about experience design to thinking about anything at this kind of tangible level where you’re actually out there trying to create the things that people are going to interact with, and thinking that there’s an there’s an intention behind it, there’s what the business is trying to do through the product, and then there’s the actual kind of interface itself, the product itself, and then there’s what the person who is using that product experiences of it and like what their sense is of the whole thing. And you’re trying to create as much alignment there as you possibly can. And part of what helps do that is this dimensional thinking, this way of of creating this sense that the metaphors are intact and you’re enriching them as much as possible.

Kate: [00:19:22] One of the examples that I think helps clarify this is, one of my favorites is really Snapchat and their launch of the spectacles product. So you could argue whether or not it was a raging success, I mean any sort of profitable or a revenue centric measure I think, you know, it looks like a success in that in the sense that it drove tremendous amounts of adoption and traffic and so on. Long term, who knows whether that’s going to continue to be a product that survives, but the understanding that they had of what they were trying to create and the sense they were trying to create in the people who were using and buying this product was really intense. Like they understood that they wanted to create these standalone store fronts. I happened to be working on a project that was a block away from the one in New York City so I got to walk past everyday that it was happening, which was a couple of days I guess. People lined up around the corner, around the block, from this storefront, and literally every time I passed (a few times a day) people were snapping selfies of themselves in the line and it was literally a spectacle. It was something that people were excited to share, you know, in this sort of selfie context, this Instagram-able, Snapchat-able, kind of context that we all so familiar with now, the “share or you didn’t experience it” kind of existence. So they were doing that. And then of course, you know, if you actually went into the storefront and interacted with kiosks, it was a whole other thing. It’s very visual, it’s very novel and original but it was like no sort of typical storefront where you’re looking at something on a shelf and going to a counter. You’re going to a kiosk and the kiosk is there to guide you through the process of buying, show you options, and even that, taking a picture of you in the kiosk, it’s all still focused on this photo-oriented experience of the world. So in every possible way, they just did such a brilliant job of thinking about what they were trying to have people experience and making it as dimensional and rich as possible. So I mean, I think it doesn’t have to be a profound interaction. It just has to be one that actually connects in a resonant way with the people that it’s trying to connect with.

Joe: [00:21:54] Yet it makes a lot of sense, and you know, you mentioned the copy in your product, for example, over the last several years we kind of see like 404 pages having a fun copy and actually like acknowledging I “oops something’s wrong, we’re not just showing you a nasty error, let’s help actually get where you wanted to go,” and have it understand that you’re a person there. Because we’re always talking about the users, or the customers, or the clients, or things like that. At the end of the day, it’s always people, there’s a person that that’s talking about. So I think the copy is a great example. We see designers, product people, business stakeholders, all rallying around the copy more and more nowadays rather than just try to cram it in at the end once a product is built.

Kate: [00:22:39] Yes. It’s so funny you mentioned the 404 pages, I actually had that in my notes to prepare for our discussion today and I hadn’t quite touched on it yet. But that is exactly right that there’s this whole 404 phenomenon, or page phenomenon, that injects personality into this error page is something that, even then as you know, there have been roundups of like the best, most creative 404 pages out there, which is hilariously absurd to think that people are like going out of their way to go visit error pages on websites just because they express some element of the brand’s personality so well.

Joe: [00:23:16] It’s kind of a good test if the company quote unqoute, gets it, in terms of communication and knowing who you are. But so I guess without, you know, calling out any companies specifically, unless you want to, but any reasoning or kind of traps you think companies are falling into where they’re missing out on this, they’re just kind of not hitting the mark. Is it just laziness, they don’t know what they don’t know? Why is this still happening nowadays and how can we avoid that?

Kate: [00:23:48] I think I probably won’t call anyone out unless something comes to mind that seems like it’s a really specific, great example. But I think in general, what I seem see happen is that companies seem to go off the rails when they start trying to be all things to all people in some way. Like it’s like they’ve lost track of who it is that they’re really trying to be connected with and they’ve spread the spread themselves too thin, or they’re going after some kind of growth that isn’t meaningful anymore, that doesn’t of keep them in alignment with who is really kind of resonating with their product.

Kate: [00:24:25] So I mean, it’s a pretty natural tendency, I think, to want to broaden your scope of influence, and widen your reach, and make more money, and all of that. That’s totally understandable and it’s certainly the impulse that’s driven in corporate America through the sort of overriding capitalistic model that we work within. But, you know, sometimes adding that one feature or that feature set or making that one spin-off product model or whatever it is, is just enough to water down the alignment that you may have had with your core audience who just loved you, who totally thought, you know, you got them, you understood what you what they were about.

Kate: [00:25:10] So I think that one tendency that I would really discourage companies from playing to is, if you need to go deeper or broader, or sorry, I guess I should say that the other way, if you want to go broader, probably the better option is to think about going deeper. Like how can you add more value to the people that are already your customers so that they are going to tell other people who are like them, who share some sort of attribute with them? How can you find that sort of dimensional connection that there’s like an affinity attribute that if you add a feature to your product that’s going to add value to your core users, it’s going to bring in this additional set of users that are excited about what you do. And I think that’s a much richer experience for everyone and a much richer opportunity if you can find it and I think that’s just the hard work of it is trying to find what that what that core feature set is, what the expansion possibilities are in ways that don’t diminish the meaning that’s shared between the company or the brand and through its product with the humans that interact with the brand through that experience.

Sean: [00:26:32] That’s a great answer Kate, thank you for that. You’re really speaking our language here because we’re all about building loyalty and advocacy and I think that’s part of the formula, is really, once you’ve landed a customer how do you make that customer a customer for 100 years? Make sure that you’re taking care of them and building real loyalty through value.

Kate: [00:26:55] Well it”s what you see happen when you do it right. You know, you see the metrics go in the right direction. If you’re doing the right work of connecting with the audience who cares, the people who are really getting value from your product, then you should see retention numbers and loyalty numbers go up, you should see churn go down, you should see cost acquisition go down, you should see all of those metrics moving in the right direction because that’s what happens when you actually offer value.

Kate: [00:27:31] And it’s so funny because I think, I choose this language of meaning and meaningfulness to to speak about, and I think it sounds like I’m talking about something very like sort of touchy feely and feel good, and it, is but it also is a very practical kind of idea because it translates into the kind of results that any business leader wants. If you want the growth, you want the loyalty, because it’s more profitable to keep a customer than to gain a customer, right, like generally speaking, or to have to go out and buy a customer, in a sense. You want the ability to see churn go down. You don’t want people leaving. So you can figure out these ways that you can model what’s meaningful and then actually figure out the data model too that tells you for sure that you’re making the right decisions, that you’re doing the right things, and that there’s something that’s really happening through all the different touch points that you’re having with people and that’s bringing you back insights about how you can continue to improve and add value for people, but that you’re that you’re doing it right, that you’re adding meaning. And hopefully that means that your work is more meaningful too.

Sean: [00:28:48] You touched on something there that reminded me of a quote I wrote down from Pixels and Place. I’m going to read it real quick, ready. “Don’t allow your algorithms to be subservient to the profit motive.” And I love that quote because it pretty much sums up everything you just said.

Kate: [00:29:05] It does, it is so much more succinct than what I just said.

Sean: [00:29:08] It’s your words. But anyways, we try to we try to live and build our entire business around that, helping companies to figure out, how are we going to take your clients, your customers on this journey through software? And you also touched on KPI so I want to talk about that real quick. We have a framework that we call the loyalty ladder, and it’s really around setting three core KPI’s for every software product. How is it building trust? How is it building loyalty? And how is it building advocacy? Because we believe that if we can get every user on that journey, trust loyalty advocacy, that we will be on the way to building that hundred-year sustainable relationship where they’re going to come back to our software product forever and in the future to solve those problems. Have you seen any other powerful KPI’s around this stuff, around meaning and building real human relationships through software?

Kate: [00:30:03] So I think that the work that I’ve seen that really seems the most meaningful and exciting to me about KPI’s and metrics in general with product work and with strategy and design work and all of that kind of tactical, hands on, work of creating the experiences, tend to be either proxy metrics for something more abstract. Like meaningfulness is something very difficult to measure, right. It’s going to be nearly impossible to come up with some measure that truly represents meaningfulness. But you can come up with proxy measures that within the context of what you’re doing give you a sense that you’re on the right track.

Kate: [00:30:46] And I think about things like memorability, which is often measured by how how well somebody retains your brand when they see it again, or you just mentioned trust and advocacy. Advocacy is a great one, how often are people referring you? How often are they sending other people your way, or can you see what kind of chatter is going on in social about you, and is it positive? So there’s a lot of that kind of stuff, but I also come back to, you know, Jim Collins and the Good to Great sort of framework of the the hedgehog principle, and what really is unique about what you’re doing? And being able to come up with this kind of complex construct. Jim Collins in Good to Great talked about the profit-per-x, you know, the denominator that’s really your thing, like what you’re doing that’s unique in some way.

Kate: [00:31:49] And I learned, I was at magazines.com heading up customer experience and product development there, and we did this work of trying to come up with increasingly nuanced understanding of what we were really optimizing for. So for example, when we first started on this work we kept talking about increasing the number of sales, the units we were selling, or talking about how much money transacted through the website everyday. But after a while, through some discussion, we started talking more and more about the subscriptions because that was more the meaningful unit of what was actually happening. People are buying magazine subscriptions, so in order to make this more contextual and relevant, let’s not talk about it as just money or just orders let’s make sure we’re kind of bringing in that one little nuance of subscriptions. But then there was another facet that was like, “well really we’re talking about, if we want to optimize this for a customer-centric point of view, we need to switch that and not talk so much about subscriptions, but rather about subscribers.” How many subscribers do we have interacting with the site everyday and coming back, and what’s happening with that? What’s our profit per subscriber, not our profit per subscription? And then one further refinement was, you know, a lot of what we understood about the model of our business was that it was really only successful, and we had really only done our job, if people were renewing the subscriptions that they bought from us. So the profit per denominator became renewing subscribers over time, profit per renewing subscribers.

Kate: [00:33:30] So you can see how that evolution is showing this process of becoming increasingly customer centric and increasingly a finer point on getting that nuanced understanding of what’s really making the meaningful connection and how can even just that little bit of thought, that little bit a shift toward that language and that model of the measurement, can make such a huge difference in the way that you’re aligning programs, and what priority you’re assigning to features, and what kind of road mapping you do to make sure that you’re getting the right things to happen that moves the needle for the company and also creates the most meaningful experiences for the people.

Joe: [00:34:12] Very, very cool. So putting myself in the shoes of a listener, let’s say I’m a CEO of a small company, or I’m in charge of product at a company, and I’m listening to this and I’m like, “wow we’re really messing this right now, we’re totally off.” Because it’s really overwhelming to think about how to fix all this at once, what would be like a good baby step that they could take to get started? Like, if they’re done listening to this, or tomorrow, and how can they maybe like crawl there rather than run there?

Kate: [00:34:44] You know. I actually think that the best step that anybody who finds themself in that sort of situation can do is, it sounds like a big macro step but it’s actually something that brings so much clarity that it’s worth doing, and it’s not that hard, trying to come up with some concise articulation of the company’s strategic purpose. That’s what I keep coming back to in my work is that the shape that meaning takes in business is purpose. And it’s this aligning kind of force that when a company understands what it is that they exist to do and are trying to do at scale, then they can actually really make much better decisions about the priorities that they set and the resource allocation that they have and everything that kind of flows from that. And it even informs digital transformation efforts because it has to be understood at this top level of strategy before operations can really align around it, and brand and marketing and experience design can really align around it, and data strategy flows through it, and technology deployments should really be informed by that same sense of strategic purpose.

Kate: [00:36:05] So all of that stuff eventually does need to be kind of in alignment and thought through, but even just doing the work of trying to come up with that one concise articulation of strategic purpose can make an enormous difference on the remaining decisions you make that day and in where where you put emphasis in prioritization and goals, even. So, you know, to be clear about what I mean by strategic purpose, I always love to use the example of Disney theme parks because they’re sort of the gold standard of strategic purpose, articulation. And they talk about “create meaningful experiences” as being their strategic purpose and everything kind of flows from that, and you can see how that works operationally. If you’re someone who’s on the front lines of customer service it makes a big difference, but even if you’re if you’re doing product development work that’s a little longer range or if you’re doing operational work that’s an financial or accounting or whatever, everything can come back in some dimensional way to that understanding of “create magical experiences.” So it’s a very helpful aligning philosophy that guides the company and guides the efforts within the company. That’s what I would say that people that could stand to do is spend 20 minutes today just trying to put into three to five words what it is the company exists to do and is trying to do at scale. And even on a product level, I believe that’ll really make a difference in a very short amount of time.

Sean: [00:37:44] I love it. Good advice. So some specific questions, and then we’ll wrap up. And by the way you ruined my last question; I was ready to ask you what company could you give us as an example? That was great, Disney is a great example.

Kate: [00:37:58] Oh, yeah, yeah. I’m sure I can come up with a few others, but were you going to ask another question?

Sean: [00:38:05] Yeah. Give me an example of a specific software product that you recently used that you felt was connected to the meaning for the company that it was built for.

Kate: [00:38:16] It is a tough question. You know, it’s funny the way my brain works, I don’t often hold on to specific examples and I have to go back to the writing I’ve done. I don’t have one off the top of my head. I think there’s been quite a number of of companies and brand experiences that I’ve had in the last few weeks or months that have seemed like, “well this one facet of the interaction is really good, and I’m really pleased that they made this thing so nice.” But then usually there’s something that’s like, “ugh, it didn’t get the rest of this right and it would be nice if the attention that was paid over here was also paid over here.”

Sean: [00:39:00] So movement in the right direction, but nobody’s got it quite perfect so that it’s become memorable. Which is great for me because it mean there’s a lot of opportunity in the world.

Kate: [00:39:11] Yeah there is a lot of opportunity in the world.

Sean: [00:39:14] It’s great.

Kate: [00:39:15] You know, you were talking about companies that do a great job of keeping meaning in the forefront of their strategy and how I used Disney as an example, but I think another one is, the kind of classic is, Southwest Airlines has never failed to come through in dimensional ways about meaningful strategy. And it goes through every facet of their brand and their operations, their experience. I mean they think about the way that they even set up the company to be operationally efficient by having all the same airplane models so that pilots could fly them all, so maintenance crew can take care of them all. They do these short hops, is their model, which means that they can they can keep their schedule very on time. Everything is so streamlined and so thoughtful, and of course the value proposition is that there a low cost airline, but that’s not really what’s happening at an experiential level. Because what’s happening is they’re sort of handing that over to customers kind of with a wink and a nod, going like, “you want a low cost airline, right?” Like you’re willing to trade frills like assigned seats and meals and things like that for being able to pay a little less and have a more no frills experience. And we’ll have a little fun with that, like we’ll do the safety announcements in a rap, or something goofy, and those stock ticker is gonna be LUV, luv, right. Like it’s all so richly understood at a really really holistic level. So I mean, I think there they have that sort of gold standard that I come back to you when it comes to thinking through every facet of how the company comes to life in different ways for people as they interact with it. And even their app, everything about what they do is a rich translation of that understanding. So there’s one great example for you.

Sean: [00:41:21] We usually ask our guests, Kate what’s one number one book you would recommend that people in our audience read? You’ve mentioned Jim Collins and his hedgehog principle and Good to Great. One of my favorites is The Speed of Trust, great book. Is there any other books that you’d recommend? Obviously your own, which we will recommend as well.

Kate: [00:41:51] Of course, yes, mine. I’m very excited about about my new one being out on the market and everything. But, no, you guys were talking about your loyalty ladder and the trust and advocacy, and advocacy, you know, of course translates into something being so remarkable that it’s worthy of discussing with people. It’s like worthy of being a representative of. And I think the best sort of standard bearer of that and that the people who are having the most informed conversation about that right now in the market that is current to today are Jay Baer and Daniel Lemin with their new book Talk Triggers. And the best thing about this book is that it has alpacas on the cover, so you can’t go wrong one way or the other. Absolutely Jay Baer and Daniel Lemin’s new book Talk Triggers I would highly recommend people look into to get a real understanding of that word of mouth marketing and how really doing the right things, doing things right, and making the experiences as dimensional and rich as possible will translate into this advocacy that you talked about.

Sean: [00:43:02] Alright, you sparked one more little thought here. You mentioned you started out as a linguist which I find fascinating becauses words fascinate me, because I believe that at the core of all culture is really words and how we define them and how we use them, right. And advocacy is one of those words that’s very important to my business, to what we do here at ITX. Here’s how we define it, I’d love your feedback on how we define the word advocacy. I believe advocacy is when your customer is willing to invest in the future of your product or service. Not necessarily in your business, but in the product or service that you’re offering, they’re willing to invest in that. And that could come in many forms, it could come in the form of a referral, as we would traditionally think of as an advocacy event, but it could also come in the form of unsolicited feedback. When they’re willing to step out of their day and write you a note, put thoughts on paper that is constructive, I’m not talking about complaints, but constructive feedback that will help you improve your product or service, that’s like gold to your business.

Kate: [00:44:02] That’s true.

Sean: [00:44:02] And that’s a sign of a real advocate. So we’ve catalogued a bunch of behaviors that we believe are advocacy behaviors and they all kind of boil up to this investment in the future of your product or service sort of definition. What do you think about that?

Kate: [00:44:17] Well I love it I think it’s a beautiful kind of way to think about the value that’s being transacted between brand and customer, right. I think a lot of people think of there being value that happens one way, right. Like if people pay you money, that’s the show of value. But you’re talking about a much more nuanced understanding of value that says time and attention and interest is also a form of value that humans show each other as well as the brands that they enjoy. So that’s a really nice dimensional kind of approach to that. And I love too that it’s investing in the future of the product and service because there’s a sort of deferred belief going on there, right, like you’re saying, “I believe in you enough to think that paying a subscription to you or giving you this feedback now is worth my trust in doing so because you are going to translate it, you’re going to to turn that around into something that’s going to improve my experience and others experiences and it will have been worth my time to do so.” So I think that that’s a beautiful articulation of that.

Sean: [00:45:35] Thank you.

Joe: [00:45:35] Cool, so one more question and then we’ll wrap. So at this point, would you consider Sean and I to be Tech Humanists?

Kate: [00:45:48] Would you like to call yourselves Tech Humanists? Because that’s what matters.

Joe: [00:45:52] I’m changing my title right now.

Sean: [00:45:55] Of course you would.

Kate: [00:45:56] So this is a really fun thing that’s been a change for me is that when I first started, when I wrote the Tech Humanist Manifesto, I had been sort of using this term loosely for myself for a while, and I decided with that manifesto that I was kind of boldly declaring that I was a Tech Humanist. I hadn’t seen anyone else use that term, so I thought like, “this is really important. I’m putting my stake in the ground.” But it became very evident to me with the feedback I got from writing that and sharing that, that other people felt that too, and it was so important that this not be my term, that it be our term. That we are all tech humanists if we want to be, and this is something that I’ve said many times in front of audiences, when you see this book come out and it’s called Tech Humanist, I want you to feel that I wrote it for you, not just for me. Although I did write it for me, but I did write it for you. I wrote it so that you would see your name, your description, in the title of the book, and know that this book is about you and how you can make the world better for all of us.

Kate: [00:47:03] So I think it really is, you know, Sean, I think you’re the one who said early on that this is the right time for this discussion, and it really feels like it is. There’s such a tipping point about to happen with automation and with artificial intelligence and every little bit makes a difference. Every little bit of everybody within their product teams, within their companies, within everywhere that they are part of making decisions about data and digital technologies and experiences, every little bit of recognizing the full complement of humanity that that’s part and parcel of what goes into that experience and who is going to interact with that experience, all of it is going to make a difference at scale at some point in the not too distant future. So yes, I absolutely want you to be Tech Humanists right along with me. So thank you for asking.

Sean: [00:47:58] Alright. Well thank you, Kate. I think that wraps us up. Your work is profound, it’s refreshing and different, and it’s really important.

Kate: [00:48:06] Thank you so much.

Sean: [00:48:07] For the world and especially for our industry. So keep after it. And for readers, or listeners, rather go Tech Humanist if you do nothing else.

Joe: [00:48:17] Where can they find it, Kate?

Kate: [00:48:18] Right now just Amazon and then it will be available in bookstores everywhere pretty soon. So yeah, check Amazon, and absolutely spread the word. I’m so excited to share this with the world.

Joe: [00:48:31] Awesome. Well thanks so much for joining us today.

Kate: [00:48:33] Thank you both.


04 / Creativity in Technology


The best way to solve a problem is to first examine the people who have encountered it instead of looking at the problem itself. Doing so, allows you to get a different point of view. Through creativity, in-depth research of the people using your product, and technology, teams can exponentially improve the way software products provide value for users.

In this episode, Joe and Sean talk with renowned artist, Sebastian ErraZuriz, about how he’s shaking up the art world and training minds to be creative in technology.

Read our blog post

About Sebastian

Sebastian ErraZuriz is a Chilean born, New York based Artist, Designer, and Activist who has received international acclaim for his original and provocative works on a variety of disciplines, blurring the boundaries between contemporary art, tech, design, and craft. His exclusive masterpieces are avidly acquired by art collectors and museums. His work is always surprising and compelling, inviting the viewer to look again at realities that were often hidden in plain sight.

t i


Joe: [00:02:08] Welcome to our Momentum podcast, here we talk about software products and innovation and how to build momentum, which you’re familiar with as we’ve had several conversations about that in the past. Thank you for joining us. Can you tell us a little bit about yourself and about your role?

Sebastian: [00:02:25] Sure. So I’m a bit of an activist now, apparently a tech entrepreneur. I’m Chilean, based in New York for the past 12 years and basically I have a studio in which we work on everything from the interior of a private jet to a giant public artwork in Times Square, or a 3D printed set of shoes, all the way to, I don’t know, some basic design product.

Sebastian: [00:03:01] Now over all of the time in which I’ve been working as a designer, an artist, it’s always been driven by the need to connect, the need to invite other people to look at what they’re missing, to see what’s obviously always been there, but we pass by and somehow inadvertently can’t notice it. And it’s happened to me, like many other creators, but we feel we’re no longer being able to make contact. That no matter how creative we try to be, whatever we throw into the world gets absorbed into this black hole of information and it immediately fizzles up and disappears. And so suddenly the role of the creative, of the art, of the designer is being incredibly frustrating and unsatisfying. And you feel that you can no longer make a difference.

Sebastian: [00:03:58] And it is within that scenario that I’m suddenly forced to get into technology as a way to use the leverage of the exponential reach of technology to continue to communicate and connect. And that’s when slowly we pass from the studio that was doing 3D prints and using seven axis robots and that end of the tech side to, actually, “maybe we need to hire a coder, and what if we could develop this bot, and what if we could think of machine learning for that problem?” All the way to, you know, “you what guys, we need to hire a team of engineers to star building software.” It’s been a very interesting path in which arriving at technology has it been through a necessity and always driven by a need and an urge to connect as an artist from an existential standpoint.

Sean: [00:05:10] That’s amazing. You’ve had an interesting journey for sure. Our goal here with this podcast is to kind of try to tap into some of your experience with creativity. And I know it shouldn’t be a hard conversation to have, at least from our perspective. I think it comes to you easier than it comes to most people, but how do you tap so how do you tap into creativity? And I find it extremely interesting that you’re moving towards technology, as, if you think about most industries, they’re all becoming more and more. We’ve had some of these conversations in the past Sebastian, but lots of things are moving towards the software and A.I. is coming in, you know, all sorts of things are going to be changing even more than they are today. So how do you tap into creativity, so to speak?

Sebastian: [00:06:03] I think creativity has long been seen as almost the quality that you either have or you don’t have. There’s a whole mystery about how creativity works. And there’s always been this cliche of creative people as being unreliable and a bit of a mess, and the roles of artists that when they’re feeling passionate, grab red paint and slap the canvas and when they’re angry they’ll go rip that same canvas apart and so on. And it unfortunately has created an idea of what creativity is that separates it from the science. Separates it from the aesthticism and professionalism that any other discipline should be tied to.

Sebastian: [00:06:58] And as a professional artist and a professionnial designer, someone who’s been raised in this, trained in this since I was 4 or 5, I generally believe that creativity can be trained. And creativity is a discipline that is completely necessary, vital today, and that it is at reach of anyone, and no matter what your current level of it is, it is your obligation to train your body your mind to be at its best possible level and in an era of interconnectivity where everyone has access to the same information. The only resources we really have in order to compete is be able to come up with original combinations of that information.

Sebastian: [00:07:50] And so I focus on that, and in a way I’m a bit of a hacker. A hacker of systems, a hacker of logic, and I always tend to give creativity and the unconscious side, the intuitive side of our brain the higher respect over the rational side. In a way, I feel our creativity is almost like an A.I., an algorithm, that’s capable of processing a lot more information than we normally can, and maybe through algorithms that have been developed in it in their own iterations where it’s very hard to understand how we get there. Nevertheless, the quantity of information that can be processed by our intuitions is exponentially larger than the quantity of information we can try to process rationally.

Sebastian: [00:08:56] So both in my personal practice, in my everyday routine, I train myself to be as creative as possible. And it’s what I’m always inviting people to do. And ultimately it’s my biggest service. I will accept any challenge to take on any creative problem no matter how hard it is because I trust that that intuitive algorithm, when used properly, could bypass a lot of what we’ve rationally always done and it would allow you to be in the position of theis underdog, this David versus Goliath that actually has a truly unique tool that can make an enormous difference. So I see creativity as the key to pretty much everything, especially in this new era, and maybe even the final area of human intelligence that A.I. will get to. So that would be most valuable within the next 20 years before we get into some sort of, I don’t know, artificial intelligence peak.

Joe: [00:10:22] So you mentioned earlier the ability to be able to observe and see what’s there already, and we talk about that a lot when we’re working on software projects and are trying to observe our users, and you know, see what problems there are that are right there in front of us to solve. So in thinking about that, in innovation and creativity it’s not always about building the next Uber or the Snapchat, but just seeing what’s there and solving problems for users that they have workarounds for potentially, but I guess my question is, how do you just very simply define creativity and/or innovation? The words get thrown out around a lot.

Sebastian: [00:11:02] Sure. Definitely creativity is the ability to come up with original combinations based on the same set of known ingredients. So if we have pasta, tomato sauce, and ground beef, we can make a spaghetti bolognese or we can make lasagna, right. And it is in the role of the creative to have as many ingredients as possible. In the same manner, if you have a machine learning system, you want to feed it with as much data as possible.

Sebastian: [00:11:40] Nevertheless, it’s how we find the patterns in that data that we are able to find new original combinations that others haven’t spotted. And a lot of the time, those patterns are the most obvious. The ones that we don’t look at because we get them for granted. They’re the basis from which we start. And so often, the best way to start working is to do exactly the opposite of what has always been done.

Sebastian: [00:12:14] So I see it a little bit as, if you think of any problem in marketing and in branding in any area, just to give an example. That particular problem has probably been tackled by many, many people. That means that there’s a whole group of experts that have been looking very closely at that problem, almost if you can imagine physically surrounding it with their magnifying glasses on top of it. Now when you don’t sum yourself up to that group and do the thing that everyone else would do, which would be to try to squeeze in for a little bit of space to try and see a tiny slot of what everyone already is looking at, which undoubtedly will most probably give you have the same view that everyone has already seen and get the same conclusion. And if instead you start by simply looking at this bunch of people looking at something, you already start from a different standpoint where you’re already seeing not the problem, but first seeing how we’ve approached it and how we can look at it. It’s a pretty silly effect. There’s a whole bunch of people all crouched up with all their heads together, all trying to observe this one thing without really noticing that the way they are observing could maybe be different.

Sebastian: [00:13:40] And so I’ll give you a very simple example. I have the first door in humankind that has two viewers in it. Do you think of any single door when you approach a door and you ring the bell, in front of you there is a single viewer looking back and the person on the other side who answers the doorbell can go up to the door, close one eye, peek through the viewer, and look who’s on the other side. Now that view happens to be one single viewer. And none of us is a pirate, right. None of us has a single eye. It doesn’t make sense that we should be closing one. We’re not looking through a telescope. We’ve had binoculars forever. Why not have two viewers? So my door has two viewers and if it’s weirdly surprising and funny in its obviousness. People look at it, see this door with two viewers horizontal, one to another and they actually wonder if it works. And they go up to the door to check if they can actually see through it because they can’t quite believe that they’ve never thought about it before. Now the door has a practical purpose of allowing you to see in a better manner, right. But also it’s simply staring at you back in the face and telling you, “hey how about we open that other eye, how about we look again? And so that practice is what I do again and again and again in a variety of areas. And the process of looking at things differently can really be applied to absolutely any industry or any problem.

Joe: [00:15:24] I was laughing loud literally on mute when you’re talking about the peep hole on the door, that’s an amazing example. And I love the pasta example too, I’ll probably steal that one as well.

Sebastian: [00:15:37] It’s really weird. I had a museum show and we sent the door over; It was one of the pieces they wanted to show. And when they sent us the door back with the two viewers, we took it out of the crate, get it unpacked. And there was this smudge of grease between the two viewers, and I go like, “what’s that smudge of grease, how disgusting!” And we check and we realized it was people’s noses. And so basically for the length of the exhibition at the museum, there had been thousands of people that went up to the door, pressed their nose against the door, and looked through the two views with their two eyes to check if it really worked.

Joe: [00:16:23] Oh my God.

Sebastian: [00:16:23] And all of that nose grease was layered again and again and again. I painted over, we cleaned it up, because I thought it was a little too discussing, but I should have left it as the proof of our need to sort of see that things could have been different and changed.

Joe: [00:16:41] Right, evidence. That’s so cool. So something I debate with my my friends my colleagues a lot is, “how important is your physical environment that you are in when you’re trying to be creative?” Like do you have to stand in a room blasting classical music, or, you know, how important is that environment around you?

Sebastian: [00:17:04] I think that’s a really good point. Obviously our environments matter a lot on us in the same manner as our own physical presence matters a lot. It’s been proven, right, that if you raise your arms more and you walk upright you tend to have more positive thoughts and you carry yourself with more determination and so on. In a similar manner, if your space is very constricted, if you feel that you can’t move too much, to do too much things without being looked at, that naturally influences the way that you feel more constricted to come up with ideas. If the space allows you to feel more free, to maybe feel less judged, feel there’s more space. It also, I believe, allows your mind to feel more tranquil and more confident.

Sebastian: [00:18:07] Now I’ll give you an example. I, like everyone else, will wake up in the middle of the night and have an idea, or I’ll be in the shower and have an idea, or walking around then have an idea. Nevertheless because I do this for a living, if you guys had hired me to come up with a solution with a design, with a campaign, I need to comply with a timeline. And occasionally I’ll need to force myself to come up with an idea. Now, to force myself, what I do is similar to what some of you might do if you meditate, right. I try to place myself in a situation where my mind can be blocked so the unconscious side of the head can do other types of work. For that, I personally need loud music. I personally need to be tapping my foot, so I’m tapping, using a pulse, and that pulse almost works like an um where I’m keeping my attention on that. And it’s almost like being a bit of a medium. I’m really trying to block out any thinking and allow my head to come up with weird associations. And I have pen and paper in front because the fastest as I can sketch to put down ideas and I just throw everything out that comes up to me. And this process could last an hour, an hour and a half. And it’s almost like in a movie when the protagonist opens the time portal and this weird circular thing opens up and they’ve got a finite amount of time to pass through it, right, and then the time portal closes. Here it’s almost like you have an unconscious portal, or an intuitive portal, where you’re pulling out much information as you possibly can as fast as you can without judging it and capturing it all. In my case, on pen and paper, for someone else it could be typing it in the computer, or on the phone, and then at some point you’re exhausted. At some point you’re done and your head just wants to go other ways. In the same manner as some point you can no longer continue to meditate and you need to end that process.

Sebastian: [00:20:34] Now, to be able to be something like that is quite vulnerable. Because first, my eyes are closed. I don’t know much around me. I’m moving my arms a lot. I probably look quite silly and quite vulnerable. I happen to be in charge of my studo, I’m the boss. So in some manner, it’s already a lot easier for me to be there. Nevertheless, I still do this process when everyone has left. It’s what I do when I’m alone at the studio. I have a hard time doing it otherwise. And so, I guess what I’m trying to, in a very long description, explain, is that the environment is vital. The need to feel protected and tranquil and feel that you can breathe, that you can think that you’re not restricted or judged, is super important because the process of coming up with intuitive, alogical associations that rationally maybe don’t fully make sense at first requires that leap of trust and requires that space.

Sean: [00:21:50] That’s fascinating. Thank you for sharing that, by the way. That was very personal and very deep. I think in our world, when it comes to software, I agree with you that creativity is the key to everything. That was your quote from earlier. Especially as the world is moving more and more towards technology, and like you said we all have access to pretty much the same information and ideas that are already out there. So if we’re going to help our customers build software that’s going to help them compete in new, unique, and different ways, creativity is the key to everything.

Sebastian: [00:22:27] Exactly.

Sean: [00:22:28] So you mentioned at some of your tactics for how you trigger creativity for yourself. I think that there’s a lot of things in there that are probably very unique to you in how you’ve sort of evolved through your amazing life. I’m giving advice that might be useful to software development teams that are trying to work together to solve problems. Now you’ve been through one of the workshops that we execute. Do you think that there’s any sort of validity to following some sort of guided process for helping figure out solutions and coming up with ideas? What do you think about that?

Sebastian: [00:23:10] I’m a little skeptical towards team development of the creativity. I think it’s similar to when you get the director of a movie to decide that he no longer wants to be in charge of the authorship of that particular film. The leads of the movie and the rest of the production team decide to do a mash up of a variety of creative and carry that through. Normally what tends to happen there is that you get a relatively generic commercial movie that fulfills a lot of the base requirements that you would expect, nevertheless, has very little soul. And that, I think is mostly because, in order to be able to make a bigger breakthrough in creativity, someone needs to take a big risk.

Sean: [00:24:10] Makes sense.

Sebastian: [00:24:12] Today the only bet you can do if you’re in a company is to innovate and to break through of what’s expected. That is the only rational, safe move for you to do. Now not everyone can do that, and especially if you’re nervous of the approval of others, especially if you’re not the boss and your job is on the line, especially if you feel you don’t have complete authority to present a paradigm shifting notion that often can sound stupid. And so I think as much as it is vital that teams can work creatively. Checking each other’s hypotheses and theories and of moving together toward strategies with solving problems, then each one can contribute different elements. Having a leadership that is trained in creativity that is not afraid to present ideas that are somewhat crazy, that is willing to put those out there without being nervous about being judged or their job and that even maybe has a certain level of pride in coming up with these contrarian notions, is vital, right.

Sebastian: [00:25:42] You should have in your team someone who just does this, and that that’s their role, and they’re expected to be the contrarian, because that in of itself almost is a feature within the process. It guarantees that there is an element of measure. There’s a forced different perspective and that perspective comes from a safe place. Now generally, it’s best if this is not the owner of the company doing it. Generally it’s best if it’s someone who doesn’t feel that the results financially are all going to depend on this decision.

Sebastian: [00:26:30] One of the beautiful things for me of being a designer artist is that at the end of day, most of the jobs I take, I almost take them for honor before money. I would say I’m not for sale. Now I’ll think about your problem and I’ll give you what I believe is the solution you need to take. If you don’t take it, that’s your problem. And if if you if you think it’s dumb, it’s it’s fine. I couldn’t care less either. I’m not going to give you a mediocre idea that you probably expect because, I, first and foremost, want to feel satisfied almost at an existential level with what I contributed. And hopefully because of that I get well-paid.

Sean: [00:27:20] That’s amazing.

Sebastian: [00:27:21] But that’s a freedom, that in a weird way, my upbringing as an artist has allowed me to have today. Nevertheless it’s the freedom that is key and it is vital and that should be incentivized at least in a couple of members within every team.

Sean: [00:27:41] I like that. You know, the underlying theme that I’ve heard you use and say at many points is safety.

Sebastian: [00:27:48] Yes.

Sean: [00:27:49] And people need to feel safe in order to thrive and ideas are never, I totally agree with that point, people are never going to feel free or have their ideas flowing freely unless they feel safe. So we have to create an environment of complete and thorough safety, especially around that sort of stuff. And I love the idea of having sort of people that are almost tasked with defending the ultimate goal and being the contrarian when they have to be a contrarian, because it’s easy, especially in the corporate world that we have to exist in the business that I run, that we’re constantly being pushed towards mediocrity. Because it’s safe. Like we know that our competition is doing this. So if we meet these requirements and we build the product in this way, we know that the competitors are doing it and they’re doing something righ,t so we can be safe by doing the same thing.

Sebastian: [00:28:50] That’s the problem and that’s what’s shifted. It is no longer safe. And people are having a hard time realizing that it is no longer safe to do what’s always been done. But the reality is that every single industry is being disrupted or has been disrupted over the past couple of decades. So no matter how big your industry currently is, if it does not have the practice of distrupting itself, it will be disrupted by someone else. So the only safe thing for any company to do today is to either hire the most disruptive person they could think of or have someone within their team paid to be disruptive.

Sebastian: [00:29:38] Because if I’m going up in competition with other people and the salaries of my team are on the line, if my investors are on the line, I want to know all the data. I want to know all the info. I want to know all the possibilities under which this could happen in a different manner. And what are all the possibilities that someone else could come up with to fuck me over? And if I’m not doing that job myself, it’s terrible. I guess it’s like trying to go to run for office and not collecting all the dirty information on your competitors, or not trying to first check everything that’s wrong about me and everything that could come out, so that if it does come out later on, right, I will have some measure of defense towards it. So we’re not safe. And it’s not safe to continue working the way we have. Disruption is a necessary part of any company today that wishes to be safe. You need to hire that person externally or you or you pay someone in your team to be that. Now, you could switch them around, you could one week have someone who’s the distruptor and who’s safe and then you ask someone else to do that role and so on.

Sean: [00:31:09] I think to your point earlier it’s going to require being conscious about it, being purposeful about it, and practicing that by making it a practice.

Joe: [00:31:22] So to that end about companies kind of trending towards doing what they perceive to be safe even though it isn’t, what are some companies, or who are some companies that you admire or do think are doing a good job with creativity and and being different?

Sebastian: [00:31:43] Well we could definitely think, for example of just this week alone Nike’s move to place, what’s his name, Joe Patrick?

Sean: [00:31:54] Colin Kaepernick.

Sebastian: [00:31:56] Yeah. Colin Kaepernick, then, as the face of Nike knowing that they make the outfits for the Football League and so on is a huge bet. That kind of risk that Nike has taken before in other situations I think is brilliant. I think that’s the kind of thing that companies should be doing right now. And for a lot of people, that would be a very very scary move. You’re basically siding with one part of a giant political discourse and you’re seeing immediately, the next day, a whole group of people posting your ad as their own personal belief on social media. They’re choosing to promote your own brand as a concept, right. They’re reposting your ads as their own personal philosophy while you’re having another group that’s burning their old Nike products.

Sebastian: [00:33:04] And that kind of bet came out of some agency in charge and it came out of one creative person who went for a coffee and figured, you know what, “we’ve got to do this.” And he managed to convince his team and they convinced the owner of the agency and that owner of the agency had to then convince the whole Nike team and then they probably had to convince, I don’t know, maybe the board. But that’s this whole series of bets that needed to happen that allows us to today be talking about it on the podcast or have it be the trending element. And I think it’s brilliant. And so once again, risk here is actually pretty safe. I mean, what’s going to happen tonight? Nothing. Everyone’s going to continue using it and this is going to die down very quickly and for most people, it will continue to cement the notion of Nike as a very edgy brand and that will be it. It’s a brilliant move, if you think about it.

Sebastian: [00:34:17] In the same manner, I don’t know, there’s many companies that are constantly changing and innovating. In the same week, again, I guess we could talk about Amazon, right. The fact that they are now a trillion dollar company and if we were to check the interviews of Bezos from just 15 years ago, the guy was already talking about the need to collect data and the importance of being able to know everything about your client. The fact that they’re capable of continuing to constantly innovate again and again and again is just brilliant. And I think in an era of exponential growth in which everything is becoming automated, in which artificial intelligence can solve most of the base problems, this constant innovative practice is the only way to thrive.

Joe: [00:35:25] Yeah I’ve heard Jeff Bezos interviews say, you know, when we come out with a great quarter for our stock and the results come out, that wasn’t from what we did last quarter, that was from three or four years ago of planning and in fact innovating. And you know it’s such essentially got to be built into your DNA to be operating like that. So things are continually being innovative as you grow and can move forward. So I’m curious, you know, you were talking a little bit before about teams and teams being creative, so when you’re hiring a creator for your team, what are you looking for in that candidate and how do you test them?

Sebastian: [00:36:02] Good point. Because in my case, my prime role is to be the disruptor and have this general creative overview, I don’t necessarily need everyone in the team to be as distruptive. Sometimes on the contrary, I need everyone to be able to execute very well. So I guess what we tend to search for, first is curiosity. I think curiosity is key, especially in this era. The need to want to know more, to understand why things are the way they are, to question the status quo, to read about any one particular issue, to have a doubt and want to follow that doubt up to understand why things are in a particular manner, I think that’s vital.

Sebastian: [00:36:59] Then I think there’s a need for stamina today. Right, there’s a notion of drive, stamina, persistence. You basically want to have a team of people that are almost capable of pushing stronger and stronger.

Sebastian: [00:37:21] And then, what I do require is that everyone is extremely critical, right. I don’t want yes people. I want the most critical people possible. I want whatever project we’re doing, whatever theory we’re having, to be destroyed in-house. I don’t want a safe environment because a safe environment isn’t safe. The only safe environment is the most critical one. So if I come up with a project or a solution, that solution, if it has a flaw, it better be spotted and destroyed in my own team, in my own space. I do not want a project going out into the world with our name on it that then gets spotted to have a series of weak points or have a series of aspects we hadn’t thought about. So having a team that is extremely critical, that is proud to be critical, in my case is important. Now I’m personally fine with that. I see criticism as a virtue and my ego is not on the line if what I present I feel is considered wrong. Obviously within respect, right.

Sebastian: [00:38:40] But I think stamina, curiosity, high levels of criticism, and then I think pride. Pride is important. I always ask the team to, no matter what they’re working on, to first work for themselves before they work for me. So even though they’re being asked to do a particular job, they’re being asked to fulfill a task, if the task they’re fulfilling, if they’re not doing it at a level that they are happy themselves, then it’s wrong. If you are not proud of what you just said if you aren’t doing it for you. If this is not good enough for you, it’s not good enough for me. So you’d better be convinced about what we’re doing, and if not you better work on it so that it’s good enough so that you’re convinced. Because I don’t want you just doing what you were told to do. I want you doing something that you’re proud of. That’s key

Sean: [00:39:38] That might be the best answer I’ve ever heard to that question, so thank you for that.

Joe: [00:39:45] Yeah, I love that, wrote it all down. Love it.

Sean: [00:39:48] I want to shift gears a little bit. You’ve been moving into the software development space for a while now. You’ve got a really cool and interesting project. I know it’s not fully launched so I don’t know how much you can share about it but we’d love to hear about it and have our audience hear about how you’re intending to disrupt the art world through technology.

Sebastian: [00:40:08] Sure. Well as an artist designer, I’ve been fortunate to be able to build a name, to have a career, and be able to exhibit and create the works that I want to make and have a group of collectors that pays handsomely for these pieces. Now the reality is that 90 percent of all art graduates don’t get to live from the sales of their work. They don’t even get to exhibit. And that’s naturally because an exhibition is expensive. You need a team, you need prime real estate, insurance, shipping, handling, etc., and that in turn means that art needs to be expensive. And so we have a very small limited amount of the population that can pay very high prices for a small quantity of work. Because of the cost involved, you can only really do an exhibition once a month, right, because it’s too expensive to be changing constantly and putting everything up again and shipping and creating and sending it back on the truck and so on.

Sebastian: [00:41:17] So you have this very old system that no longer makes sense in an era of technology. And for me, the logic there is, first, I have a responsibility as successful artist designer to try and help everyone else to exhibit and show. I believe that in an era where we’re no longer, maybe buying books or albums, maybe we’re no longer buying movies, but instead we’re all renting or paying for the experience. Being able to change the system where you pay a lot of money to own permanently something by very few people, instead having a system where you pay a tiny micropayment for an experience that you don’t need to own because there’s so many interesting experiences that you want to have a different one all the time makes complete sense. And within that, the logical step is to move into the virtual to use elements of reality as a platform where the art and creative experiences can be shared.

Sebastian: [00:42:31] And so we’re currently building a platform for artists, designers, architects, creators in general, to be able to exhibit their works, share their works, and receive some form of payment for that experience, and be able to do that in a democratic manner using the natural technologies that are coming up now that will be part of our future routine. So we’re basically trying to build, not trying, we’re building the future art world in which most of the creative output we believe will be made of experience.

Sean: [00:43:14] That’s great. What’s the last book you read, Sebastian that sort of changed the way you think that you would recommend our listeners read.

Sebastian: [00:43:22] I don’t have a proper answer because I actually stopped reading books about a year ago. Books have this problem, right, which is a format problem. And it’s similar to pretty much every single other industry. You get a writer that has a series of ideas that can be perfectly summed up in 20 pages but because the system of the book market requires 220 pages or 180 pages to be able to be sold and packaged within the normal system for the whatever, 15 dollar price, they need to fill it up with crap.

Joe: [00:44:01] I agree.

Sebastian: [00:44:02] And I have no time. I already read all the self-help books of the airport bookstore, and I already went through all of that info, and books are no longer an efficient tool. And so what I do instead is I read about 20 newspapers every day. I go through every single newspaper I can think of and I’m trying to search for an article that could be related one to another. In a weird way, I think we need to change the concept of books and leave those with the very enlightened. So in the case of Grano Harare, the book is so successful partly because of the originality of the theme and its time an its urgency. Nevertheless most probably due to the actual depth of intelligence from a guy who meditates five to six hours a day. He’s actually capable of packing a 220 page book with real, original content worth 220 pages and not a 20 page series of hypotheses accompanied by a lot of bullshit examples and little stories and anecdotes.

Joe: [00:45:22] It’s so true, and you read the first one or two chapters of a book and then all of a sudden things start to repeat themselves and you’re like, “oh boy I’m going to skim the rest or just put it down.”

Sebastian: [00:45:32] Exactly. We don’t have time for that anymore, and it’s a system that’s broken.

Sean: [00:45:37] So there’s is a quote from Albert Einstein that kind of reflects that point: “There comes a point in your life when you need to start reading other people’s books and write your own.” Books are really looking at the past and they’re somebody else’s ideas, right. So it’s interesting.

Sebastian: [00:45:52] And again, books, there’s nothing wrong with books. It’s a medium to capture ideas. The problem is that there are very few people today capable of coming up with enough original ideas that can actually fill 220 pages. And it’s hard to find a same set of ideas that can still be contingent and still be contemporary within the time frame in which the book comes out. So for example, I tend to listen to a lot of podcasts and TED talks and so on. If I check a TED talk about technology, I will not listen to a tech talk about technology that is older than 2017, even if it’s some futurist version, because it’s already too old. It’s already gone.

Sean: [00:46:46] Right.

Sebastian: [00:46:46] It’s of no use whatsoever, right. So I need to know that whatever I’m listening to is from a couple of months ago and unfortunately, the book system, because of how exhausting it is to write a book, you could be listening to the ideas of someone that’s been brewing them for ten years, and that some of that might be contemporary, some of that might already be past.

Sean: [00:47:14] All right, last question. You want to ask one, Joe?

Joe: [00:47:18] Sure. Yeah, so I always like to gauge from people comeing from different perspectives. What do you think is the single most reason a product, or in this case maybe an idea, fails? And is there anything you can do to avoid that?

Sebastian: [00:47:36] That’s really good question. I guess the single biggest reason a product fails is the lack of data. Steve Jobs used to say that people don’t know what they want until they see it, right. And there’s a philosophy that many of us use of trusting our gut and believing that we know what needs to be put out. And I think as a creator whose sole purpose is to push the ideas that go against the system and are contrarian, the first requirement is data. If you do not have all the variables, you cannot come up with the right product, the right idea. So if I’m going to rob a bank, I can come up with the most ingenious way of doing the perfect bank robbery, but if I don’t know that the guard’s just been changed and that there’s a new alarm system in.

Sebastian: [00:48:44] And so I think there is an arrogance that we all tend to exercise in which we believe we know, and we haven’t updated our information, and we haven’t maybe allowed a contrarian to ask about potential scenarios that we didn’t know about, or we cast a net that was too small in terms of what we thought were the requirements involved. And so I think it’s vital to first cast as wide a net possible, understand as many of the variables that are about to go into the system, and then within that, be able to distill it in a contrarian hypothesis that that could have the potential to disrupt each and every single one of the variables that were considered key. Yeah, that’s it. Data.

Joe: [00:49:46] Good answer.

Sebastian: [00:49:47] If the algorithm can’t wo, A.I. can’t work, nothing can work without the appropriate data, even for a creative studio.


03 / Using Gamification to Build User Experiences


Gamification is a means for eliciting specific behaviors from your users by depending on the premise that humans are naturally competitive. Some more than others, but competition is the basis for our economy and drives many of our behaviors whether we admit it or not.

In this episode, Sean and Joe discuss gamification as an important part of the system design process, and then have a great conversation with Gregg Gordon from SSRN.com about how they use gamification within the SSRN.com environment to both foster friendly competition and encourage users to increase their usage of the system by making it a pleasure to use.

Read our blog post

About Gregg

Gregg Gordon was the President and CEO of the Social Science Research Network (SSRN) for many years before transitioning to Managing Director of SRRN when the company was acquired by the Netherlands-based publishing company, Elsevier. SSRN is devoted to the rapid worldwide distribution of scholarly research in the social sciences and humanities fields. Thanks to their continued growth under Gregg Gordon’s leadership, SSRN now has a collaborative of more than 1.7 million users, nearly a quarter-million authors, and a database of over 682,100 articles contributed by scholars.

l t i


Sean: [00:00:21] Hi, welcome to the Product Momentum Podcast, a podcast about how to use technology to solve challenging technology problems for your organization.

Joe: [00:00:30] All right, so welcome to the Product Momentum Podcast. I’m Joe Hoffend, here with my cohost Sean Flaherty. Hey Sean, how’s it going?

Sean: [00:00:38] Good Joe. How are you today?

Joe: [00:00:40] I’m doing alright. You know I’ve been talking to a lot of people, getting feedback about the podcast, how it’s been going so far, and it’s been overwhelmingly positive. Morning time, recording this in the morning time, big mistake number one. And you know, people have said the content is very rich, it’s good. You know they maybe want it a little shorter for drivetime, which we’re going to aim for moving forward. But overall it’s been good. And I was going through all the requests of the feedback for the podcast and surprisingly, you know what number one was?

Sean: [00:01:16] What, what was it?

Joe: [00:01:17] More dad jokes. They want more dad jokes.

Sean: [00:01:21] Hahaha. How about some dad bod jokes because we’re reaching that phase in our life.

Joe: [00:01:24] It is summertime. It is summertime. Beach bods are made in the winter though. People should know that.

Joe: [00:01:31] Yeah we had Jeremy on from Paychex, that was great. We had Adam on from Amazon, people loved that as well. And so today we’re gonna be talking to Greg Gordon from the Social Science Research Network. And our topic for today is gamification. I know you’re super hyped up about gamification, even have your own little spin on it that you’ll talk about. But I think really what we want to just talk about is, “what is gamification?” Because I think it’s one of those terms where people have their own kind of idea of what it is, and it’s kind of been morphed over time as different mobile apps come out, and different, you know, just products in general try to use it.

Joe: [00:02:13] And so to me, my definition of gamification is really taking game-like elements that you would see in like video games, so, you know, powering up, ranking up, leaderboards, progress indicators, those kinds of things; it’s taking those and applying them to let’s just say everyday kind of products. So things that you wouldn’t normally typically think are you know fun, or you’re doing for entertainment, but things that you need to get people’s attention in. And we’ll talk about that too, like it’s not getting their attention in a negative way, but you want them to be able to engage in a product and, you know, want to engage in the product.

Joe: [00:02:57] So I have a couple examples. So personally I was trying to get into running a few years ago and I found this app. And it was called Zombie Run. And I’d always hated running. I just thought it was so boring. You know, I could do for a few minutes, and then my mind would wander, and I would just want to be done with it. But I found this called Zombie Run and it makes jogging fun. So it makes you part of a storyline of a zombie invasion. And basically what you’re doing is you load up the app on your phone, and you’re listening to it, and it’s “oh my gosh, the zombies are here, you’ve got to get to the supply warehouse, it’s a mile away, so you can start collecting food and water.” And so you got to run the mile. You know, you get there and then it’s like, okay, “you got the supplies now you got to get to the school gym to free a couple of people,” or whatever. So it kind of takes you through this experience and you kind of sort of forget you’re jogging as you’re going through that. You want to know what happens, next you’re thinking about it, you’re kind of picturing it. And, you know, that was just a really simple example where I was like, “oh my god,” they’re taking something that some people dread but want to try to do for their health and making it fun.

Joe: [00:04:08] And then there’s a really extreme example where kids with cancer, they need to track their pain in a journal. And so the hospital created an app called Pain Squad, and you know, these kids they’re just in so much pain, they don’t want to be tracking it in a journal, they don’t want to be writing it down, thinking about, “oh my gosh, this hurts that hurts I’m miserable right now.” And so what this app does is it helps the kids pretend that they’re basically part of a police force and they need to hunt down pain and eradicate it. Because reporting where the pain is and how much pain they’re experiencing helps the doctors and the care providers so much in being able to know what to do next basically in curing their cancer or helping them, you know, rid themselves of the cancer. So this whole game a curates an experience of, you know, “you got to help us,” you know, and it’s got videos of real-life police officers like telling them how, “it’s up to you, you’ve got to help us, we’ve got to get rid of this pain.” And it’s just a really, really cool example of, you know, taking something that can be a really terrible experience and trying to make something that’s a really important part of it, you know monitoring the pain, you know, achievable.

Joe: [00:05:21] So those two examples really were speaking to me when we talk about gamification. And you know, there are times though, when gamification can go bad and it’s used poorly and it’s probably where people get a bad connotation about it, or a bad representation of what it is. What do you think of that?

Sean: [00:05:39] So gamification, first of all, I don’t like that word, and you know that. But let me take a half step back because you talked a lot there. So I want to go back and I want to start with the conversation on Gregg joining the show later. He has 20-plus years experience actually game-ifying what started out very small but became one of the largest academic social networks on the planet. So I’m super excited to have him on because he’s seen the good, the bad, he’s seen things that worked and haven’t worked, he’s seen where again gamification has gone terribly awry and created behaviors that could have possibly been very detrimental, and we had to put all kinds of checks and balances in place to address that stuff. So it’s going to be a really, really interesting conversation.

Sean: [00:06:29] So then talking about gamification, so we’re more in the business space than the game space. And gamification obviously grew as a term in the game space, and now it’s widely used in the business space as well. But it’s very deceiving. And I think it can go very bad, and it has gone very bad. I mean there’s examples out there if you do a Google search you’ll find lots of good examples, or bad examples depending on how you look at it, of gamification gone bad. Brands like Marriot building Facebook games, as a great example. Or using badges, you know, sometimes badges just don’t make sense. And even companies, big companies that you’ve heard of like Zappos have made big mistakes with using badges where they just they just don’t make sense. So I guess the one piece of advice here is that it’s gamification isn’t for every business problem. It doesn’t work everywhere, and you should never start with tactics like gamification, it’s really a set of tactics, you shouldn’t start with those tactics before you really come up with a good vision for your product and have a good…

Joe: [00:07:35] Yeah, so don’t say, “We want to gamification. How do we do we do that. Where do we get things to fit in?”

Sean: [00:07:40] Exactly. So you have to think. You’ve got to put a lot of energy into thinking about the psychology behind what your user is trying to accomplish, what they care about, what needs you’re meeting for them, what their motivations are, so that you can apply the right tactics to to do something positive in the world with these things. So we like to use the word motivation mechanics. We’ve done a little bit of research on that with our clients and with people that we work with in this space to try to find the right language to use. So we call motivation mechanics a set of tactics that we can use to influence behavior in a very positive way. And it is things that we borrow from the gaming industry, things like progress bars and badges and leaderboards and you know, competitive things that you could apply in a software space that can be extremely, extremely powerful.

Sean: [00:08:32] But again, it’s not for everybody and it’s certainly not to be used blindly. “Let’s just put some badges on the site to give people a reward.” For what? Like let’s work backwards and really figure out what people want to be rewarded for, what do they care about, how are we going to make them look good to their peers, right? So those are things that we want to be talking about. Does that make sense, Joe?

Joe: [00:08:55] It does, and you know, anytime we talk about gamification there’s a particular author who comes to mind. His name is Nir Eyal. So N, I, R, E, Y, A, L. And he’s got some really great writings he does and specifically a book. It’s got this kind of like a figure eight model. It talks about forming habits and hooks and getting people to come back to your product, essentially. And the four steps of it, because you really want to look at the visual of how it all plays together. But essentially, it goes trigger, action, reward, investment. And so it’s just a really simple way of thinking about, you know, if you’re going to try to do gamification, if you think you’ve got a good fit for it, that’s kind of an easy little model to follow in terms of, you know, how to implement it step-by-step, per se, and make sure that that experience kind of goes all the way around.

Joe: [00:09:49] Exactly. There’s a Harvard Business Review article; I’ll link to it here in the podcast, from like eight or nine years ago that talks about what really motivates people. And they were talking about on an internet or motivating workers with technology, but if I remember correctly, it’s really around progress. And they found that the power of progress and actually showing that you’re making progress, you’re learning something, you’re achieving something, and just showing this sort of achievement over time is one of the most powerful motivators that keeps people positively enthusiastic about the work that they’re doing. And when we’re building business applications, this is the thing that we want to take into account.

Sean: [00:10:32] There is another book called, um.. I love Hooked by the way, Nir Eyal, great author.. there’s a book called Flow by Mihaly Csikszentmihalyi which is really about peak human performance, but a lot of his tactics and methodologies can be applied to software development too, so we use that in our workshops. And it’s around setting up people for this progress and figuring out what is the right level of challenge to present to them and understanding what their skill levels are so we can constantly be moving them up this pattern that we call flow.

Joe: [00:11:09] Yeah that’s a good one we use in the workshops and just kind of talk about, you know, “don’t make it too hard, don’t make it too easy,” and you know what is. You know, “what is flow?” You know, because everyone kind of feels that when you’re in that state of flow working. You’re really doing focused work, but you know, to actually dig into it a little deeper. It’s important to understand. Nir Eyal’s actually got a new book coming out, it’s called Distraction. It kind of like piggybacks Hooked and talks about, you know, what does it mean to really do focused work so near if you’re listening, you know, please come on you can plug it again. I just did for you. He’s listening.

Sean: [00:11:41] I’m sure he is. Another, another great one. One last book plug is Charles Duhigg, it’s The Power of Habit. In the audience there, if you haven’t read that one I highly recommend it. It’s a good one, and one last one, a buddy of mine Gabe Zichermann, haven’t spoken to him in awhile, but here’s a plug for him. Make sure we get him engaged here. Gamification by Design, he’s actually got two books out, but Gamification by Design was his first one. It’s a very tactical book about how to apply some of these gamification tactics that we’ve used in the past. It’s been super useful.

Joe: [00:12:15] Very cool. So are we are we doing a pivot right now? Is this becoming a book review podcast?

Sean: [00:12:20] It sounds like it, right. Well, you know, we’re big learners and learning is a part of the process right.

Joe: [00:12:27] There’s a lot of good blog posts out there too, you know it’s not all about books. There’s lot of good content all over the Internet these days, and some of those authors, you know, they write up good content online as well.

Sean: [00:12:37] For sure, for sure.

Joe: [00:12:40] Okay, well was there anything else you want to go over with gamification? I think that’s a nice, good introduction to what it is. And we’re going to talk to Greg in just a minute about it.

Sean: [00:12:47] Sounds good. I’m super excited.

Joe: [00:12:52] So our guest today is Greg Gordon, and he’s from the Social Science Research Network or SSRN. Greg, thanks for joining us, and would you mind starting off, just tell us a little bit about what SSRN is and your role there, in how it’s evolved over the years.

Greg: [00:13:08] Sure Joe. Happy to do that, and thanks for the opportunity to talk about SSRN. Always enjoy these kind of conversations. So probably about 20 years ago, Michael Jenson, one of the forefathers of finance, was a friend and I helped him found a SSRN which started off as a social science research network has become much broader since then. But basically the idea was we wanted to take research that Mike was seeing at Harvard but wasn’t necessarily seeing at the University of Rochester and make it available to the masses, make it available freely, as quickly as possible so people could use that research to write better research, or as we’ve really been focusing on more recently, just to help them answer questions, you know, to have actual real scholarly research at their fingertips.

Sean: [00:13:55] Hence the tagline, “tomorrow’s research today,” right, which I’ve always liked.

Greg: [00:14:02] Haha, well thanks Sean. Yeah that was kind of a funny piece because we were actually in the Stanford Law School faculty lounge arguing about what the heck we were doing and what we were trying to create a long time ago and I said, “guys, guys, what we’re just trying to do is just make tomorrow’s research available today.” And they said, “well, available doesn’t really work,” so we decided on “tomorrow’s research today” as our tagline, which is basically what we do. We do try to bring the research that’s going to be published tomorrow and make it available online today.

Joe: [00:14:33] It’s interesting because if the audience just needs to get an idea of the influence that SSRN has had in the world, you know, up until now and continuing through now, you know it’s kind of timely that Malcolm Gladwell who’s a very prolific author and has written some popular books like Blank and Outliers, he’s got a podcast and he just referenced you guys on there as his favorite web site. So that’s a pretty big deal.

Sean: [00:14:55] Did you know that, Greg?

Greg: [00:14:57] Yeah, I had heard that somewhere somewhere along the line, I had heard that possibly he had mentioned us… No, we’re very proud of that. You know, we know that Malcolm and a number of important influential thought leaders, authors, journalists and others use SSRN because of the access because you get us to surrender content, because you get thousands and thousands of papers every single week made available to you for free. And again, it is early. It’s the currency the information that provides real value. And we do know that they use that. It’s referenced in Supreme Court proceedings, it’s referenced by authors and journalists around the world. But, you know, when Malcolm Gladwell says that you are the best site on the Internet that’s certainly a nice feather in everybody’s cap here at SSRN, and that’s something we work hard for, and we’re very very proud of that.

Sean: [00:15:53] Well along that line you’re probably familiar with Nir Eyal, who’s also an SSRI author, and he wrote a book called Hooked. It’s about gamification, and that’s we’re here to talk about today. What we call motivation mechanics, which is the act of trying to find triggers and things that will help motivate users to behave in a certain way or to behave positively in our ecosystem, right. So as you know, SSRN has been a big purveyor of some of these tactics since the very beginning of your site. So can you talk about that a little bit and how you use some of those tactics to progress the platform and the authors on that platform?

Greg: [00:16:37] Sure. So many, many years ago, some of us were were meeting with some professors and we were talking about the ways that we were actually trying to help people work their way through the research. So some of the original concepts were to use tools that weren’t available for this early-stage research, like downloads, to help people find things that may be of interest to them. You know, this is more popular than that. And so we started to come up with basically download counts. And I’ll talk about it in a minute, about the costs of coming up with rankings and things like that, but basically what we said was, you know, hey we should kind of create these things that back then we started to refer to as tournaments, where we basically ranked the papers based on a variety of different metrics. Downloads per author, downloads, citations citations per author, timing, just a number of different measures that allow people to kind of customize the ways in which they wanted to get in and out of the content themselves. And we thought that was pretty cool, and then we started to talk to some professors, law professors and economists and others, and they said well this is just gamification, you know, and we weren’t even thinking about that way. We were thinking about it as a set of tools to allow people to get easier access to the content as the volume of content started to increase. And it increased significantly in a very, very short period of time.

Greg: [00:18:05] So information overload was the number one thing we were trying to figure out. What we then realized was is that this gamification, you know, points, badges, leaderboards and like, was actually something that the community found very very valuable as a way to measure themselves against each other. We started to see authors start to be much more creative with the titles of their papers, with the abstracts, how to be much more engaging, how to allow people to find things in a way that they just hadn’t been thinking about previously because it was just a research paper.

Greg: [00:18:37] So we started to get some cute titles, some more vulgar titles, but we start to get titles that were much more engaging, that allowed people to try to find the content in a different sort of a way. And I think that despite some of the costs in and around that where people do try to cheat, so to speak, I think that the efforts by a researcher to make their research more engaging, to make it more open to the masses, I think is a very very good thing.

Sean: [00:19:10] So I know that download count seems to become like the currency for SSRN, like how many times your papers have been downloaded, and citations is another one that’s big. And you guys have even come up with another ranking called the Eigenfactor. Can you talk about one a little bit?

Greg: [00:19:29] Sure. So we think about these things in different ways. But one of the things that we find interesting is that we also have abstract views. And the ratio of abstract views to downloads can tell an author about how well maybe their title has been written or their abstract has been written. Does it does it provide the answers to the reader? Does it help them get into the content? But downloads in general are what we believe to be one of the best sources for the currency of the information. How important is this right now? Because it’s an immediate impact, right. I download the paper, it ticks the ticker, and keeps moving on. With citations, which are kind of the grail of scholarship, of publishing in general, we think that those are very very important, because they tell us a slightly different thing. And that thing that they tell us is, what was truly influential in the research that you were doing? What really really made the difference to that author to the point where they said, “this needs to be referenced in my research paper,” because it had a material impact.

Greg: [00:20:35] And then third, as you mention, Sean, is Eigenfactor. And we worked with Carl Bergstrom and Jevin West at the University of Washington. And the Eigenfactor is basically a twist on the eigenvector, which Google uses in a much much much more complex manner to identify which sites are related to which other sites and then how that helps them rank it in our world. It’s more like a random walk though the woods. “This paper influenced these papers. This author influenced those authors,” and instead of a one to one relationship, like citations- “this paper cites that paper,” it’s more of this random walk through the woods, where this paper influenced that paper which influenced these three other papers which influenced those seven other papers, and it’s kind of this random walk through the woods that then is much more mathematical and algorithmic than I’m making it out to be. But basically that is to show the ecosystem of influence throughout the different nodes of the network that allows you to see a different way to look at how impactful something is as compared to something else.

Greg: [00:21:40] And again, there are different ways to count usage and there are different ways to use those counts to help either the researcher understand how impactful their research is, or how engaging their research is, or for the reader to try to filter through this mass of information overabundance that we’re all dealing with every day.

Joe: [00:22:02] So downloads being the currency of the papers, that’s pretty important that the data be accurate. Did you have any problems with users trying to game that system at all to increase their download counts?

Greg: [00:22:14] Well, you know, the moment that you start counting anything somebody is going to try to cheat and make themselves look better. So we have had a few people over the years. We’ve built a very nice system with ITX to watch for fraudulent activity, downloads that lack integrity, we couldn’t verify or validate them. And, you know, interestingly we would then write to an author and say we’ve noticed an irregular download pattern. And almost always we would get back within a few minutes a response to the e-mail saying, “well, I had a research assistant and he/she must not have understood not to repeatedly download my papers from the site over and over again. Please make whatever corrections need to be made immediately.” So we’re on this trek to find these handful of bad actors, these bad research assistants who seemed to be doing all the cheating. Because no researchers, no authors, have ever done any cheating themselves with SSRN land. But we do see that the moment that you start to put up some blockers, just flash up a little warning that says we’ve noticed an irregular pattern, or a required sign in, you know, do a variety of different little things that are relatively lightweight. They made a huge impact on the amount of quote unquote cheating that is being attempted.

Joe: [00:23:50] Yeah that’s a really good point about putting in checks and balances and warnings. Would you ever, you know, tell someone who’s thinking about implementing some kind of gamification practice into their product to not do it because people are going to game it somehow, or what would your recommendation be to them?

Greg: [00:24:07] You know, it’s a really interesting question. I deal with most of the major publishing houses, a lot of other startups and people in this industry, and the topic has come up a few times. And some people say that they just didn’t think that they were gonna get the benefit out of it. Others basically are babes in woods where they probably knowingly to some extent are being gamed but they’re okay with it. You know, in our world, we wanted to build a trustworthy set of metrics. And so we spend a significant amount of money every year verifying and validating our download counts because we want them to be trusted. We want them to be used by a lot of people in a lot of different circumstances. So for us, the benefits far exceed the cost. So I guess I would say, don’t be afraid of somebody trying to game your gamification system or somebody trying to cheat your metrics. I mean, that’s unfortunately just part of of the life that we live in right now online. What I would say, though, is just to do an honest assessment of what are the values you’re going to get out of it and what are the costs you’re willing to bear to get those benefits?

Greg: [00:25:20] For us, it’s been incredibly valuable. We have most of the major universities, many of the major publications, and other people around the world using our metrics for evaluation and other purposes. And that’s because they trust them. But we knew that going in. We knew going in that we needed to provide a system that could be trusted. So if you want vanity metrics, then open the doors and let the horses out and enjoy them. But if you want something that people are going to use, actually use, in their work world, then you have to spend a little bit of time and effort to make sure that they are trustworthy.

Joe: [00:25:58] Great. Great point.

Sean: [00:25:59] Yeah I would argue that if you’re at that point, that means that they’re working. So if you have to put antifraud measures in, like you guys did, that means they’re actually working for something. Because if I remember some of the anecdotes when we were doing studies and analyzing how your systems were being used, I remember some authors were putting their SSRN statistics on their resumes, right?

Greg: [00:26:25] Sean, we actually still to this day have authors write in and say, “can you print out my author page and sign it and send it to me?”.

Sean: [00:26:35] Exactly.

[00:26:36] We have the including it in their tenure and promotion packages. We have them including it in their dissertation conversations. We have them using it all over the place. So it’s definitely used and found valuable by a lot of people in a lot of different markets.

Sean: [00:26:56] Yeah. There were also some calculators that ranked colleges and universities that were using your statistics in their analysis, right.

Greg: [00:27:05] Well, within SSRN, what we have is a number of papers, some of them early on, some of them more recent, but a lot of the papers tried to give context for these. And so they would look at US News and World Reports or some of the more popular rankings of departments and compare SRRN rankings to them, see how they were relatively closely correlated, and then try to understand why there were differences. Did a top scholar move from one school to another? Did the school open up a new center or a new area of research that was pretty hot right now, so that was creating some buzz or or getting them to do some new cool stuff. So again, it’s definitely been used and compared in a lot of different ways. And to us, that just meant we had to work harder, we had to make sure that the quality of what we were putting out there was was even better, because more and more people came to value it. I think the point you made a minute ago about if it’s not worth anything then nobody cheats at it is a pretty darn good one.

Sean: [00:28:18] Thinking about, you know, gamification and making a product more fun or easy to use for a user versus coersion, because a lot of people when they think gamification they think, “oh it’s manipulative” or “it’s going to make me do something that I maybe didn’t want to do.” How do you tow that line and think about that as you implement, you know, certain ways to get people to take action that you think is going to benefit them?

Greg: [00:28:42] Well you know, I guess to my mind you need to take a step back. My younger son plays Fortnite which is a popular online game right now and he plays it with his friends.

Joe: [00:28:58] I know it well.

Greg: [00:28:59] And Fortnite is a good game, right. It’s just a really good game. If it wasn’t a good game, then nobody would be playing it. So all of the points, all of the badges, all the leaderboards in the world, only work if they’re on top of something that is good. If SSRN was a terrible platform, if we didn’t have any content, if the content that we had was junk, then all the gamification in the world wouldn’t matter, right.

[00:29:37] When I hear people say that, I come back and I say, “Well I think you’re missing the point,” right. There’s no gamification system that I’ve ever seen that makes me do something I don’t want to do. I mean it makes it into a game, right. And I’d rather play a game than not play a game. And so if the gamification makes what I’m doing a little bit more interesting, a little bit more fun, and motivates me to try a little bit harder, then I’m all for it. But I just don’t see any gamification systems out there that are going to force somebody to do something that they don’t want to do. So to me, it’s making it a little bit more fun. And then as the platform, we then have the responsibility to make sure that what we’re putting out as metrics we stand behind.

Joe: [00:30:30] Reputation and trust.

Sean: [00:30:31] Yeah I think that’s the key is the factual side of it. Like these are just facts that you’re using in the game, so to speak, and the more evidence that you can put behind their truth, the more support you have, the more powerful it is. And your example is a great one because that’s really an example of an industry that’s been disrupted. Arguably a lot of it by the gamification, itself by the motivation mechanics that you guys deployed.

Greg: [00:31:01] You know, so I use the word responsibility, right. And I think that links to the comment that I made earlier in our conversation about trust, right. And so you need to actually be thoughtful about it. You need to bear the responsibility to put out good quality metrics. But you also need to be responsible to think about what you’re doing. You know, many of us have had detailed, lengthy discussions about the Matthew effect, right, you know, where somebody who’s on a top list is going to be getting more downloads becuase they’re on a top downloaded list, right. And so we had to be thinking about, “what impact does that have?” And so we came up with some different measures, and people would roll off that after a certain number of days, or it would take this requirement to be on there because it was classified here or there. So we had to have those responsibility conversations about, you know, what were we going to rank? How were we going to rank it? What were the rules that we were going to use? And be very very transparent about them. It’s hard to get that right the first time, but if you’re honest with yourselves and you allow other people to give you feedback about it, then I think you can get to a system that actually works great for your communities. But it has to bear responsibility. You can’t just throw this stuff out there and hope that it’s going to work. It needs to be actively managed and managed on an ongoing basis.

Joe: [00:32:18] So thinking about that, you know, being thoughtful about it and making sure it doesn’t have an adverse effect that wasn’t expected, do you guys ever tests these kinds of features before you release them just to make sure?

Greg: [00:32:32] Hahaha, we AB test the hell out of everything. I know more about statistically significant factors than I care to. But yeah, I think that we’ve become fans of the Silicon Valley Product Group and their approach to product market fit and their approach to iterative process, not just iterative design or iterative agile programming, but really truly the whole thing needs to be agile and iterative. And so we test, we run a lot of experiments to try to figure out how these things fit together. Very lightweight experiments, you know where you click the button and it leads nowhere, but we now know how many people are interested in clicking the button.

Greg: [00:33:22] So yeah, we AB test the heck out of things, and we do that because it’s so easy when you’re moving fast to make a mistake. And it’s so easy when you’re moving fast to not even realize what you don’t know and what you’re not even thinking about. So baby stepping these things through with what seems at times like excruciating detail and “why the heck can’t we just jump to the end?” But, you know, we spent a lot of time running a waterfall process. And I know how badly that doesn’t work; that the small timing frustrations at different points along the path are irrelevant compared to what we ultimately achieve. So yeah, we do a lot of small micro experiments and AB test a lot of the things along the process. And we think it’s become a much much better system because of it.

Sean: [00:34:15] So let’s talk about tactics a little bit. So you guys chose the tournament, so it’s really the big first began gamification thing. You had downloads for a while, but it really wasn’t until you introduced the rankings, or the tournaments before you started as what I remember as the hockey stick. You know, when the traction really started to go up. The really cool tactic was the top tens, the e-mails, and we call those triggers, right. So the triggering people to come back and take a look at their paper, their abstract page, maybe update it, maybe upload other papers. I remember when we implemented the top tens, the e-mails that would go out telling people what list they’re showing up on and their ranking, that it drove all kinds of interesting behaviors. Want to talk about that a little bit?

Greg: [00:35:03] One of the things that I love is gaining perspective and it’s easy for me, you know, where we have papers that have tens of thousands of downloads, to be presenting at a conference and have somebody come up to me and say, “I just got that email from you and I really appreciate that it really helps motivate me going forward.” And you know, I’ll catch the name, and I’ll go and check it out. And the e-mail that would have been sent may have been one for for two dozen downloads, but it really mattered to that person, right. In other words, if you look at the communities and the environment that we’ve worked for the last twenty years, it’s been an environment where you would spend six months to six years maybe, working on a research paper, submit it into a black hole, and have that process, have you be waiting for the feedback. Right now, you submit a paper to SSRN, it’s available. You can make it immediately available, we process it within a matter of minutes or hours, and the world can go and download it. And you then immediately get feedback of how many downloads you got. So the point about these notifications, these triggers, as you’re referring to them, Sean, is an interesting one.

Greg: [00:36:27] One, we wanted to show acknowledgement and appreciation for somebody being part of the SSRN platform. But two, you know we’re all really really busy. And I don’t have time to go and check every single thing I’d like to check every day. So this notification that says, “hey you’re one of the top downloaded authors,” has real value. It makes it more efficient for me to be made aware of that, and then I go back and I can check in I can see, “oh, why did this paper have this downloads to that abstract view ratio versus this other one?” And I start to think about how I’m connected to my research, about how I’m allowing others to be connected to my research. Because of this trigger, it helped me be made aware of something that’s of interest to me. But, you know, I just may be too busy to go back and check.

Sean: [00:37:17] And on the flipside of that, there’s other tactics that you chose over the years not to deploy. Things like giving people special statuses or maybe badges like Foursquare or Yelp where you level up and you get different badges. You guys never deployed anything like that. Can you talk about that a little bit, maybe why?

Greg: [00:37:41] You know, we did have a lot of conversations about it. We’re actually right now looking at acknowledging certain papers based on non metric related information. So the last year and a half or so we’ve been giving away awards with the Wharton School of Business and we’re trying to figure out a way to to show those papers in the system with a badge. But those are not metric-related badges. The reason that we shyed away from, you know, this power user this greater level user, is that we wanted SSRN to be a level playing field. We’ve always felt that it didn’t matter whether you were at Harvard Business School or Monroe Community College in Rochester, New York that your research deserved to stand on its own two feet. So instead of trying to figure out how one researcher is better than another or how one user was better than another or higher ranked or whatever the case may be, we avoided that because we wanted it to be a level playing field.

Greg: [00:38:48] We wanted everybody to have the chance to share their ideas and let the research shine through, not necessarily an individual, but let the research show that this was important stuff. So we didn’t want to try to possibly muck that up by adding a set of badges to individuals. We thought that could be problematic more than helpful.

Sean: [00:39:14] Great. So it sounds like, you know, from everything I’ve seen online, whether it’s from Malcolm Gladwell or other very influential people that I follow that they use SSRN. Have you seen the industry change at all because of SSRN and what you guys have done? Or what kind of influence have you seen your product have in the world?

Greg: [00:39:34] You know, if I was arrogant I’d say that we’re making a really huge impact all over the place. I don’t think I’m quite that arrogant. I guess what I would say is that we’ve been heads down on making really good quality research available for free as early in the process as possible. So we started last year with allowing research to put up ideas. You know, what are you working on? Are you looking for collaboration? Are you looking for funding? Do you have some data that you want to work with somebody to create a research paper around? How early can we get in this process?

Greg: [00:40:13] The other thing that we’ve been focusing on is working with some of the greatest branded journals in the world to get their content out early. So we’ve been working with Cell Press, and we just announced last week we’re working with The Lancet. So in biology and in medicine, we’re now working with some of the top brands to bring their research out earlier, which I think will continue to enhance the impact of the research that we’re looking for. I remember several years ago, well ten-plus years ago now, when I first found out that the U.S. Supreme Court referenced an SSRN working paper in their proceedings. And it was a game changer for me. It really helped me realize that, you know, some of the most impactful things happening in the world, Supreme Court decisions, are being influenced by research in SSRN. And it’s that early research that’s helping Supreme Court justices make better decisions. I mean, that’s that’s pretty damn cool. And so from that point forward, we just have been striving to make more high quality research available as early as possible in the process, as easily accessible, as freely and broadly disseminated as possible, because that’s what really matters.

Greg: [00:41:44] So yeah, the Gladwell thing is awesome. It’s just spectacular. We’re so proud. The team is so happy about that. And every time we get referenced in a major publication anywhere in the world we’re very very happy about that. We always want to celebrate those successes, those milestones where we’ve been acknowledged or recognized for the hard work that everybody here has been doing for a long time. But from the moment that we first got recognized by the Supreme Court, from that point forward we realized how important what we were doing was and that we needed to be focused on that and not just getting you know acknowledgements and awards.

Joe: [00:42:23] Yeah thanks for mentioning that. That’s something that we talk with our teams about, and you know for all the other product managers out there and people in software development, celebrate the wins. You know, the milestones, the recognition, celebrate that becauase, you know, we’re usually working so hard just to get products out there. But when they actually have an impact, make sure to celebrate it. It helps motivate your team.

Greg: [00:42:44] Joe, I couldn’t agree with you more. I think it’s one of the biggest most important things. You know, we use an OKR process here and some people looked at like I had three heads, but we have a refrigerator full of beer and at 4 o’clock on a Friday, each of the different teams who were working on one of the key results celebrate their key actions for the week. And you crack open a beer and it’s all a step forward. But that celebration, the cadence of celebration I think is missing in a lot of companies, and we don’t do it perfectly here at SSRN, I would be lying if I said we did, but the cadence of celebration I think is a really important factor in letting a team enjoy their hard work.

Sean: [00:43:37] A meticulous sidetrack: so you travel more than anybody I know. And one of the biggest industries for gamification is obviously the airline industry. Talk about your experience with that, with airline points and what works, what you think doesn’t work, what could be improved there.

Greg: [00:43:57] Well, you know, I think I do travel an insane amount.

Sean: [00:44:02] Insane is a very conservative word for the amount of travel that you do.

Greg: [00:44:08] Hundreds of thousands of miles a year. I travel a lot. So I think that it’s in three kind of, tiers. I think that there’s people like, you know, like my children who travel a lot, but they’re about price, right. How do I get the cheapest flight? Because I’m going take two or three trips a year, and money is much more important because I’m not going to be able to accumulate points at a level that that that’s going to make any real difference. So the gamification for them is irrelevant. You have people like me who are at the other end of the spectrum who travel a tremendous amount, who, you know, have a high-level status on a multiple different systems. And so for me, because I’m traveling so much, the convenience becomes a driving factor. Now I will say that I am shocked by the difference between traveling with status on a particular airline or to a particular hotel and traveling without status on a particular airline or to a particular hotel. So there’s a huge difference in the quality of the experience. But for on this end of the spectrum, convenience starts to become a factor because I’m gone so much that I don’t have days to try to manipulate around or try to do things.

Greg: [00:45:38] But this big, fat middle section, this middle tier is really really interesting. And so I travel with a bunch of people who are in that middle tier, and these are people who have, you know, generally status on an airline and status with a hotel chain. And the amount that they are willing to modify their schedules to make it work for their gamification is really interesting sometimes. Now the gamifications there are not just points on a ranking, those are, you know, free nights and free flights and it may make a difference between you and your family going away or not going away. But, I mean they have figured out to an incredible degree how to motivate and change people’s behavior. And I know people that will get up an extra half hour early in New York or London to take the tube or the subway to stay at a branded hotel that much further out because it gets them points. So I think that in those situations, the gamification is not just about status. You know, nobody says to me, “oh my gosh, Greg, you’re this status or that status.” I think in those situations, the gamification maybe is a bit more towards I was going to say the rewards, but maybe the perceived rewards. You know, I don’t know that they’re always worth what they are perceived to be, but the perceived rewards are very influential in that middle tier of travelers.

Sean: [00:47:23] You obviously think about that stuff a lot.

Greg: [00:47:26] I’ve spent a lot of time on planes.

Sean: [00:47:29] So you’ve been extremely generous here Greg Bowness a lot of time I really appreciate it. So Greg, one last closing question here. What’s the last business book you read that you thought was valuable, and why?

Greg: [00:47:43] As you know Sean, I consume a lot of content. I listen to a lot of audiobooks while I’m working out or running or whenever, and so I do consume a lot of content. So it’s really kind of a funny question. So I don’t know if you’d consider it necessarily a business book, but I’m a huge Steven Pinker fan. I think that he digests and plays with research and a really interesting way. So some of his latest stuff has really been great. The new Dan Pink book When, I’m a huge Dan Pink and Malcolm Gladwell fan obviously. The books that I’m going to answer your question with, I’m going to give you two. The new version of Inspired, about the Silicon Valley Product Group, really starts to attack some of the issues that we’ve been talking about in this conversation. And, “how do we start to think about things differently?” And, “how do we start to break apart the processes?” and not just think about gamification as good or bad, but how does it fit into your world. How does it fit into your product and into what you’re trying to get accomplished so that you actually accomplish something beyond just having rankings?

Greg: [00:48:49] And then the other one which probably is one that you wouldn’t necessarily think about is, Going Clear. It’s by Lawrence Wright and it talks about Scientology and it talks about a business being created in and around a religion and a set of practices that are mind-boggling that an organization could continue to exist with those practices. And again, you know, not whether or not you are a fan or not a fan of Scientology or religion or whatever, but the thought process about how people buy into purpose, right. This whole concept of motivation: autonomy, mastery, and purpose; that they buy into purpose, just reminded me of what I say all the time here at SSRN, is that that if I’m spending less than eighty percent of my time helping the team understand the purpose of what we’re doing, then I’ve messed up. I’ve done a bad job about training or hiring or leaving them alone or whatever. So it really helped me remind me why I believe I’m here doing the role that I’m doing which is to help people understand the purpose of SSRN and why we do what we do.

Sean: [00:50:20] Great answer. Thanks Greg.

Joe: [00:50:22] Greg, thanks for joining us today. This was very insightful and we hope you, the audience, learned something about gamification and how to think about your communities, your ecosystems, and your users.

Sean: [00:50:34] So if you have any questions, concerns, comments, ideas, anything you’d like to share. We’d love to hear from our audience on Apple podcast, on Google Play, on productmomentum.fm. Reach out. Send us an email phone call or you want to give us your ideas. Thanks. Thanks for listening.


02 / Creating Advocacy & Building Product Loyalty


Advocacy is the goal for any software product that interacts with humans. Whether you’re building a website, a mobile app, or any digital technology that interfaces with people, the goal is to build product loyalty among users. This is done by moving the client up the loyalty ladder, starting with earning trust, then building loyalty, and ultimately earning advocacy.

In this episode, Sean and Joe discuss the different aspects of creating and measuring product loyalty and advocacy and then talk with Adam Bates from Amazon about how he works on creating loyalty within his own customer base.

Read our blog post

About Adam

Adam is a digital product leader with an entrepreneurial track record of envisioning and building experiences that transform how we live and work. A UX research and design expert, hands-on technologist, data cruncher, and energetic team-builder. Currently, Vice President of Product for FreshBooks, Adam has also led Product and UX for innovation groups at Amazon, Paychex, and Topspin Media.

l t i


Joe: [00:00:30] OK, so Sean, today we’re talking about really what the ultimate goal should be for pretty much every software product out there. And we propose that that’s advocacy. It’s looking for users and building users who are advocates of your products. So everyone really is an advocate of some product, some company that’s out there in the world, and we’re going to talk about really, how do you earn, and we use that word specifically, how do you earn advocacy, and then measure where the users are at in their journey in becoming advocates? Because for us, in our opinion, advocacy is not done overnight. You have to be very intentful of how you build advocacy.

Sean: [00:01:06] So you can take that one step further, Joe. I mean advocacy, if you think about, it should be the goal for any business, right. So if we’re in business to solve problems for people, anyway maybe not all businesses are but most of us are, our goal should be to turn every one of those consumers into an advocate. Think about what you get out of that. Someone who’s truly an advocate of a product, or a service, or any business, is going to do amazing things for that business. Even if it’s B2B, you want the people that buy from you, if they move from one job to another, to bring you with them. You want your customers to talk about you to their friends and family. That’s why there’s this big play to understand what our customer sentiment’s all about. We need to know that we’re at least on the path to earning advocacy from our clients. If we’re not, what can we do to change that? Right.

Joe: [00:01:48] Yeah, totally agree. And you know a lot of times when we’re speaking with clients or other folks in the industry, they always kind of inherently know that they’re trying to do this, but they aren’t really able to put their finger on what it is they’re trying to do, meaning building advocates. And a lot of companies, they go out and they decide, “we’re going to start measuring Net Promoter Score, we’re going to make that all about what we’re doing,” and we have our opinions on NPS, but that’s OK for them. Ultimately they’re trying to measure advocacy. That’s what NPS really is. It just kind of leaves out that whole part about, “OK well how do you get users to be high scoring in NPS? How do you get them to be advocates? Like, what does that actually look like for us building products to get people up to that level?” So there’s a simple three step model that we like that can help everyone out there gauge where users are at in the journey of becoming advocates. So do you want to go ahead and break that down for us and then we can deep dive into each step?

Sean: [00:02:36] Absolutely. So let’s start by talking about the definition of advocacy and what it is. And to us, it means that your customer, your consumer, whoever’s using your product or service, they’re so moved by your product or service that they’re willing to invest in its future. So that means that they either share it with their friends or family, maybe in conversation, maybe electronically, who knows. Maybe they give you an email address of a friend who might like your product or service, they share something publicly like a review on a review site of some sort. Or even better, they give you feedback, they reach out to you and invest their time and energy in telling you how to improve your product, right. So these are acts of advocacy. This is what we’re really looking for in terms of measurement system.

Sean: [00:03:10] But we also believe, going back to the journey framework, so to speak, that in order to get an advocate that consumer must first be a loyal user of your software product. Meaning, they come back self-determined to use it again if they have the same problem to solve. They’re not looking for any competitive solution. And before they’ll ever become loyal to your product, they first have to trust your product. Meaning they make their first investment of time, energy, money, or whatever in your product or service. That essentially becomes what we call a loyalty ladder, or our users’ journey on any software product through trust, loyalty, and advocacy. We want to look for those three key things and the behaviors that would indicate that those three things are occurring. And if you are doing an NPS or any sort of sentiment analysis, you should find that if you move the needle on those factors, you’re naturally going to move the needle on user sentiment as well, right Joe?

Joe: [00:03:56] Yep, that’s absolutely right. So if anyone’s driving or on the go and can’t write anything down, we’ll just recap once more. First you gotta earn trust, step 1. Step 2, you’ve got to build loyalty and that takes time. And step 3 is you’ve got to earn their advocacy and help them tell their story.

Sean: [00:04:13] All right, so let’s think about the simplest e-commerce application. You may define ROI on your shopping cart investments as like an increase in monthly revenue or dollars spent, right. But in the longer run isn’t an advocate more valuable than a single purchase in almost every case? Shouldn’t we be more interested in building a loyal relationship with a client and getting them to come back and use our product? Even if they’re not buying from us every single time they come, they’re learning about a product, they’re learning about a service, they’re reading reviews, things of that nature. We want to know that our product is actually adding value to their purchasing decision or to their life in the context of whatever it is we’re selling. That’s going to get us on the path to advocacy. It’s not just selling them a products through a website, right Joe?

Joe: [00:04:50] Yes.

Sean: [00:04:51] For example, if a customer refers five friends to us, we know that’s more valuable than a single purchase. It’s just common sense. So no matter how you’re measuring ROI, advocacy fits into that measurement somehow. Imagine if you had a way to measure that. What would that mean for your product? If everybody on your team was building your products to target advocacy versus whatever hard core business metric you had in place before, I just think you’ll get a better product at the end of the day. You’re working to build a hundred-year relationship with a consumer versus working to make a sale, or tricking them into buying something or signing up for your newsletter, right. Similar to the software-to-vendor relationship, right, traditionally. Companies don’t hire vendors to produce software products. They think they do. They reach out to a software vendor with a pile of money and they say, “hey we want all these features built into our software product.” But the reality is, they’re are going to be more successful if they hire their software vendor to create advocates for them, right, from as many of their users as possible. But they don’t.

Sean: [00:05:44] So we’ve been using the analogy of the loyalty ladder to create our customer journeys, and to discuss and keep our teams aligned on that overall goal of getting as many advocates out of the users of our software products as possible. In the loyalty letter, if you think of it as a ladder, we’re talking about the micro-transactions that move people up or down, right. They move towards trust, towards loyalty, towards advocacy, or they move in the other direction. Joe, do you want to give a couple of examples of that?

Joe: [00:06:07] Yeah. I think it would be helpful to just kind of picture how the latter works. So for the micro-transactions concept, let’s think about an e-mail newsletter subscription. So you’re on a website, you’re in an app, and they say, “hey you want to give us your email address we’ll send you some super valuable information.” So you think, “OK well they seem alright. I’m on this website, it seems good, seems legit, maybe it’s helpful. I don’t see any red flags. I’ll give them my email address.” So you sign up and you get a nice success message. So you have an expectation now. So you took a step up the loyalty ladder. You’re not at loyalty yet, but you’re heading in the right direction. And so, eventually you check your e-mail and you get the first newsletter. You read it and you see a link that looks useful so you click it, and ooops, 404 page-this page does not exist, broken link. At that point in time, you just got bumped back down the ladder. Just a little bit. So I’ll stop there because that’s an example that could keep going on and on. You know maybe they then respond to you well through customer service or they actually send a follow up e-mail. Maybe they notice that they have the broken link and they fix it.

Joe: [00:07:03] But every interaction with the product, with even your brand, like you said earlier, across all the channels nowadays whether that’s online, offline, it’s all incredibly important. It has to be considered as a whole of the customer journey. And so we’re going to ask Adam about that actually, we’re going to ask him about the customer journey and how Amazon looks at that.

Joe: [00:07:19] But you know, I have this experience with banks all the time. I love the app, I use the online interface, it’s great. And then I call, I get stuck on a phone tree, I got to hit five options before I actually get to a person or solving my problem somehow. It’s brutal. So we’re talking about software in this podcast, but it really applies everywhere. And so if you’re not thinking about the experience holistically that’s a problem. And that’s why we’re seeing chief experience officers, that title popping up a lot more these days. The whole concept that customer service is not a department.

Joe: [00:07:45] And something that’s really interesting to think about related to e-mails: what are the quality of the e-mail addresses that you’re collecting? I’m sure everyone here has got like a primary one they use. Maybe it’s their full name, you know, at Gmail or something. You’ve always got this secondary one as well that you use for junk mail. So it’s always been interesting for me to think about, “are you collecting someone’s primary e-mail address that they check all the time, or are you talking their junk e-mail address?” And that’s really hard to measure, but I just think it’s an interesting concept to think about.

Joe: [00:08:08] So back to loyalty ladder. I had someone recommend to me a year ago to try to a FitBit. So I’d been trying the brand and people around me are wearing them, so you know, I just decided to buy one. And then I began using the app, and I really enjoyed it. And it would prompt me to give feedback and overall improve my data, make sure it’s correct and everything. So I wore it enough, I invested time into it to try it. I then thought, yeah I’m getting use out of this, I like looking at how I’m doing. So I started to go in and adjust my data. I had earned their trust started tweaking my numbers, entering water consumption, all those kinds of things. So at that point, I knew I wasn’t gonna buy another health/fitness watch or wearable. I was loyal to it. And then finally, since I was enjoying so much I decided I wanted others to try it. So it was right around Christmas time. I started to give it as a gift. And I thought to myself, “wow I’m really an advocate FitBit because I’m willing to give it to friends and family and tell them that this is worth them investing their time in and trying.” So that’s more of a hardware product example. Software played a large role in that, but it doesn’t always have to be something major, either it’s the little things along the way that move you up and down the loyalty ladder.

Sean: [00:09:10] Totally agree, Joe. Each interaction you have with any business, it’s a step up or down. I’m a little disappointed, I have to say because I clearly wasn’t on your Christmas list. But I’m impressed.

Joe: [00:09:19] No no no. When’s your birthday by the way? I gotta…

Sean: [00:09:24] Hahaha. So I got another story. So this is not just a software product problem, this is also a cultural problem for business, and the reality is if the business culture is not aligned with the software product, you’re not going to be able to truly earn a loyal customer because they have to have a consistent experience across all of your channels, in your store or wherever they are, not just with the software products. But the software is a good place to start.

Sean: [00:09:44] So I have a little story to tell about my mechanic. So I bring my car in for standard brake pads. I’m not, you know, expecting anything major to happen, but at the tail end of the day he’s working on my car, my truck actually. It’s a GMC Acadia and I’m on my way to pick it up and get a phone call. He broke the brake lines. And you know, in my younger years I had a similar event happen and the mechanic told me that the brake line was broken and he found it when he got in there, you know, and created a very bad experience and I never went back to that mechanic again. This guy says, “you know what you’re on your way in. Why don’t you just come on in and I’m going to let you borrow my truck so that you’re not without a car because I know that your wife’s driving you in and you need the car. I’m going to take care of this. We made the mistake, it was a total mess that we created, we’re going to take care of ya.” So just think about that think about how that made me feel. And we all have this experience where we have a service disruption in what’s expected from a business that we’re hiring. In theory, I will never use another mechanic, right. I’m always going to go back to this guy because he knows to be honest, because that’s a big key piece of trust formula, which we’ll talk about another episode. He knows that if there is a problem and he stands behind it, I’m going to trust him forever into the future. I mean, service disruptions are possibly the biggest opportunity that you have for generating that story that your customers will tell when they ultimately become advocates, right.

Sean: [00:10:53] People do not expect you to be perfect, but they do expect you to have integrity. And the reality is, integrity is rare. It’s rare, it’s memorable, but it’s cultural too, you have to fit it in your business’s culture. Software should support that, but you still have to build it into your business culture around your products and services. I’d even go as far to say you could stage bad experiences to help boost your clients over that advocacy threshold.

Joe: [00:11:18] Hmmmmmm….

Sean: [00:11:18] But that would be a little unethical. Probably a bad idea. So anyway, we believe in the power of this cascade: trust, loyalty, advocacy. We’ve even built workshop exercises dedicated to getting everybody on the team kind of aligned with getting our customers to advocacy as quickly, as effectively as possible, without manipulation. But we’re going to save those conversations for future episodes as well. All right, so Joe, let’s talk about trust. Let’s talk about how we define trust and what it means in the context of a software product.

Joe: [00:11:44] So yeah, this is trust in the context of the product we’re talking about. It’s not trust in the way a lot of people think about it, like, “I trust you so I’m okay with marrying you,” or, “I trust you with my life.” This is trust in the context of our users behaving and using my products in the sense that show that they trust it. We usually look for this in, are they willing to make an initial investment in the product? And this can come in a variety of ways. It may be an investment in a monetary term, like they’re willing to give money for the product. It could be an investment in data or information.

Joe: [00:12:19] And so some examples of this would be, you know, signing up for e-mail list. Your email address is very personal to you. People live in their email all day now. And to get spammed by something is a big deal. People actively avoid it, and if they don’t trust a brand or a product, they will not give them their email address or they’ll give them a fake one. But in this case that is an investment. Facebook, for example, you are trusting them to some extent because you’re willing to give them certain information about yourself. Your birthday, for example, there has to be a trust aspect there because your birthday is used across many other software products to be part of your identification. So we are looking for that also in e-commerce products with, are you willing to buy a product, even one, even if it’s for five dollars? Are you willing to do that and take that leap? And then as we talk through loyalty, which is coming up next, you know, we’ll learn about, how do we see that trust evolve into loyalty? What are we looking for there? And I know Sean, you’ve been thinking about loyalty lately and how that transforms from trust, and what that looks like.

Sean: [00:13:14] I also think time is one of the things that you could measure trust for. Like they may tell you who they are but it doesn’t necessarily mean that they trust you yet. Like if you have a product that it takes a long time for you to earn enough trust for them to actually sign up for your service or sign up for your product. It may be the amount of time spent learning about something. The key is that it’s an investment. It’s that first initial investment that we think should be the measure of trust for any given product, and it’s generally time, money, or data.

Sean: [00:13:42] On the loyalty side, I think it’s a little more complicated and a little more simple at the same time. When I say that, I mean it’s a little more complicated because it’s so different for every different product. Daily active users, for example, is one of the measures that Facebook looks at for loyalty. And for other things, like online research portals, they often use a monthly active usage metric of some sort. But it’s more complicated because you have to figure out what is the appropriate regular expected usage from that particular user that would indicate that we think they’re loyal. You may have to measure it over time and try to figure out what we think that expected usage is. You know, and every software product is different. So you have to figure out like how often would we expect them to come back and reuse our service to indicate that they are loyal users?

Sean: [00:14:25] Again, our definition of loyalty means that they come back to our product and use it again because they want to. Self-determined: not because they have to use it, not because we’re forcing them in order for them to get some discount or something manipulative down the road, but they’re coming back to use our product because they choose to, right. That means something important, it means that they’re not looking at any other competitive solution for this particular problem in the world that they’re solving. So that’s how we measure loyalty. It’s an indication of their recurring expected usage of our software product. And we want to get the most people from our ecosystem into that regular state of usage. Right, Joe?

Joe: [00:15:01] Yeah. And then finally, you know, if you’re lucky enough to get them that far, which a lot of companies are not; they don’t know how to turn loyal users into advocates, well you have to give them tools, you have to empower them to be advocates. And by advocates, they’re essentially a mini sales force for you. They’re out there, spreading the word, basically. You know, word of mouth, it’s negative. But for when it’s positive, it’s typically an advocate-type situation where you’ve just had a great experience and you want to tell the world. You want to tell your friends, your family, you want to recommend them on Facebook. You want to tell whoever you can, and the second you hear someone asking for a product or service. If it’s a need, you have that example ready to tell them about. And not only that, but if you ever hear anyone badmouthing that product or service or company that you’re an advocate of, you defend them. You step up to the plate and you say, “that’s not true, I actually had a good experience with them.” And you try to fight that fight for them. And so, they become an extension of your company essentially. They don’t work for you technically, but for all intents and purposes, you want to be treating them as if they do. And collect feedback from them, give them the first chance to give feedback. These are prime candidates for beta testing and collecting feedback from, so pull them in and empower them to be a voice for you.

Sean: [00:16:03] Right. They clearly care about your product at that point, right. If they’re an advocate, they’re self-motivated to invest in its future. Feedback, sharing, all those sorts of things. Those are clear measures of advocacy and that’s what we’re after. Like, for every product. Like we’ve talked about ad nauseum at this point. So what are some examples of some specific products that you use, Joe, or that you’ve thought about, and what kinds of measures would you think you’d put in place that would indicate that the user trusts the product, the user is loyal to the product, and ultimately that the user is an advocate of the product?

Joe: [00:16:33] Sure. So for trust, you know, we like to look at kind of first line metrics. You know sign ups, are they willing to create an account? You know, regardless of how much information the account is asking for. But you know, that’s another thing you can look at is, “are we more likely to get more accounts if we just ask for a first name and an email rather than a first name, last name, and email?” That’s all things you can work out, but we’re looking for, “are they willing to invest their data in that sense?” In terms of time, “are they willing to use the product for a few minutes and invest into it to get better results out of it? And then monetarily, we look at, “how can we make that first purchase the easiest?” It can be, “how can we get them to do a trial and then sign up, or just make that first purchase maybe at a discounted rate or some other way to at least pull them in and get them to try it?” That’s all something you learn through the user feedback mechanism of figuring out why they’re not taking that first step. Because before trust, there is that awareness factor of, “do they even know about your product?” So once they get there, we can figure out, “how do we get them to build trust with the product?” And make sure we’re approaching it properly.

Sean: [00:17:32] Right. We were in an interesting conversation yesterday with a very large insurance company, and that question came up. Like, “all right so we want to earn trust, we all agree this is important. But where do we start with awareness? Like how does marketing fit into this whole thing?” And the reality is, if you’re running a business, you’ve got to be thinking about that. What does your lead generation funnel look like? But what you’re doing out there in the market to bring people to the product is not quite in the domain of like, what can the product do to actually convert these people, right?

Joe: [00:17:59] Yeah, and in certain industries even, they come with these misconceptions attached to them because they’ve been beat up in the press, or they’ve had something happen years ago that’s no longer the case. And you may have to work harder to build trust with your users. If that’s the case.

Sean: [00:18:12] Right. But I think it’s really important to distinguish the difference between marketing and when you get that customer to your product and actually earning trust with the product.

Joe: [00:18:21] Mhm. You know, when we’re doing sales pitches, we hear that there’s maybe a competitor to us, but they’re better SEO, or they’re better at this or that. We say, you know, it’s important: you’ve got to get people to your product. But once you get them there, the experience and how you move them into being advocates is even more important. And that’s what we try to focus on is that actual product, that user, and getting people ultimately to be advocates.

Sean: [00:18:42] Right. So let’s talk about incentives and paying people to become advocates. Right. A lot of companies are doing this these days. It’s like, “here’s five bucks if you prefer a friend in.” “We’ll give you free product for a month if you sign up five your friends to do this.” Multi-level marketing stuff. So lots of companies are finding success and the ability to grow through that but then are seeing a lot of fall off after they get these customers in the door. Or they’ll switch to another company because they’re not actually thinking about and earning and keeping trust and loyalty, right.

Joe: [00:19:10] Right. So you know, I’ve got friends who, they’ll refer me to things and it’ll say, “you’ll get 20 bucks and so will your friend if you sign up for this.” And the first thought I get is, “well my friend didn’t really refer this because they think this is a good fit for me and they just want the 20 bucks.”

Sean: [00:19:25] Or the free product, right.

Joe: [00:19:27] Exactly. Exactly. It does come with strings attached, essentially. So it kind of reduces the authenticity of that referral in that case.

Sean: [00:19:35] Yeah and I think that’s where there’s a key distinction between what’s marketing and what’s actually building trust with the consumer.

Sean: [00:19:42] Awesome. So we’ve talked about ways to build trust, loyalty, and advocacy, and ways to measure trust, loyalty, and advocacy, and we plan to dedicate future episodes to each one of those specific things. Today was really about being cognizant of the fact that users do go on a journey with you, and you do have ways to measure and influence how they move up the loyalty ladder into advocacy. And we have to always take into account, and again, we’ll have another episode on this at some point in the future; it’s their journey and not your journey. And this framework, this trust, loyalty, advocacy thing, is a way for us to have a healthy, thoughtful approach to measuring our long term relationships with the customers that we’re bringing into our ecosystem.

Sean: [00:20:17] We believe that brand advocates are the lowest cost path to your next hundred-year relationship, right, that long-term customer relationship that you are really after. Not just the short-term sale, or the short-term lead gen, or you know, getting that customer in the door. You shouldn’t take it for granted. We have to treat each and every advocate as though we intend to maintain a hundred-year relationship with them, because we believe in our heart of hearts that an inspired customer willing to defend and promote our brand is the best investment that any firm can make.

Joe: [00:20:44] Yeah, so any time you see any sign of advocacy or potential advocacy, someone’s wanting to be an advocate but can’t for some reason, your spidey sense should go off and you should pursue that with everything you’ve got. Once you get those true fans, those advocates, that is how these Silicon Valley companies scale so well, is they take that and they run with it and they build advocates very, very fast.

[00:21:09] (music).

Joe: [00:21:09] So Adam, thanks for joining us today. We have Adam Bates here from Amazon.

Adam: [00:21:12] Hello.

Joe: [00:21:13] First thing we should really just say is congratulations to Amazon on a hundred million Prime subscribers. That is pretty insane.

Adam: [00:21:20] Yeah, they’re getting up there. Hahaha. No end in sight.

Joe: [00:21:24] So Adam, can you tell us a little bit about what your specific role is at Amazon and what you’re working on day-to-day?

Adam: [00:21:29] So currently, I’m head of product management and UX for a business unit of Amazon that is focused on the household consumables shopping experience and how we come up with ways to really grow users’ participation in that set of products. By consumables, we’re really talking about things that you might typically buy at the grocery store or other stores related to that. I think folks often think about, like fresh products, you know, produce that sort of thing, and that’s certainly one aspect that’s available in certain markets to Amazon customers, but there’s also a much wider set of products that’s available. Snack foods, pantry type items, household cleaning supplies, your personal care supplies, you know, beauty and grooming sorts of things that Amazon offers. So yeah, we’re looking at what we can do innovation-wise to get people to partake in more of that from Amazon.

Joe: [00:22:16] So Adam, how did you get your start in building software? Was it something that you consciously made a choice to go, do you kind of fall into it, how did that all happen?

Adam: [00:22:23] You know what, I’ve always been a nerd. I’ve loved messing with tech and using it to solve tough problems, or simplifying various aspects of life. I definitely was around back before the days of Spotify. So you know, like things like getting your music files into MP3’s, moving them around, organizing them, sharing them, making party playlists, you know that sort of thing. Those are the sorts of things I would find myself just piecing together my own solutions or scripting or doing things to just make that optimal. So, you know, even before it became my profession, you know, I’ve always heard myself doing things like that. I think in general, I love creating experiences that transform how we live and work. I love diving in a new domains and, you know, getting into the minds of the users, understanding what’s really, truly important to them, things they don’t even know how to ask for. And then with, that sort of clarity in mind about what you can make their lives. Then from that, designing, building, and refining products that continually blow their mind in terms of what they do for them.

Joe: [00:23:19] Cool. So Sean, in our episode today we’ve been talking about trust and loyalty and advocacy and how we build that. And everyone thinks about Amazon as one of their favorite products, typically, and you know, there’s a lot of hard work. That is not an overnight success. And you know, we wanted to start to hear firsthand how they make that happen.

Sean: [00:23:36] Yeah, so Adam, you guys are arguably one of the best business models in the world out there in terms of earning advocacy from your users. You guys have figured it out. I mean you guys have so many great things going on. I know you have a corner of the Amazon world that you’re responsible for, but you know we’re hoping to get a little bit of insights and maybe share with our audience some of the things that you guys have learned about earning advocacy, earning loyalty from your users, earning trust, and maybe even learning a little bit about how you guys measure that stuff.

Adam: [00:24:01] Yeah absolutely, it’s critical Sean. You know, I think that’s one of the things that Jeff has really emphasized, even from the early days in the 90’s there. It’s like, “hey puy the customer first, build that trust, and maintain it, protect it at all costs.” And you know, that’s really endured today. Every team, every person working on it really has this drilled in and is focused on, first and foremost, protecting that trust. And I think that’s really how Amazon’s been so successful at building that loyalty.

Sean: [00:24:28] Certainly shows in your results. So you’re familiar with the loyalty ladder, right.

Adam: [00:24:35] Yeah, yeah. I’ve heard you talk through it before.

Sean: [00:24:38] So the goal is to earn advocacy. And if everything we do is kind of lined up with that, there’s things that you have to do to get to that point, right. So one of those things is loyalty, which we believe means that they’re using your product when they have that problem to solve. They’re not even searching for a competitor. They just, “solve this problem, I’m going to your product.” And before that ever occurs, you have to have trust, and you just hit the nail on the head there with the Bezos strategy of making sure, no matter what we do, we never fracture trust, right. Always take care of the customer. So when you think about your individual products, how do you guys measure, like what do you consider KPIs, how do you measure those things?

Adam: [00:25:13] I think that there’s a lot of ways to look at measurement. Certainly the business outcomes we’re looking to get, but, you know, that always translates to some CX outcomes, customer experience outcomes. And I think every undertaking we have has that defined very early in the process. So that when we’re pulling in the broader team, everyone knows what behaviors we’re going to look at, what cues from our customers we’re going to look at, that are telling us that we’ve hit success. We can get into a few specific examples now, or later in the discussion, that show you some of the specific instances of this.

Joe: [00:25:47] Yeah, we can definitely do that. So how do you actually use metrics? You know, a typical kind of question is, “what’s your day look like? What’s a typical day like for you?” So how are you looking at metrics on a day-to-day basis, or keeping in tune with them frequently with what the customers are doing?

Adam: [00:26:00] Sure. Well, every project has what I call a product realease score card. So, you know, we’re always releasing, and each release has some objective in it. Whether it’s a CX change that’s intended to drive more add-to-cart, or to generate more subcategories shopped, or whether it’s a behind-the-scenes technical improvement that’s designed to adjust latency in the experience so our customers are getting a more speedy response, every release has those. So we’ve got dashboard setup for each of our products that has these metrics in it. So everyday it’s one of the first things I look at. It’s like, all right how are these changes trending…? Some of these are hard to see from a day-to-day perspective, you really need to get a trend over a week or longer. It’s basically woven into every release plan and our daily routine.

Joe: [00:26:49] So Adam, just, you know, piggybacking off of that, a lot of customers, kind of a fad out there is Net Promoter Score. That tends to lean more towards advocacy because, you know, the whole idea of the question is, “would you recommend this to a family member or a friend?” What is your opinion on Net Promoter Score?

Adam: [00:27:05] Well, I think one thing that is undeniable is that you need some form of sentiment measure. But, you know, sentiment measure alone is not enough. You’ve also got to have a large-scale behavioral measurement to go right alongside that. You need both. We’re talking about loyalty here, but, you know, speaking of product loyalty, I’m a big advocate of Kerry Rodden’s HEART metrics framework that she put together while she was at Google. It’s a framework for defining metrics to measure your UX. You know, you’ve got you know behaviors or things that you think about looking at, and you put measures in place very intentionally, things like task completion rates, error rates, adoption retention figures, you know these are all bread and butter. But I find that they’re incomplete if you don’t know how your users are actually feeling about your service, so I tend to start with behaviorial first. But a sentiment measure is the best way to find gaps in those behaviors that you’re measuring or how you’re interpreting them and to verify that what you think of as success in your behavioral metrics actually translates into user happiness.

Adam: [00:28:00] So NPS is one way to capture that and I’ve found that it can be very effective if there is close alignment between your overall product and service and the specific sentiment that you’re looking to measure, and one specific example might be, say, Google’s inbox product. It’s something that a lot of people use and are familiar with. Now the vast majority of a users impression of that product come from a narrow set of experiences: a feature set that’s well bundled and contained within something that they clearly understand as inbox and they don’t confuse with adjacent Google products. Now if you pop an NPS survey there routinely, you can get a pretty good confidence that longitudinally, you’re understanding the effects of feature service changes to sentiment. However, take a very different example: you know prior to Amazon, I worked for a company called Paychecx, and they had a broad suite of service and product offerings, a mix of technology and live service. And we found that it was very difficult to isolate user sentiment of the tech products from their sentiment of the live experiences they had with service givers. So to capture the sentiment on the digital experiences we were creating, we had to turn to other means. You know, one thing that we used were micro questions placed in the context at the moment of usage. For example, “hey, how satisfied are you with this new feature, scale of 1 to 5,” so that we’re sure that we’re getting input that’s really tied to the sentiment that we’re looking to measure.

Adam: [00:29:26] Another method I’ve used with success and that provides much more nuanced sentiment quantification is UEQ, the User Experience Questionnaire. It’s basically a series of questions that users can go through on their own and it’s designed to get quick top-of-mind responses on things like how intuitive is a product, how attractive is it, how innovative does it feel? So it goes a bit deeper than simply, you know, “would you recommend it?” Now the tradeoff is that it’s multiple questions. So it’s designed and its instructions say clearly that respondents should respond to the questions quickly with the first thought that comes to them, so it doesn’t actually take long to finish. But you know when you go from one question like in an NPS survey, to several, you’re limited in terms of when you can present it. But again, that said, clear feedback on sentiment around some of those things I mentioned: attractiveness, intuitiveness, ease of task completion, dependability, is it exciting to use? Etc..

Adam: [00:30:21] You know, I might add one more point of sentiment measurement in like single air quotes here, and that’s anecdotes. This is certainly a very Amazonian thing and something that I identify with is that PMs and designers should make it a regular practice to seek out anecdotes, you know, to ask users or create mechanisms in-app to collect what they think about something specific, a specific experience, and to collect unstructured, open feedback that can be looked at. This is a highly effective way to find blind spots in your other measurements and really to say if what you’re hearing in an anecdote after anecdote is not supported by your other large scale measurement, you may have a measurement or interpretation gap and you really need to start digging to figure out what it is.

Joe: [00:31:02] Awesome, awesome. And for anyone who doesn’t know, HEART is an acronym. It stands for happiness, engagement, adoption, retention, and task success. And we’ll put a link to that in the show notes.

Joe: [00:31:13] I was at a conference recently and they were talking about, “how do we measure our software experience, or you know, how do we measure it across all our channels?” Because in the loyalty ladder, we talk about moving people up and down the ladder, but moving up on the ladder can also be online and offline because you could have a great online experience with that and have a terrible experience when you call into customer service. So measuring across, it matters. It’s great to pinpoint the experience metrics, but you can also use that feedback from the customers. You know, “I love your online product, but my god I try so hard not to call in and avoid that.” It’s kind of the whole customer service is not a department of philosophy. You know, looking for feedback everywhere. So that’s awesome.

Adam: [00:31:50] Yeah exactly. So you know NPS can be good to give you the big overview of your service and where to dig from there, and then you know, if you’re looking for more focused feedback on aspects of that service, that’s where some of these other ways of measuring that sentiment could come in handy.

Sean: [00:32:07] Cool. Just out of curiosity, I’ve often heard, So NPS measures what they might do in the future, right, so this is what their sentiment is around what they might do in the future. I’ve heard a lot of arguments about it being also interruptive, right. So the only way to really capture it is to interrupt your consumer in whatever they’re doing to collect that feedback. What are your thoughts on that?

Adam: [00:32:26] It’s certainly a balance. You need feedback, but yeah, it’s certainly possible to nag users too much to the point where they get annoyed. There’s a variety of ways to present, you know, when you get that feedback. There’s times where users can choose when they give it if you’re delivering it via email or some other notification mechanism to trigger that. If you’re going to a mode where you’re collecting that, like in the midst of an experience, you know, you’ve certainly got to be mindful of how you’re doing that. I think the days of these interstitials that take over your experience and try to get you to dive in and fill out the survey are probably gone. That’s probably a little too disruptive. But you know, you see any number of experiences now that raise it in a bit more subtle way, a more diminutive way that makes it feel less imposing, less interruptive, and personally with those sorts of mechanisms, my teams have had good success with upping our participation rate in the surveys.

Sean: [00:33:19] So you had also talked about culture, sort of. You know and how it can’t just be the whole service side of the business and the whole touch point, the human aspects of the business. How do you implement those sorts of, or it is something that you are even responsible for thinking about, how do you implement those cultural changes? Or how do you use the data that you get in context to make an impact on the cultural changes in the organization? Is that something that you have had any experience with?

Adam: [00:33:44] You know, I think in talking about some of the reasons for these measures and large-scale quantification, whether it’s of behavior or of sentiment, you know not only to direct you and your teams in terms of what strategy and adjustments you’re gonna make or innovations you’re going to go after, but also in influencing other partners. And obviously Amazon is a very large organization, and each business unit, each team has their priorities; the things that they’re focused on. The best way to influence those priorities to affect a much larger change is with data. I think that’s one thing that Amazon really values. It’s like, “hey, put the customer first, do what’s right for the customer.” But in explaining what’s right for the customer, you’ve really got to come to the table with some data that supports that. So I think it’s pretty much expected that you’re going to come up with a data-backed case to support your viewpoint of what’s best for the customer.

Adam: [00:34:32] Very cool. So just for our audience, you know, if they’re newer to metrics and they’re new to these kinds of concepts, let’s go back to your earlier offer of, what are a couple of metrics that maybe are peaking your interest rate now that you’re looking at, or maybe were quite surprising the way the feedback came back. How are you using that to make a couple different decisions now?

Adam: [00:34:51] So yeah, you were asking about some of the metrics that we were looking at recently, perhaps even tied to loyalty. How do we measure loyalty? Now typically the best metric for this is a retention metric, and these can come in various forms. You’re typically looking at this for specific cohorts, so customers that first started using the product during a certain window of time on a specific version of your software, or maybe tied to a specific campaign that brought them in and it made them customers. It’s the sort of metric that can lose meaning if it’s too highly aggregated, you know, so like across your entire user base it may get diluted.

Adam: [00:35:27] But in one example, we are looking at how do we create more of an ongoing dialogue with shoppers to better help them maintain the inventory of things that they need in their household? So as you might imagine, something like that not only is an experience you encounter when you come to Amazon deliberately, but also when you’re off Amazon and we’re reaching out to you to engage you in employing to engage tactics. So in that sort of example, I think some of the things that you might look for are, “how many notifications are they actually responding to, and does that get degraded over time?” You know, that might tell you something about the quality of the messages you’re putting out there when you’re notifying them, you know, “are the things you’re notifying about just not that important to them?” Those are the sorts of inferences you begin to make.

Adam: [00:36:12] In another example, we are looking at, “hey how do we go out and help specific brands acquire new customers to what they’re doing?” And one of the tactics we use in that is by sending unsolicited products to them on the behalf of brands. Now in that sort of example, it’s pretty straightforward. We look at the activity of those customers really by a few different groups, you know, by the targeting strategy, by the campaign that went out to try and acquire them. And then once they’re on Amazon, how many brand orders are they placing in a 90 day window. That’s a sign of their loyalty. What does the frequency distribution look like for the number of orders from folks that receive the sample? And that really begins to give us a feel for, “all right, yeah. How effective is that particular tactic in attracting loyal customers versus ones that may just try a product once or twice?”.

Joe: [00:37:00] Yeah that’s awesome. So what do you typically see it takes to earn someone’s trust to order something like that that maybe they used to get from the grocery store but to start to get it from Amazon?

Adam: [00:37:10] Great question, and it’s something that a lot of people look at here. Some things that are probably pretty intuitive when it comes to people’s tendencies to buy these sorts of products online versus at the physical store. I think some of the things that we see from users is their, “I’m already at a physical store. It’s easy for me to just throw this at my cart.” Or, “I ran out of this and I need this right now. Maybe I don’t have Prime Now available on my market, so I can’t wait a day or two for Amazon to send it to me, so I am going to go down to the store to use it.” So I think two of the big things that come out of that are, one, you know, how do we make our customers more aware that Amazon carries these things in addition to books and electronics and other things that they might traditionally turn to us for. And then, two, how do we help them with that problem of not running out of things? Are the things that we can do to help them more intelligently manage their household inventory and keep them out of those situations where they’ve got no choice but to drive down the store and pick some stuff up? So to generalize, it’s really identifying the unique problem that online consumable shopping can solve for them and coming up with unique experiences to address it.

Joe: [00:38:18] It’s such an important problem because if you really think about it, these are real problems for families. You know, running out of core household items, you know, maybe for their kids or really reducing stress day-to-day. I know there’s things that we run out of all the time and it’s like, “if we were a little more proactive of knowing we were going to run out of them, it would save us a lot of time and stress.”.

Sean: [00:38:36] All right. So let’s switch gears a little bit. Ask some questions that are related to the topic but a little off of Amazon. What other software products in the world are you personally an advocate of, and why?

Joe: [00:38:44] I think the examples that come to mind, Sean, are a combination of software product and sort of physical experience. You know, one I recently used was called Bird. It’s an on-demand the electric scooter service that was recently piloted in the Santa Monica, you know, Venice Beach area, I think San Diego as well. I’m a huge advocate of it now, a big fan of them. I like to tell anyone that will listen. It’s fun because they created a whole new fun way to get around a community. It’s so easy to sign up, easy to find a scooter, either with the app’s MapGeo feature or just by walking around, like Venice Beach, because they’ve invested in ample scooter inventory and have these things spread out like all up and down the coastal neighborhood there. And the nice thing about it too is when you’re done using the scooter, you can just leave it whereever. A combination of an app experience that just makes it so simple, and then the physical experience that’s blended pretty seamlessly with that.

Sean: [00:39:36] Oh that is a very cool idea. I can’t wait to try that out. Are they in other cities now or just Southern California?

Adam: [00:39:42] They landed a huge funding round, I forget exactly how much and who led that, but their plan is to rapidly expand from there. Another thing just kind of being a product guy and whatnot. The scrappy way that they put together that pilot, how quickly they executed it, and how quickly they are going from that pilot to expansion is amazing. You know, I think they really did this in a manner of 90 days or something like that. I really have got a lot of admiration for doing something on that scale and being able to move so quickly from it.

Sean: [00:40:12] That seems to be the way the world these days.

Adam: [00:40:14] Yeah. Hahaha.

Sean: [00:40:14] Gotta keep going faster, right. Faster and faster.

Adam: [00:40:17] Yeah, yeah. You know I’m a Fitbit Ionic advocate. So that’s their first foray into smartwatches in addition to fitness tracker. I got to admit, it’s probably not the most aesthetically pleasing watch that’s out there, but man, they really solved the problem I have with iWatch. And that’s the way that the battery lasts the whole week. You know, and it’s still got GPS. It’s got the ability to stream music without my phone from services other than Apple Music. It’s a very fitness focused smartwatch, so you know I like that they’re taking their experience in the fitness focus and branching out into a smartwatch that really matches with that lifestyle. It doesn’t make me carry a charger around with me, even for a day or two business trip. So I recommend checking that one out.

Sean: [00:40:59] Neat. I’ve had a Blaze for quite awhile, so I’ve been looking at the Ionic as a stretch into the more smartphone functionality and features. So good.

Adam: [00:41:08] One other example that comes to mind, and it’s not new by any means, it’s been around a while longer, but it is again a combination of app experience and real-world physical experiences. I’m a big advocate of Lyft, specifically over the other popular options like Uber. You know, from an app and driver experience perspective they’re not too different, right. But there is a set of other factors that have me advocating for them in favor of an alternative like Uber. I make a point of talking to the drivers, you know, whether I am in a Lyft or Uber, and drivers that drive for both almost to a person report that Lyft treats them better, has better incentives and training. And also, my personal experience with Lyft customer service is that it’s incredibly responsive and they seem to, again and this is with every interaction consistently, err on the side of the customer if there’s a question of dispute in terms of like what you were charged or whatnot. Whereas with another service, you know, I might not even get a response. So you know, that’s all there. I won’t even go into the management cultural issues that were popularized over the last year.

Sean: [00:42:07] I think that goes back to what we were talking about earlier that the culture is just as important as the software product. And if you don’t have a way to give the company that feedback and create a circular loop and really make sure that the culture is also creating loyalty, is building trust, is earning advocacy….

Adam: [00:42:24] Yeah.

Sean: [00:42:24] Lyft has clearly figured that out. I’ve had very similar experiences and have pretty much exclusively switched my preferred ridesharing service over to that, and again, it is largely because, I mean they’ve changed the market. That whole ridesharing thing has changed the market for personal transportation in so many ways, but the culture is just as important as the quality of our software products for sure.

Adam: [00:42:42] Agreed, yeah.

Joe: [00:42:44] And again, for anyone who may be just getting into software development or is maybe trying to become a product manager or somwthing like that, the number one piece of advice is probably, you know, experience overall for your product, and your service, and your company, you know, experience at every level is what’s winning the customer battle these days. And something I’ve been wanting to ask you Adam is, what do you think is the most single reason products fail, and what can be done to avoid that?

Adam: [00:43:09] I think the single biggest problem with product failure is focus on the part of the team that’s building that product. You know, a brilliant idea of this innovative business model but it having so many moving pieces and complexities that you’ve really got to align assumptions stacked on top of assumptions to get this thing to work. So for someone that’s leading a product, it’s like, hey make sure you’ve got crystal clear focus on what problem you’re trying to solve with that product for your customers and for your business. And you really kind of take the, “hey what’s the one thing we could do with this product this week, this month, this quarter, this year such that everything else that we might have to do is easier, or or maybe even unnecessary.” And man, I find myself quoting from a book I’m an advocate of here in that, um. But I think that’s critically important and teams that don’t have that focus are spreading their attention and their time too thin.

Adam: [00:44:05] I also find that for teams that don’t have that clear focus from leadership, the individual participants on the team building that product, you know what, they may be the most genuine, most skilled folks, but if they don’t have clear focus, what they’re seeing as the priority for the team could be vastly different from the other parts of the team. You know, you may have your software lead’s head in one space while your design lead’s head is in another, and you know, you may not be running in the same direction. I’ve found that when teams can focus, get on the same page, and be very specific and clear about what success means, that the results you get from that have a much higher probability of making the impact you want.

Sean: [00:44:39] All right. So I got a question for you. What do you think is the next industry that’s most ripe for disruption through experience?

Adam: [00:44:46] I think medical care is one that if you know anyone that is anything but a 100 percent healthy, you’re acutely aware of just how difficult it is to find quality providers, get them to listen to you, and have some structure around the programs that they’re putting together for you. And if your care spans multiple providers, which it usually does, again if you’re anything but 100 percent healthy, getting those providers to coordinate, collaborate, look at each other’s work and come up with something cogent for you is a huge problem. So just because of some personal experiences…

Sean: [00:45:20] That’s a big one.

Adam: [00:45:20] That’s huge right. And not just from a monetary perspective, but man talk about how you could change someone’s life that’s dealing with chronic issues or issues that are really hard to diagnose. You know like, several friends or family members that have been dealing with issues over years still don’t have clarity on like what the path to the happiest life is when it comes to their health. So I think from a customer problem perspective and from a business opportunity perspective, just so in need of disruption there. And I think there’s a number of services that are starting to attack it, you know, particularly in some of the metro areas, but I’m very excited to see what can be done there.

Sean: [00:45:58] My theory is that, because this is probably the way it needed to evolve, but the healthcare system, everything about the healthcare system is focused so intensely on the health event, right. You have a health event and everything is engineered to solve that health event. The staff is trained, the culture’s built, the facilities and the tools are all built, just to manage you through that event. And they don’t really take into context the fact that we’re human beings, we’re born and we live a life, and we’re gonna die someday, but hopefully way out in the future, and really where we need the most help in healthcare and in our health is in all those in-between spaces. In between when we have health events that the systems are set up to support.

Adam: [00:46:34] Yeah.

Sean: [00:46:34] The first one that set out is going to be the one that wins the game, I’m sure. It’s got to be about putting the patient and their life at the center and not the health event, and only to the degree at which the health events have to be solved for patients. We have to change how the entire system is structured. It’s a big change but it’s probably the most ripe industry. Looks like Amazon’s getting into that space now.

Adam: [00:46:54] That’s not one in terms of details that I’m really in a position to talk about. But you know, yeah, from the inside view in terms of the way that they attack these sorts of problems, I certainly am excited to see what we’ll do.

Sean: [00:47:03] Fascinating.

Joe: [00:47:04] You alluded to a book earlier that you quote from a lot. What was that book? Do you give it out as a gift, give it to your team?.

Adam: [00:47:09] I think the book has been around, for a little while at least. It recently came into my sphere and I’m totally caveating here because it’s one of those books that I think has a lot of kind of fluffy marketing kind of speak to it, but is still absolutely worth reading because focus is still such a problem. It’s a book called The One Thing by Gary Keller and it gives some very practical ways to achieve focus. You know, all the way from long-term strategic planning to what you’re going to be working on today to make those long term plans come to be. Again, some very practical, teachable ways to align groups of people to think in this focused manner. So yeah, It’s one I definitely advocate and say it’s worth sticking through use of the fluffier parts.

Joe: [00:47:50] Yeah, we’ll put that in the show notes as well. But you know, again, just to talk about focus again and how it relates back to our lives in software. We see our scrum teams trying to fix their problems thhrough retrospectives, not trying to tackle their problems all at once but one by one, focusing on that. And then in the product world, we see a lot of our clients or customers, you know, maybe they’re great at getting customers but they’re not great at keeping them, the retention, the loyalty part of it. So I’m going to check out that book and maybe give it was a gift this Christmas, so thank you.

Adam: [00:48:18] Yeah, no worries. To get a customer experience right, like holistically, throughout the entire phases of a customer journey is hard. It’s not something you can spend a week figuring it out, and then ship, and then you’re done. You know, like things come up and you’ve got to respond to them and it can often take a lot of time from the principal members of your team to figure out like, “hey why is this a problem?” Plus, you know, then figuring out what you do about it. So that’s why the clarity and focus from a product leader is so vital.

Adam: [00:48:46] You know, like how many different areas are you asking your team to spread their attention across? The more aspects of your strategy or the more diluted that your focus is, the more distinct problem areas come up and then you just don’t have the time to give the quality of thought into them. And so what comes out the other end is oftentimes not a great experience.

Joe: [00:49:08] A quote by Ron Swanson from Parks and Recreation, “you can half ass two things or whole ass one thing.”.

Adam: [00:49:16] Hahaha… Sage advice, sage advice.

Sean: [00:49:21] So I’m curious Adam, how do you guys handle customer journeys and customer journey mapping? Because obviously, as you deploy things, you learn that the customer’s journey is their journey, it’s not your journey. We tend to think that our customers will follow this pretty little path to get things done in the context of our software products, but the reality is that their lives are complicated; they’re busy. So how do you guys handle mapping and keeping those journeys updated?

Adam: [00:49:43] You know, I think there’s a few primary ways to look at it. The first one I think kind of maps well to the loyalty letter concept that you’ve talked about before here. These are the phases the customers go through with you over time. They’re researching a product or a solution to use, they become a new customer, they become a repeat user. Then they go through a stage where they deepen their usage and are using more of your service, not just in repeated fashion, but in a broader fashion. And then, as you mentioned, you know, they advocate which can happen along that journey. And also generally, there are users that don’t follow that happy path. So I look to make sure that we not only understand those phases, but understand why users aren’t following what we think the natural progression is, find out why that’s occurring and then to make the most of it. And then, I think once you’ve got a good handle on that overarching framework, you find that that overarching framework does not change routinely. Like if you’ve got a good understanding of your customers, you know, at that level, you kind of understand those major phases. So you spend a lot of time looking at the sub-journeys that transition them from one phase to another and what really affects those changes. What are those inflection points that happened with them, and then how do you improve those? Then of course, you can get more micro. What are the individual tasks within each phase and the steps with each? And that’s where you really start to get some nuanced behavioral metrics that show how you’re doing on that, so you optimize there. You’re then kind of bubbling up and helping speed the transition from some of those more macro phases.

Sean: [00:51:06] I have one more key question and it revolves around that end goal. So we all agree, I mean the Holy Grail, any software product, is to have your customers talking about it as it’s the nuts. This is the thing you’ve got to use, right. So I define advocacy as the point at which they invest personally in the future of your software product. That’s a key indication that this person’s an advocate. And that could come in many different forms. It could come in the form of a simple Facebook like, it could come with them actually having a conversation with their friends and family, right. NPS is kind of designed to keep some sort of sentiment measure on that. You know, we look for the actual behaviors and try to help trigger them; that’s going to come up in a future episode.

Sean: [00:51:44] Another key thing that we’ve found is an advocacy behavior is feedback. Like if your customer takes the time out of their day to give you a feature idea, or a product idea, or just to simply say, “thank you, this product has improved my life,” that’s also an advocacy behavior. Is it something that you guys measure, is it part of your product success metrics? Just curious.

Adam: [00:52:03] Yeah well when you look at the Amazon retail experience, one of the most fundamental blocks is a review. And that’s so fundamental to the Amazon ecosystem, right. And just like you said, yeah, if a customer will take the time to leave a review, it’s a signal that they’re invested in a couple of ways. You know, one they’re invested in the Amazon platform to leave a review on it. They’ve got some investment or or strong response, you know, that could be a negative one, on the product itself. But those are so valuable and they feed into the algorithms in terms of how people discover new products, when products are recommended and surfaced, and that sort of thing. And they’re signals to us behind the scenes of who are really customers that are engaged at Amazon as a platform? And I will say you maybe too, you know, a less structured way, I’ve done this at various products including Amazon.

Adam: [00:52:47] You know, looking at activity that can happen in social media is also interesting, you know, usually less quantifiable than something that’s within your ecosystem that you could measure, but you know, healthy signs that people are reacting strongly enough to take the time to tweet about that and have that be something that becomes part of their online identity.

Sean: [00:53:06] Right. We’ve talked about the investment that they make. So they make this investment in the future of your brand, they’re making an investment, use the example of reviews, the more investment they make, the more invested they are, right. Because they’ve sunk this cost in; they’re showing that they care about this platform. It’s powerful. The value you get out of an advocate is immeasurable as well. Think about Apple versus Android, right. They defend you in the marketplace. They are your army of salespeople helping you to talk about your product and sell for you. They are also, and I think we tend to underutilize this, they are going to be the best source of feedback for how we can make the product even better.

Joe: [00:53:40] True.

Sean: [00:53:40] And it’s an opportunity. You know, they honor when you reach out to them ask them for that sort of help in a non-aggressive, non-interruptive way, just as an invitation.

Adam: [00:53:49] Yeah, agreed, agreed.

Sean: [00:53:51] If you think about it, another way to look at it is they’re putting their personal name and their own seal of approval on your brand. It’s so valuable.

Sean: [00:53:58] Thanks for joining us Adam. You never disappoint. You’re an unbelievably valuable human being in the world and I value the relationship that we’ve been fortunate enough to have over the last several years.

Adam: [00:54:08] Thanks Sean and Joe. Appreciate you guys having me, always a pleasure to chat, talk shop, and especially about products and tech and customers and that sort of thing. So thanks.

Joe: [00:54:18] You got it. Thanks Adam. Talk to you later.


01 / Developing Personas


Personas have been used by marketers for decades. They are valuable because they help everyone on your software product development team to empathize with the users of your products. A great persona is one that helps your team inform those subtle feature decisions that can inspire and connect with your users. In fact, an accurate persona that is representative of your users can make an impact on your product experience that results in a truly differentiated experience.

In this episode, Sean and Joe discuss personas and their role in the software development process and then proceed to interview Jeremy Durham from Paychex about his experience developing personas for its software platform.

Read our blog post

Jeremy Durham

Jeremy Durham is the Director of Program, Product and UX at Paychex, based in Rochester, NY. Jeremy is a key member of the Product Development and Technology Senior Management team at Paychex, overseeing 200 employees. Throughout his 20 years of growth working for Paychex, Jeremy has held a number of titles, including Senior Software Engineer; Enterprise Program Manager; and Senior Manager of Product, Technology Strategy, and Client Experience.

l t i


Sean: [00:00:00] So Joe, we’ve been using personas for years now for the software products that we produce for our clients. And in general, they’ve been used by marketers for decades. So we’ve found them valuable because everyone on the team has a better ability to empathize with the users of our products, would you agree?

Joe: [00:00:18] I do. 100%.

Sean: [00:00:19] I think that better personas are those that help inform the teams around subtle feature decisions, the day-to-day decisions that they make when looking at features. They can help to inspire and our connect developers and our teams, our designers with the users. In fact, I think that the more accurate a persona is, the more likely we are to empathize with them. We’ve talked about having them databased and improving them over time. Joe, what do you think the goal of a persona is?

Joe: [00:00:47] Yeah, so the goal of the persona, in my eyes, is not only to accurately represent the greatest number of your users in your system, or our users in our systems, but like you said we want to bring the team emotionally closer to the users. Because a lot of times our team, you know they’re building it, but they haven’t really talked to the users themselves, or talked to the clients themselves, or the sales people, or the service reps, so these personas really help them empathize with those users and their core concerns. And, you know, we even go as far as identifying what the key emotions are for the users as they use that product. Lots of times we see with financially-centered products, people have a lot of anxiety, and they are nervous, and they’re scared because it’s tough to deal with that kind of subject matter sometimes. They might feel like they’re dumb because they don’t understand certain terms. But ideally, our product development team, we want them to see the world through the users’ eyes to the greatest degree possible. And so to do that, to be able to see through those lenses, you have to understand the context of our users. What are their concerns? What are the jobs they’re trying to get done? How do we help them with their frustrations or whatever that emotion is that they’re feeling when they’re using our products?

Joe: [00:02:03] So to give you an actual example, before we dig in further; I was at a conference a couple of years ago and I was at a particular session and the presenter talked about personas. He told a story about when Apple was building the Apple Watch, which is several years ago now, and when they were creating it, Apple had to make a kind of a decision for what version one was going to be. And ultimately, we can now see that they chose for it to be for health-conscious individuals. And we saw that with how they do the heart rate tracking, and tracking your steps, and the beautiful UI’s that are around your health. And so they made a persona as they were working through that of a user, in this case a man, who would be using the watch and wearing the watch. And part of the persona was that he likes to run outside. So that led to discussions around, “Okay so the watch has got to be really snug on his wrist so it doesn’t fall off when he’s running,” or “Maybe if he does drop it and he’s in the woods or something, it can’t scratch easily.” But then someone asked, “What if it’s raining outside?” And that led to better prioritization around the watch actually being water resistant. And so that’s a really basic example, but we see in our lives day-to-day when we’re using products. Like we say, “How did they not think of that?” So personas help us think of that, basically.

Joe: [00:03:23] And what’s cool about personas is they really aren’t just for software. You know, in this case this was a piece of hardware with a software element, but we can use them in all kinds of different ways. So I know Sean, when we talk a lot, we’re always talking to our users and our team members about the personas, but can you speak to what are those key elements or key things we should be thinking about when using the personas day-to-day?

Sean: [00:03:45] Sure. So first of all, you have to accept that the personas that you create will never be perfect. Your team should always be refining them and incorporating learnings as often as possible. You know, we talk a lot about the voice of the customer and we’re going to have another podcast episode, or maybe many, on how to manage voice of the customer and how to make sure that we’re getting good voice of the customer information incorporated back into our products, but it’s just as important to get those elements incorporated into the personas as well as we learn things about the people that we’re building these software products for. The second thing is that every product has multiple personas. In fact, in an idealistic world you’d have one persona for every single user. But that’s not generally economically feasible, right, so we need to prioritize and continuously refine the personas that we are serving with our software product. And our team should create an inventory of personas and decide together, “What are the right number of personas to manage and groom for the product?” Joe, I know you’ve thought a lot about how many personas is the ideal number personas.

Joe: [00:04:37] Yeah. So this is something that depending on who you ask, they might think that you can get down to just one persona for a product, or you could have fifty. And so really what we’ve found is not necessarily a magic number, but a magic range. And so the range that we typically aim for, and I’ve seen this in other spots too, other places, but what seems to work well is five to nine. And the reason for that is, like we’ve talked about, there’s lots of times many consumers or many users of a product. And that means there could be potentially a lot of personas, but we can’t be making dozens of personas and expecting a team to be able to empathize with all of those dozens of people all at once.

Joe: [00:05:16] But we also don’t want to oversimplify, because then we’re not looking through enough lenses to really make sure we’re considering what we need to for the product as we build it. So what we do is we try to think through the base users, we prioritize them, but then also to stay within that five to nine range, we look at the roadmap and we think, “Okay we’re targeting three months out, six months out, nine months out from now, that these new features might be built and these new personas may need to emerge.” And so we like to look at the roadmap and use that as a tool for, “What are the personas we need to be considering as we build the product today?” Because if we know that our persona’s name is Bob, and Bob’s going to be using the product not today, but six months from now, we want to be considering Bob as we make choices today. That can help us scale the product, it can help us build features better, structure the data better, there’s a lot of decisions that we can make in a better sense when we know who the future users are as well as the current users. So I’m kind of getting ahead of ourselves here talking about five to nine. Sean, I know you’re really good at talking about convincing people how to get started with personas, like where do you even start when you’re maybe not using persona’s today, but you want to be?

Sean: [00:06:24] Yeah, well you need to start somewhere or you don’t get anything done. So to get our software built and tested to start with our MVP’s, we always start with at least one. But how do you pick the one you start with, right? That’s the challenge. And we have a good process that we use for prioritizing personas. I’ll briefly describe it, but we can probably dedicate an entire episode just to that. If I draw a pyramid, and I talk about all the different user communities, and we identify the entire group to prioritize and choose one persona that we’re going to focus on, at least in the short run. If you don’t choose one, you’d end up trying to build things for too many people and it’s very hard to keep focus. So we always force our product development teams to have one primary persona and to balance out how we’re developing features and how we’re taking on different perspectives for a software product. So what do you think about that, Joe, what’s your experience been?

Joe: [00:07:11] I totally agree. They’re not going to be perfect when you start this, you’re going to find out what works for you and what doesn’t. It’s definitely something where just, you know; get a minimum viable persona, for lack of a better term.

Sean: [00:07:24] Yeah exactly.

Joe: [00:07:24] Just get something basic in place and just improve from there, because it’s something that you’re going to learn over time. You’re going to start to feel if you’re hitting the right persona’s or not, like if you’re actually understanding, you know, who your persona is, and then one day you actually see a user using the product that is that persona. So that’s really cool to see. But just in terms of getting started, basic stuff: understand who the core demographic is. Basics: gender, age, income level, education level, Where do they live? Climate could be a factor. Do they have kids? What’s their occupation? Where do they come from in their life? What’s their background?

Joe: [00:07:58] A lot of people think that those kinds of attributes of a persona can be really fluffy and not useful and they’re more for just making commercials and stuff. But, I love to use examples because it helps people to kind of connect the dots. We were working with a healthcare organization discussing a consumer facing product. This is basically where their customers could book a medical appointment and perform some other basic tasks through the app. And in our case, our persona was a woman, and she was in her thirties, she was a teacher, she had two kids. And as we talked through just this basic information, a couple of key insights that came out of the discussion were, “Okay, so she’s a teacher, so she’s got classes all day, which means in between classes she’s really only got a couple of minutes. And so, if she needs to schedule an appointment for one of her kids, this has got to be a super quick process with this little friction as possible.” So we can literally use that in our user testing eventually- can she get this done in ninety seconds, two minutes? And secondly, she’s got kids, but she’s got her own health to manage as well, so through the app it’s got to be really super-simple to either be able to toggle between herself or her two kids and who she’s managing appointments for or seeing test results for. So that kind of experience we’ve got to make sure it’s really seamless and she knows who in her family she is reviewing the health care information for. So that was really, really important, and just those basics got us there.

Joe: [00:09:16] I would say for the team, bringing it back to empathy, definitely give them a name. It can be a fun name, you know a name that’s not typical, or whatever is going to help. At the end of the day, sometimes the different names are more memorable, but where I’ve seen that really play off is when the team actually starts using the name, they use it in their user stories, which we’ll talk about in a future episode- how to write user stories that are really great for the team. But they also talk about it like it’s a person in their day-to-day, in their meetings, everything like that. As far as we brought a new team member to our project one day and people were talking about Anna. And for about a week, somebody thought Anna was a real person working here until a team member said, “Oh, no that’s a persona.”

Sean: [00:09:58] Yeah.

Joe: [00:09:58] So the team was really using them and empathizing with them in that case.

Sean: [00:10:01] I’ve heard stories of some of our designers actually having conversations with their personas hanging on their workspace walls. What are your thoughts on using a real person versus like a known customer that they talk about a lot versus having a fictitious persona?

Joe: [00:10:15] Yeah. So the real customer or the real user has not worked out so well. What happens is, as we’re using the persona with the team or with the client, in this case, they get a little too attached to that person, and you know, that person may have a couple of things about how they experience the product or how they use the product that really isn’t typical for the majority of the rest of the users. They might have some edge cases or specific use cases for just them, and it’s been detrimental in that case. So we try to keep it fictitious as much as possible, but then take inspiration from the real-life calls that people have received with these users or the letters they’ve received, things like that.

Sean: [00:10:51] I agree. I’ve found that when you choose an actual customer, they tend to pigeonhole and try to answer questions from that specific customer’s perspective and it doesn’t work.

Joe: [00:11:00] Exactly. Okay, so again I love giving examples. Let’s come up with a persona here on-the-spot to some extent. Let’s do a real estate agent.

Sean: [00:11:10] Sure. Let’s say Donna is a 43-year-old real estate agent who works for the largest agencies in Santa Barbara, California.

Joe: [00:11:18] Okay, does she have a dog, is she married?

Sean: [00:11:20] Let’s say she lives comfortably in the suburbs with her husband, an attorney, their two children, let’s give them names too. Sarah and Michael. Sarah’s 12. Michael’s 15. And they have a dog named Ian.

Joe: [00:11:29] And Santa Barbara is not cheap. So what kind of lifestyle do they have?

Sean: [00:11:34] Together they make about 180k per year. They vacation twice a year, once on a school holiday week, and one time in the summer, let’s say for two weeks.

Joe: [00:11:42] All right. Donna is ready. So what we typically do next is we ask two basic questions about Donna to help inform us about how she makes decisions. So what we ask is, “Is Donna more data-oriented, meaning she likes to look through numbers, look through all the nitty-gritty details, or is she more people-oriented when she is using the product, meaning she likes to hear stories, get opinions, testimonials. What end is Donna on?”

Sean: [00:12:10] Right. So when we ask these questions, we have to think about a couple things. So if our product, if our software product is aimed towards the general population, we’re going to get a range of answers here, right. But if we focus on Donna who’s a real estate agent, and we think about what Donna is doing, let’s say this software product that we’re working on is a product to serve real estate agents. What mindset is she most likely to be in when she’s performing the job that she has to perform for the real estate agency that she’s working for? And in this case would tend to find that the Donnas are probably more people-oriented, right. She’s selling, she’s a real estate agent, and she’s in that frame of mind of, “How do I get people to buy from me?” So she’ll probably be more people-oriented and intuitive in this specific case.

Joe: [00:12:52] Right, exactly. We always like to make sure if the user, in this case, Donna, more extroverted or more introverted? So when they’re making decisions, are they more vocal, or are they a little more reserved when making decisions?

Sean: [00:13:06] Right. And in this case, we would say Donna, as a real estate agent who’s in a selling mode, is going to be more extroverted, more assertive on that one-to-ten scale.

Joe: [00:13:15] Totally. Well think about just something really simple like when you go to buy a TV. I have all these developer friends who are more data-oriented and they’re more introverted, so you know what they do? They make spreadsheets and they have spreadsheets of every TV that they’re looking at and considering. And what you would want to give them is all the specs about the TV: how many pixels, what’s the top resolution, how’s the sound quality? Is it a smart TV? What are the features? Everything, and then they’ll go home and they’ll toil over which TV they’re going to pick. Whereas in this case, Donna, more people-oriented, you want to touch on, “You’re going to have great family experiences with this TV, you’re going to watch movies together, you’re going to have memories,” all that kind of stuff.

Sean: [00:13:57] But the key thing here is, we ask these two questions. So first question: “Are they more people-oriented or data-oriented?” The second question: “Are are they more extroverted or more introverted?” Answering these two questions allows us to draw a four-quadrant model, otherwise referred to as the DISC framework. That’s a whole other subject that we can talk about another episode, but it’s an important one. It allows us to just put her into a quadrant that we can then use to make decisions around the user interface.

Joe: [00:14:21] Right, and that’s D-I-S-C just to be clear for everyone.

Sean: [00:14:25] So the DISC model goes way back in social sciences to Maxwell Maltz and Marston Maston, other folks before them even. It’s kind of a socio-dynamic model for personality analysis. The D stands for driver, the I stands for influencer, the S stands for supporter, and the C stands for compliant. And we found that it’s a super simple model. It’s just a basic framework for figuring out, “How do people make decisions, interact with technology and other people?” And we use it as a way to influence our design decisions and our feature decisions in our products. So Joe, now that we’ve established where they are sort of on the personality scale, how do we determine how much competence they have? How much do we need to build them up in terms of the use of the app or in the specific domain?

Joe: [00:15:08] Right. So you know, a lot of times when you download an app on your phone nowadays, the first thing that you see when you open it is kind of like a step-by-step how to use the app. So as we’ve talked to users and listened to feedback and seen where maybe the product is confusing, we’ve essentially boiled it down to two types of competency. You can really know about the topic at hand, but you may not know how to use the software, or vice versa. You could be like super savvy , you try a lot of apps out, but maybe you’re diving into a topic that’s new to you.

Joe: [00:15:39] So what we do is we look at one type of competence that we call technical competence, which is, “Do you know how to use web apps?” You know, if you’ve got a filtering mechanism, is that something you’ve seen before, you’re used to using, like Amazon’s got. And then on the flip side, we talk about domain competence. And that is, “Do you understand the content at hand?” essentially. If you’re using a product that’s in the financial industry, do we need to define for you what a 401k is versus a Roth IRA, or are you already familiar with that terminology? So we use those two types of competencies because it’ll affect how we on-board a user, to what extent, and then also during the experience, “How easy does it need to be for them to access help features, or support features, or ask questions, or have an FAQ, for example?” So those two have worked out really well because those especially lead us to make some pretty important product decisions, especially around prioritization of how we prioritize on-boarding into our experience.

Sean: [00:16:44] Right. We’ve found that knowing where Donna is starting, knowing where our user persona is starting, gives us the ability to better construct the on-boarding experience.

Joe: [00:16:54] Right. And so I know you have talked a lot to us about the work of Clayton Christiansen. Can you talk about what he’s about, how we’ve used some of his work?

Sean: [00:17:04] Yeah, well he’s written a ton of books, so trying to describe all of his work in one podcast would be impossible, if not challenging. The next thing we do is we think about what we call the jobs to be done framework, which is from Clayton Christiansen’s work. When a user decides to use a software product, they’re using it for a reason, there’s a job that they’re hiring that software product to do, basically. And in the context of our product, we want to understand where her goals are in the persona documents. We catalog her goals in terms of the jobs that she’s hiring this product to do.

Sean: [00:17:32] For example, if we’re building an online vacation planning tool, the goals might be: “find possible vacation locations that might fit my budget, location, and timing restrictions,” or “find possible activities to do while I’m on my vacation,” or “find places to eat and sleep on my vacation.” And you could take it to the nth level with that. Once we’ve identified what the goals are, we want to identify the pain points that are standing in the way of the user accomplishing those goals and align what we’re trying to do with our goal of turning them into an advocate.

Joe: [00:18:00] I’ve been hearing a lot more about jobs to be done myself, I’ve been reading a lot more about it, I’ve got a new book on my desk here that just came for me to read as well. But I heard a certain example that stuck with me, it was from a hardware company, and they sell drills, right. So when they were looking at how to market the drill, sell the drill, build the drill, they always said to themselves, “Our customers aren’t buying our drills, they’re buying the hole in the wall that the drill creates.” And so I think that that’s kind of an interesting way to look at it. You know, your product is a means to an end, essentially, at the end of the day for your customers. They’re trying to get something done. So you want to think about , what is it that you’re trying to get done, and then how can your product facilitate that?

Sean: [00:18:39] Right. Another great example that Clayton uses in his book is the milkshake example. They found at one of the fast food restaurants that he studied, a lot of people were buying milkshakes in the morning. And it turned out that the job that they were hiring that milkshake to do was to make them feel full and fulfilled, but not too fast. Like, the act of actually taking the milkshake in slowly, because it was slow to drink, made them feel like they were full, basically. So they were hiring the milkshake to make them feel full.

Joe: [00:19:07] Yeah that’s a good one. I forgot about that one.

Sean: [00:19:12] Cool. So lastly, we establish KPI’s for the product in the context of what we call our journey framework, again, another episode that we’ll do in the future. But we have to have a set of KPI’s that we all agree on for the persona, like how are we making sure that this product is successful in the eyes of our user and in the context of our business? And with all of those things in place, every member of our team will be armed with the ability to ask themselves, “How would Donna want this feature to work?” And the result is a better product on all fronts, at least that’s been my experience. Would you agree, Joe?

Joe: [00:19:44] Totally agree, and we’ve got some other parts of our personas, or attributes of our personas, that we’ve been playing around, with trying out. But, for a starter set, if you haven’t done any persona work yet, or you’ve got some more basic personas, the things that we talked about today, these are going to be a good either first step or next step as you guys get to use personas more in your work.

Sean: [00:20:06] Cool. In terms of building product momentum, we believe that having a really solid persona is a key part of the vision. Like understanding who we’re serving and how we’re serving them will ensure that we at least have a big part of the vision for that product covered, and getting everybody aligned on that vision this is going to be key. So making sure that your team and everybody is working on the product agrees that this persona is the right persona and has the ability to influence that persona’s description in the future is important.

Sean: [00:20:33] One last thing too, we also believe that the personas need to be beautiful, like designed and beautiful so that people are proud to hang them up in their workspaces, on the walls of their workspace. Because we believe in customer empathy at the end of the day as one of the key driving factors for successful products. So make sure that your personas are easy to look at, easy to consume, and actually help your development teams influence product decisions every day.

Interview Transcript:

Sean: [00:00:00] So welcome Jeremy, thanks for joining us today on the momentum podcast. We’re extremely grateful to have you here. So we’ve worked on a few projects together in the past when I’m over at Paychex, and we’ve gotten to know each other a little bit. You’ve been doing software product development, you’ve been in this space for a long time, and you have a unique thing in the world that a lot of people don’t have today. You’ve gotten the opportunity to watch Paychex go through really building the same applications many different times because you’ve been there for a long time, and you’ve learned too. You’ve kind of grown up in the software product development space. I’d love to learn, that’s what you’re here for today. We’re going to learn a little bit about that, but our subject matter is today’s about persona. So we’re going to dig deep on the persona thing. Would you mind sharing with our audience, Jer, a little bit about your role, your history, your responsibilities at Paychex.

Jeremy: [00:00:51] Not at all. So number one, I kind of started with the company back actually in an operational role and working my way into a development organization after having some opportunities via an acquisition that the company did that happened to be written in a coding language that I had some experience with. You know, right place, right time and an opportunity to get in and start getting a little bit more involved in the technology side of things. And with that, you know, over a number of years of development and working on one of those platforms, I got the opportunity to take over and move into the program and user experience space and some other pieces. And that’s really where things started to evolve that you’re referencing as far as the numerous times that we’ve gone through some transformations as a company. And and that really happened in large part due to how the industry was changing both from a user experience perspective and from an HCM now. What was payroll, then a payroll and HR type of industry.

Jeremy: [00:01:49] And just to give you a feel for some of that, that’s kind of very interesting and the topic: as we went from a world where, as a company, we were doing very well in the payroll space and we started out very much a call-in, call-out with a little bit of a dabble into an online payroll experience, mostly just for people to drop off, you know, hours and amounts on their employees and that was about it. And as we got into a larger client space where people were having hundreds of employees and doing a lot more themselves, we started into a world where we did an acquisition where people were using PCs and entering data into there, and so they were bringing in a disconnected world and just connecting up to drop things off. And it really was very much a basic employee information and then shipping payroll. And so it was kind of contained to a handful of use cases.

Jeremy: [00:02:41] We then evolved into a world that went much more online, and through those acquisitions, started adding on more and more expectations of the user. So people wanted to have the ability to track time off online, not just to put in 40 hours a week. People wanted to have the ability to start doing things like track performance of individuals. And so, you know, we acquired a number of different products, grew products ourselves. But every single one of those things were in a world of integration from a data perspective and not really from the user experience.

Jeremy: [00:03:15] So, you know, as we did that evolution for payroll and got employee and payroll management evolved into kind of one experience online and did that update, people had all those different product lines from us but they weren’t integrated from user experience perspective. And those individuals would have to maintain things that are baseline capabilities in any one of those. So you acquire a product for performance management, and it has a way that you track basic employee information, which you also already have in payroll, which you also already have another product lines. So we had to go through an evolution to start moving those things together, and at the same time the world was coming into the mobile experience, and so we had to do a lot of work there to start to introduce a native application. But then eventually, people wanted everything to work simultaneously on different devices with the same experience. So yet again, another evolution where you bring those things together. Not all complete recreations, but you got to start to evolve those experiences one after another. And you know, that comes with a lot of great opportunities, a lot of great challenges, and opportunities to go after from both a technology and experience perspective.

Sean: [00:04:23] Yeah that’s awesome, and again, I think that what you bring here to this discussion of what we’re talking about is extremely powerful because you’ve seen these transformations occur all while working on the same sort of product set and the product domain and the same personas really that you’re targeting.

Jeremy: [00:04:41] That’s correct.

Sean: [00:04:42] In today’s day, people tend to job jump a lot, right. So to have the experience you’ve had with this one product, it’s awesome. It’s awesome for us to be able to tap into it. So again, thanks for being here.

Joe: [00:04:56] So you talked about, you know, having to go online at some point and going to mobile at some point. So the personas are typically part of the overall vision. So how do you typically just get started with that, with the vision and then working into the personas, where do you even start?

Jeremy: [00:05:12] For us, you know, the big thing is trying to understand the various users and what their needs are before we even start into what the vision is for where we need to take things. And part of it that gets extra complex, for the stuff that I’m involved in, is the fact that we have a number of different types of users that can be into the same system. So everything from, an employee who just has to go in and punch in an hour or maybe look at their check stub, to an administrator who is responsible for everything in a certain sized company. But the extra complexity comes into play where you have a system that can have a client that’s, say, a construction company with a handful of employees and then a company that has, you know, a number of locations of the same business, say a restaurant or something like that, and they’re all co-owned by the same person. And some may be using a CPA who also comes with the system, or not. So starting with that user, that persona, and understanding what the needs are is critical. And then from there you can build into what, from those needs, what the vision might be, and you really need to understand from that that you may have varying needs amongst them, even in the same exact system.

Sean: [00:06:33] You guys have this unique scenario too where your payroll manager is also an employee. So you have to be able to jump between different personas, even for the same person, because they’re going to have the same needs in different scenarios. It’s interesting.

Jeremy: [00:06:54] Yeah, I’ll tell you what, it brings some real challenges that are fun to try to overcome in the sense of creating a world where you may have somebody who needs to go in between both in the same session inside your application. So for us, you know, we found it best to be able to create an experience that’s very much driven towards them operating as an employee, and then one that is clear that they’re operating as the administrator. So that way they can understand the difference between systems are in when they’re operating is what’s impactful to them personally, and also so that we can have an experience that, for an employee coming into our system, it feels a system that they want to be in and it doesn’t feel so institutional and transactional. But for an administrator, it can go the other way.

Joe: [00:07:46] Right. So we talked about where you start. And it sounds like you’re really starting with the user, which is the persona. And so, if that’s the case it’s got to be really, really valuable from the start. So in your opinion, what do you think makes for a really valuable persona?

Jeremy: [00:08:03] And so for us, you know, it’s getting and understanding the various types of personas that you need and the needs inside of each one, what might be motivational, but also what might be a frustration or a distraction for them, as well as really understanding the varying skills that may be brought to the table. So when you have a very diverse base of users and you get into them, try to fit them into a reasonable, manageable number of personas, it’s really important to understand that you may have somebody who, like I said is out on a construction site, rarely ever goes onto a piece of technology, and you may have somebody who might be a data guru and wants to do really complex analytics and things with the system, and understands technology really well, and may jump between devices even in the same day. “I want to start something on the desktop and move on to a tablet or a smartphone in the same even even transactions at the start and stop.” So understanding that diverse user base and getting it into a manageable number of personas is really critical.

Sean: [00:09:20] All right. So you mentioned the diversity of the customers, right. So in your space specifically, let’s talk about the employee products. Your consumer base, your persona, is essentially everyone. Everyone who works, which is most people, most adults in the world, right. So how do you hone in on individuals to make the solution more personalized? Is that something that you…like how do you think about personas from that perspective…like how close can you get to the actual needs and desires of the employees, and how do you handle that?

Jeremy: [00:10:00] Yeah, so we do a couple things to try to make sure that we’re really understanding the user base that we have and what their needs are. You know, everything from what we’re doing just from analytics to get profile data and information about the individuals that are using our system to focus groups and some user research, both industry-wide and and inside of our system and inside of our client base. Because, you know, it’s really easy to look at industry trends and what is going on out there, and people talk about things like the difference of a Milennial user today versus somebody who’s maybe a Gen Xer or a Baby Boomer. If you start designing for things that maybe haven’t actually occurred in your base, you can get yourself in trouble. But also, only catering to your current base and not being ahead of that can be the same issue that’s just lurking and waiting to hit you. So for us, it’s very critical for us to not only survey and pull real life users into our both research activities our testing, but to also get those people to have the opportunity to provide in-app feedback to us. Combining that really gets us to kind of an analysis that allows us to know what’s really going on and what people are using, and what’s important to them.

Sean: [00:11:21] Right. Do you actually incorporate that information into some sort of an artifact, like into a persona artifact, or does that data sit, because that’s a problem I see in the industry. There’s a lot of user research and data analysis going on but I don’t see a lot of companies really taking advantage of the research that they’re doing. Do you guys feel that you do take advantage of a lot of the research, how do you do that? How do you take advantage of it? Like how do you put it in front of the people that are building the products to actually make it useful?

Jeremy: [00:11:54] Yeah, sure, so we have a couple of different groups inside of our overall user experience organization that handle getting that information. And obviously with the number of things that we have going on at any given time, we have ninety-plus teams agile working on our software. We obviously have to go after the most critical projects to do that research, but we have a team that does research and another team that does field studies. And they develop a set of content that’s put in front of the agile teams as they’re getting through their release planning and getting ready to kick off an effort so they have an understanding of what those key components are and what the feedback is of our end users. In regards to your question of, does that translate into like a specific persona that’s managed? We definitely do create personas. I think to your point, it’s a little difficult, still. There’s not a great way to keep those things clear and up-to-date that fits the need of every single one of the different efforts going on. So we find ourselves actually producing that content; feeding it into our design team and our leadership team that does the vision. And for us, we start a vision and build it into detailed designs as we go, and that’s based off of both the information coming from our product teams as well as what we develop from that research and those field studies.

Sean: [00:13:23] Yeah ninety teams, man, that’s amazing. Ninety agile teams, that’s huge.

Joe: [00:13:28] Well you think about ultimately how many people that is. You want them to be empathizing with the users, and we do that through personas. That’s incredible, you know, to think about the scale of that and how valuable a persona can be in making all those people thinking about who these end users are as they build the product. So let’s say, you talked about rebuilding the products and the history, and that’s excellent. I know you guys are coming up with new ideas, brainstorming new products. So when you when you have a new product idea, do you approach the vision, and then the personas obviously, differently than for when it’s an existing product that you’re rebuilding? How do you go about that?

Jeremy: [00:14:12] Absolutely. I think there’s parts that overlap into both, but one of the key things for us is that when you get to a point where you have a large enough user base like we do, you know, eleven, twelve, thirteen million sessions at any given time, you know, millions of users are logging into our system. When you’re changing over something that already has an existing base like that, you have to be very cognizant of not losing them and losing that engagement from that group, while also knowing that we are going to continue to add more and more new users as we go. So that one kind of has to fit both worlds. Whereas when you’re building something completely brand new, you kind of get that opportunity to start from scratch and not have that drag effect of, you know, the way things have been done in the past.

Jeremy: [00:14:59] And that’s what made things so challenging for us. When you go through all of those transformations we talked about before, there is this sense that you get varying groups who have gotten very comfortable with what they’re doing, and so developing something that’s very simple yet powerful, something that is very intuitive and makes people feel like they can jump right in and use it day one is critical to not lose the engagement, that base that you already have. And again, like I said, you still want that simplicity in the new product, the new users, but you have to take a very different approach when you have an existing base.

Sean: [00:15:34] So you mentioned earlier that when you make a mistake it can be significantly detrimental. Especially with the tools and the products that you’re building and the reach that they have, I would consider that important. Do you have any examples of where your teams have made a mistake on a persona and what happened, what was a result of that?

Jeremy: [00:15:57] Absolutely. I wish I didn’t have any, but we absolutely have some examples where that’s occurred. One great one for us would be inside of some of our reporting capabilities, reporting something that spans all of our product lines, and people want to be able to get things in and out of your system quickly. And one of the things we had, at the time we did our first real redesign of our reporting capabilities, was we made a bad judgment or assumption in that we felt that the more capabilities and the more sheer volume of reports that we provided to users, that that was going to add more value to them. Just thinking that know if we got 100 reports, 150 reports is even better. If we got a 1000 data elements, 2000 data elements is even better. And, you know one of the things we learned after rolling some of those capabilities out it was actually more important than the two or three things that were really critical day in day out, week in week out, for our users, especially those who are our clients, like a CPA, was way more important than the volume. In fact, the real thing that they were trying to was to get to answers and not really get to a report. So that actually ended up allowing us to make really valuable changes to our product. To give answers to the user right as soon as they go in, we know is one of the top things everybody is looking for, and giving the ability to get to those other capabilities, but not at the service level, because that wasn’t what they were doing on a regular basis. So again, for us that was a failure, and you know, fail quick and learn, right. Well that was one where for us, where we kind of made an assumption that volume was important, and it was really more speed to a handful of answers.

Sean: [00:17:43] That’s a great answer, man. Gives us something to think about for sure. Do guys you have a standard that you use for personas? Or is it just team by team, design team by design team?

Jeremy: [00:17:57] The standard for us really comes in the fact that we have a handful of people responsible for it.

Sean: [00:18:02] Yeah.

Jeremy: [00:18:04] So they have the standard in how they accomplish that.

Sean: [00:18:07] That makes sense. And are there any tools that you use to kind of keep them aligned or to visualize them, other than the standard things that you see?

Jeremy: [00:18:17] Yeah, for us there really is no secret sauce or secret tool when it comes to that, we really just have a very basic set of personas. There’s plenty of tools that we from the overall design process, but for personas themselves, they’re very basic documents.

Joe: [00:18:33] Cool. So you know, the podcast, our listeners we’ve got people who are pretty experienced, pretty seasoned, and then also people who are maybe coming out of school and whatnot. So what would you recommend to someone who is just getting started with creating and using personas? And then, the second part of the question, if it’s separate or related, either way is fine. Any kinds of pitfalls or misuses of personas that you have seen that you’d like to point out for people to avoid?

Jeremy: [00:19:03] Sure, so a couple things on that. I would say number one, in the question of people just starting out, I actually happened to be talking to the overall user experience team today in an event that they were doing, and someone made a comment and asked a question about the interaction with somebody who was newer to the team. They brought up the idea that there’s an emotional connection in the world of dealing with user design and that, you know, that piece or that interaction in that engagement with the users is so important to people in that field. And I think that’s something that’s really important for somebody starting out. Realizing that, you know, that part or understanding what kind of makes the end user tick, that those things that like I mentioned that might be frustrations, versus things that might be motivations inside the system is critical. As far as pitfalls are concerned, I think it’s important to know that, one, it’s gotta be a reasonable number or you’ll get lost in the activity. So for us, you know, if we put out a whole grid of all the different components that could go into this with different user types, you’d be stuck in analysis paralysis there trying to figure out how to design a system that might work. So to me, getting down to that smaller number is really important.

Sean: [00:20:26] Of course, that makes sense. The industry has got some magic numbers that I’ve read about, about well, “you should have this many personas for your product, but not too many” you know, you got to find that balance. Do you guys have a balance, or is it something you just kind of figure out by product?

Jeremy: [00:20:41] You know what, in most things that I have responsiblity for, I try really hard to keep the teams flexible as much as possible to say, you know, let’s not set a standard that people get caught up in the black and white of something, and kind of have the ability to kind of adapt to what they’re doing. But as a general rule of thumb, I usually tell people that regardless of what they’ve read and what numbers are out there, single digits is really important to us.

Sean: [00:22:12] That makes sense.

Joe: [00:22:12] Yeah, agreed. So you’ve obviously seen a lot of software products either succeed or fail, what do you think is the single biggest reason that products fail, and any advice on what could be done to avoid that?

Jeremy: [00:22:12] Yeah, so for me, I would say probably the biggest reason something fails for us in what I’ve seen in my experience is what somebody thinks that they know better than what the users tell them.

Joe: [00:22:12] Ah, good one.

Jeremy: [00:22:12] And so you need to get out of your own way and realize what the end users really want. And I can tell you that I’ve even seen it in my own experience. For example, thinking that something would be really important on the dashboard for somebody and over and over people were telling us, “I don’t want to see that there” I felt like that was probably just a small number of people saying that, so we reduced its size, reduced its size, and eventually really just had to give them the ability to block it and ask it. So there’s really just so many of those, and if you can’t get out of your own way, then you’re going to fail.

Joe: [00:22:26] Great answer. Loved it.

Sean: [00:22:28] All right, so Jeremy, one of the things we put into our personas in a pretty standard way is we use the DISC profiling method. Are you familiar with DISC?

Jeremy: [00:22:39] I’m not.

Sean: [00:22:39] It’s a four quadrant model, it’s very simple to understand. There’s on one scale, it’s introverts versus extroverts, and on the other scale it’s data-oriented versus people-oriented. And if you’re more data-oriented and you’re extroverted you tend to be what’s called a D or a driver, and if you’re more extroverted but people-oriented that would be an influencer or an I, on the bottom half it’s introverted but people-oriented that’s a supporter or an S. And on the left side it’s compliant, a C, and those are people that are generally more introverted, but more data-oriented. And we use that because it’s a very simple sort of structure for figuring out what mindset is this person, and by the way we all will fall into all four of those quadrants in different places in our lives when we’re making certain decisions, right. If we’re making a big financial decision, we’re going to behave very differently than if we’re looking at our paycheck and the information that’s in the app for our paycheck, for example. So we try to help our people get into the mindset, the same mindset that the consumer has when they’re using the software products that we’re building. So we’ve been using that tool. What do you think about that approach?

Jeremy: [00:23:58] You know, I think that fits very well into some of the other topics we’ve had, or discussion we’ve had today, around the fact that you need to understand the different ways that people are comfortable and the skills they have, what they’re bringing to the table at the time you’re using the system.

Sean: [00:24:11] Yeah, we build a lot of things bespoke from scratch so we have to guess. We don’t really know what the users, what mindset they’re going to be in, so we this is how we start. So we use that model just to think through, where do we think the customer or the user of the software product is going to be? And then we can obviously test that later. Like once we get a group of consumers using the product, we can ask them a couple of questions to figure out where they are when they’re thinking about that.

Jeremy: [00:24:37] You know, it is great to have a framework like that, and I think we talked earlier about the new system versus trying to make a transformation or an update to an existing system with an existing base and some of the benefits you get with a new system. That you can start from scratch, as you mentioned, brings kind of that challenge of not having a base of the users to understand through analytics of what they’re already doing and what’s important to them.

Sean: [00:25:04] Right. But you got to start somewhere, right. That’s our theory. And then you can groom it. We groom our personas just like we groom our user stories and other things for our products. So the other thing, the other tool that we kind of have that we think is somewhat unique is, we use these competence scales. So we believe that we have to really know how competent the consumer is on two different dimensions. One being with technology, which often comes from the age demographic and the profession demographic as well, you can pretty much determine, “how competent is this person with the technology that they’re using?” The other competence scale is in their domain, like “how domain competent are they?” So if solving a problem in payroll, like how competent are they about what’s being taken out of their paycheck, for example, right. And we’ll build the software products very differently depending on where they are on those scales. Like if they’re low competence, we have to educate them more, or use more onboarding tools in the user interfaces, right. So those are other tools that we’ve been using. Do you have any thoughts on that?

Jeremy: [00:26:12] I think those are critically important in understanding the users and like I said, what their skills and capabilities are. I think the one other piece I would add to it if I was in your guys’ space, is you know, understanding what the engagement means to that user in a sense of, like you know, in a marketing system you’re trying to keep the person engaged and drive them to a sale. Whereas for another system, like you just mentioned for us, like getting and seeing your check stub, out you might want to be getting in and out of that is quick as you possibly can. So understanding what they bring to the table in skill set in understanding and knowledge, and then also what’s really driving them.

Joe: [00:26:51] Yeah. We’ve been using a lot more design thinking lately and so in the personas we do a really just basic and simple scenario. It’s just five steps, you know, kinda just whiteboard it out real quick. Like you were talking about earlier, what is their pain point potentially, or what is their goal, what are they trying to get done, and where does the technology involved in that become involved in their life, in solving that problem? Are they on the go, are they not, are they at home? What kind of time do they have? All that fits into that, so that’s totally onboard with that.

Joe: [00:27:23] So Jeremy, when we were trying to schedule this podcast, you were on vacation. Do you mind sharing where you were to make everyone jealous?

Jeremy: [00:27:30] So I was in Aruba for about nine, ten days. And it was gorgeous. An awesome, awesome place with a lot of great people, and the weather is just so consistent day in, day out, all year long that, you know, for me, if it wasn’t for the necessity of sunblock sunblock sunblock, I might never want to leave.

Joe: [00:27:57] And so it was very hard to set up an offshore bank account?

Jeremy: [00:28:03] Hahahaha, really wild there, nice joke. But you know what’s really wild is that I looked everything up there of what it would take to live there, and something crazy like 59% tax rate or something like that.

Joe: [00:28:17] Oh God. Hahahaha.

Jeremy: [00:28:17] And only your first, if you’re still US-based, only your first like 108 thousand dollars is only taxed in Aruba. After that, you still get federal tax on top of the 59%. Not any place for ex-pats to go and live.

Joe: [00:28:17] Yikes.

Sean: [00:28:17] Ouch. I think Bonaire right next door is a little more friendly. Somebody told me that.

Jeremy: [00:28:17] Yeah. But we’re a tax company so don’t quote me on that.

Sean: [00:28:17] Well thanks, Jeremy, thanks for joining us today and sharing all your great knowledge about personas and how you guys have used them, obviously for many years. I really appreciate your joining us and hopefully we can do this again sometime in the future.

Jeremy: [00:29:02] My pleasure. Thanks for having me.


Enjoying the podcast? Sign up for email reminders to be the first to know about new episodes.