In this session, we asked Adam Fishman, the former VP Product/Growth at Patreon, about what opportunity he sees in Loops for product managers, and to share some tips about experiments and alignment between product managers and data analysts. Enjoy.
Transcript
Tom: Great to have you here with us at Loops.
First question I wanna ask you is why Loops? Why did you decide to invest in Loops?
Adam: It’s a great question. I’ve spent close to 20 years steeped in data, spending time in spreadsheets, then dashboards, then online analytics tools, SAS tools. I think the fundamental challenge has always been, where should I pay attention? There’s so much to consume, there’s so much digging that needs to be done, it’s hard to find some of those nuggets and insights sometimes.
For me, anything that can shortcut that process and say, “hey, Adam, look over here, maybe you and your team should be paying attention to this, maybe you might want to think about this”, and narrow in the aperture on the right stuff; that’s what’s important for me. It speeds up the work, the planning, the opportunities, everything. That’s why I think more people should have that ability, through software, to narrow in the the aperture of what they are paying attention to, and Loops gives that to you.
Tom: It makes total sense. You know, one of the things that eventually Loops is about [is] the insight that you mentioned, that are supposed to be translated into experiment, into initiative that you are eventually doing.
One of the things that I’m really intrigued to ask you is what is the misconception that you think growth teams have around experimentations? What are the things that they are missing in their experimentation plan? I know you’ve been talking about those things as well, but what’s your biggest insight there?
Adam: Yeah, it’s a great, great question.
Some of the things that product managers and growth teams miss with experimentation is this idea of what is actually going to be an impactful experiment that helps you learn something more about your users and truly tests a theory that you have or in an opportunity area. I think there’s too much experimenting for experimenting sake motion over progress, I call it. I think the thing about product like Loops is it helps you tone down the motion and turn up the progress.
And so that’s what like about this, and one of the kind of common challenges that teams have is just doing too many experiments that don’t have any real bearing on your learning or your eventual business results. This turns that on its head a bit.
Tom: How do you usually avoid this from your experience?
Adam: I think one way to avoid it is you actually don’t avoid it and you learn through some painful lessons that you’re not focused in the right areas and that’s through repeated failed or non needle moving experiments or, you know, no lessons learned coming out of an experiment, things like that, and so you end up spinning your wheels and wasting a lot of time, and I think most people would say that’s happened to them at some point in their career before you see, before the the introduction of something like the Loops product.
Tom: Yeah. I think one of the things that, eventually, to be able to focus on the right experiments, you really need to have the right insights. Now, to have the right insights, there are many two stakeholders that we see at loops, right? One is the growth product manager, and the data analysts. This relationship is something that we see is key, this collaboration is key for basically success of your growth initiatives.
I wonder if you have tips from your various experience working with various data science analyst, data science teams, tips for better alignment for product and growth team and the data analyst team.
Adam: One of the things that we’ve done at many of the companies that I’ve worked at is really bring the data analyst into the early part of the planning process and the prioritization process as a equivalent stakeholder to help them understand the type of business problems that we’re looking to solve, and give them a little bit of a view into “this is where you should focus your efforts because this is actually what we believe matters right now for the business or our strategy” or something like that.
Just treating them like a partner as opposed to somebody who clacks away on the keyboard like a puppet, and writes the sequel queries that you need, I think is much better, because I much prefer data analysts who then can go do independent exploration and find things that you might never have uncovered in your strategy. The only way to do that is if they know your objectives, they understand what you’re trying to achieve and they feel like they have skin in the game and that success is based on the combination of their ability to surface things to you as well as then your ability to action those things.
Where that falls apart is when that person doesn’t feel like part of the team, they’re either surfacing the wrong stuff that’s not actionable or the wrong stuff, that’s not really a priority for you, or they’re surfacing stuff and you’re sort of on a different planet in terms of what you’re working on.
So my hot tip is bring them into the trust circle, the trust tree, and get them doing the analysis that’s actually aligned with your product strategy or your growth strategy.
Tom: I think you see a major difference, Adam, between those that build dashboards for the purpose of building dashboards and the ones that understand the business and as a result are able to move the needle.
Speaking of moving the needle, eventually, we see that across the industry, if you have data, but you don’t really leverage it, if you don’t actually transit it into a real initiative, into real taking action, those insights obviously worth nothing.
My question to you is, from your best experience, if you have an example of a real case study where you actually leveraged, you saw some insight from the data, you translated into a clear hypothesis and you saw a major win in your online metrics.
Adam: A really good question.
I’ve seen a bunch of these. One of the examples that I can give you is kind of related to my time at Patreon.
One of the really interesting challenges that we had was how to identify, amongst all of the creators who were onboarding to the platform, who was worth intervening in the experience, and trying to handhold a bit more, because they had a really high potential for success.
And so there are a few stages of data analysis that we went through. One was looking at the characteristics of creators who were very successful to try to understand, quantitatively, what set them apart from everyone else, and it wasn’t just follower size, it was also a deeper level of the engagement that they have with their fans off platform. It’s really mining external data sources to figure out that “this is what a good creator, good high potential creator looks like, this is one that maybe looks good on paper, but probably won’t be that successful”.
That was sort of step one and then step two, once we kind of knew those data points, that mattered from partnership with our data team, it was how do we identify them in real time in the experience? What data do we need to collect What friction should we introduce in the process so that we can more appropriately bucket people in real time and then decide if we should intervene in that moment to try to get them connected to a human being.
So it’s a little bit of that kind of bridge between product led growth, product led sales and sort of surfacing that human touch point. And there was a lot of experimentation there on what mattered and how you surface people and how you got in touch with them. That was a third phase of sort of data analysis and experimentation that we did.
But what we ended up seeing at Patreon was that if somebody was worth intervening with, and we did that intervention, that person, that particular creator had 25% higher first month revenue, And why that matters is because first month revenue is very indicative of your lifetime value on the platform. So if you get somebody off to a better start, they will be much more valuable.
We did that through really rigorous controlled experimentation to actually prove out that theory, that it was causal in who we interacted with and when we interacted with them and that the human touch had an impact for the right people.
That’s a story that I like to tell because one, it has multiple components of doing rigorous analysis and follow on analysis, and follow on analysis. Two, it’s full of a ton of experimentation to validate what it was that we were seeing in our data. And then three, it shows that the hard work that you put in there does lead to success.
And that started from just a kernel of an idea and ended up leading to a 25% improvement in first month revenue from creators. And that’s a really big impact to the business.
That’s my story.
Tom: It’s amazing story. I like it for so many reasons when it comes to what Loops doubles down on. From the depth of this the segmentation, instead of just slicing it by different basic things, devices, countries, what usually companies do, you went deeper there into the segmentation.
Second is instead of looking at just general engagement level and conversion, you actually look at the ones that have high intent, that should succeed with your product.
Usually company looks at “I have 2% conversion rate, let’s try to generally prove it.” No, let’s focus on the ones, within our ICP, that they should really succeed with us.
Obviously, you mentioned our keyword, which is causal, you prove that it’s our flagship models that we focus on those model. And iterating, and you completed the Loop of iterating on this hypothesis and improving that.
That’s a great example.
Adam, thank you so much for your, your time for having us, thank you.
Adam: Of course, of course.
