“Dear Data Scientists…” or What I’d tell my former data scientist self, as a product manager
After my role in Azure data sciences, I transitioned to product management in Microsoft Power Apps. I now oversee user experiences for the Power Apps home page like first run experiences and how users leverage the native data platform, called Dataverse. I made the switch to PM because I wanted to be at the center of software product development, rather than in supporting role. The change has been rewarding and incredibly humbling as I’ve reconfigured my skills and mindset.
Having spent years delivering business insights to support Azure programs, I am often asked, “What have you learned as a PM that would change how you approach data science?” I distilled my perspective below. Hopefully it helps data scientists gain a wider perspective.
*Note- yes, “data scientist” is an ambiguous term. For our discussion, it is anyone who’s called upon to work with data at scale, analyze it with statistical methods, and present the information to influence managerial decisions.
- Understand the totality of decision making
Although every team strives to be “data-driven,” the fact is that the decision-making is a messy result of negotiation, time available, and multiple inputs. Most product decisions involve stakeholders ranging from UX research, UI design, leadership, product, engineering, sales, finance, and marketing. Data can be used as a powerful steering mechanism, but it can’t capture and weigh all of the variables. In fact, I bet that you have been in a meeting where critical decisions were made with minimal quantitative backing.
Knowing that other stakeholders will have their point of view, I think DS’s should get to know the point of view of these teams ahead of time. Too often those angles are overlooked and we, data people, believe that it is enough to say “but the model recommends,” or “the data says.”
2. Time value of information
I can’t emphasize how important it is to develop answers quickly and get that into the right hands as soon as possible, even if it is preliminary and directional in nature. However, typically data scientists need to spend time pulling (then cleaning) data, developing analysis, peer reviewing, iterating, then finally presenting their insights. The risk is that by the time the meetings are scheduled and information presented, the organizational attention has moved on to the next pressing issue. We’ve all been in those situations where a meeting gets rescheduled and suddenly the analysis drops off of the agenda. It is incredibly frustrating.
Data teams that are centralized and not embedded with product teams are especially susceptible to these priority changes. For example, if a product team needs to quickly pivot their roadmap to respond to competitive pressures, then their attention is on re-prioritizing engineering sprints. Giving the heads up to partner teams often gets lost in the shuffle and can come quite late.
As a data scientist, think about the value of the information you generated with a time-decay function. There’s a critical window in which your analysis can deeply influence business impact, but after that, its relevance drops precipitously. Timing is just as important as the validity of the analysis.
3. Be bold. Embrace ambiguity. Don’t hide behind methodologies.
How many times have you seen a DS memo that reads like an academic paper? I know I’ve been guilty of writing emails like this…
“In this paper, we present a data-driven approach to optimizing Call-to-Action (CTA) conversion rates on websites.”
“Our approach combines A/B testing with machine learning to identify the most effective CTA variations … We use logistic regression and gradient-boosted decision trees to model user behavior and predict conversion rates…”
Avoid this temptation. Deliver insights that are punchy and applicable. Give the answers up front.
When presenting their work to a business audience, many new data scientists are surprised to learn that most of the audience’s questions are about the implications of the findings, and not how they derived it. Generally audiences trust that the methodology is sound. After all, it is what we pay smart people to do.
But yet, many data scientist can’t resist the urge to over-explain. Why? Because the nature of the job requires that the data scientist be “buttoned up” and have every answer. This is instilled from their academic training, to the job interview when they’re asked a variety of technical questions, down to peer reviews at work.
Having worked with many DS’s smarter than I am, I loved the peer reviews that instilled rigor and new ideas into the work. However, the downside is that the need to be “buttoned up” often drove analysis paralysis, ending up in iteration after iteration with decreasing marginal returns.
Be bold with your recommendations. But how do we deal with uncertainty? De-risking is the key here. Let your audience know what set of assumptions went into your answer. Then, explain how your answer would change if those assumptions were to change. Effective product leaders live in a constant state ambiguity, never knowing the exact right answer but balancing risk-rewards to get more rights than wrong. As a DS, take a similar approach. Your audience will be more forgiving than you might expect.
But what happens if you’re proven wrong at a later time? I think about it this way. I want to get the right answer, but that doesn’t mean I have to come up with it all the time.
