To be honest, no one asked me. Those models, everybody should know, I mean every time, instead of just... it's a core concept of ML. So basically, yeah, you should understand the fundamentals of this, because those are models.
You should know machine learning fundamentals, 100%, but it's not a deal-breaker. Sometimes, I mean, it depends on your role, it depends on your professional level. If you're senior, yeah, you should know those type of things. This is a core concept of PM. If you want to work on any product, you should know details. That's the core thing. Keep it in your mind. Bear it in your mind.
Where This Vocabulary Shows Up
One of the sources in my pile keeps a running bank of AI PM interview questions, and the technical section reads like a pop quiz on exactly these five things: precision versus recall, overfitting, supervised versus unsupervised learning, model drift, asked cold, with no prep material behind any of them anywhere in the same guide. scan of 100 live postings puts "ML/AI Fundamentals (LLMs, model evaluation, inference)" at 83%, a base skill sitting next to roadmap prioritization and stakeholder alignment, not a specialization. The gives it an entire required phase before you're even allowed to touch LLM-specific vocabulary.
The basic thing is, it's not necessary to hand people interview guides, because the role is quite new, and that's why every time someone tries to guide people about the interview, they quite possibly mislead them. There's an opportunity there instead. What people trying to convert from traditional PM to AI PM need is to know the core concept of PM, and additionally, understand the fundamentals of AI. That's the core thing, and it's the most important part. Post 1 already covered why the job runs on probability instead of certainty now; this is the vocabulary that probability runs on.
Here are the five, one at a time: what the term means, why it lands on a PM's desk instead of staying inside the data science team's, and what it actually looked like the last few times I ran into it.
Training vs. Inference
The Playbook splits the learning process into three phases, and only the first one is training: "The model is fed a large dataset (the training data) and identifies patterns between the features and the correct outcome... this is like showing the child many examples of each fruit" (Playbook, ch. 2, p. 20-21). Everything after that, the model sitting in production and answering real requests one at a time, is inference, and it runs on a completely different cost clock than training does.
Why a PM has to care about the split at all: deployment is where the two costs stop being theoretical. Choosing where a model runs "based on factors like cost, latency, and scalability needs" is explicitly framed as a PM decision, not just an engineering one, because it decides "the implications of the deployment environment choice on user experience (latency), cost, and scalability" (Playbook, ch. 10, p. 267-268). Training is a project, something that happens once or every so often. Inference runs on every single call once the thing is live, which is why it shows up on a bill.
I have dozens of agents. I do a brainstorming session with them, but at the end I design the product myself. Before I started developing, before doing any vibe coding, I prepared documentation for almost three days, with different phases, and I ran that, and yeah, the first product had some bugs, of course, but the system was okay. And that's helpful also for token burning, for AI. That's the part where training and inference met for me: the three days went into shaping what the model would be trained and prompted on, and the token burning started the moment real lessons started generating for real users, one request at a time.
Supervised vs. Unsupervised
"Instead of training a customer support agent with labeled tickets (supervised learning), you're now giving a market researcher a massive dataset of customer purchase histories without any predefined categories... asking them to find interesting patterns, groupings, or insights, without telling them what to look for" (Playbook, ch. 3, p. 55). Supervised learning means somebody already told the model the right answers ahead of time. With unsupervised learning nobody did that, so the model has to find the shape of the data on its own.
Why it matters for a PM is less about the algorithm and more about what it implies for your roadmap: labeled data has to come from somewhere, someone has to define the categories, and that work has a cost and a timeline attached before a model can ever ship. Unlabeled data skips that step but trades it for a model whose output you can't grade against a right answer, only judge.
What will be challenging when you change your title from traditional PM to AI product manager is that you could get those type of questions from anyone, doesn't matter who. All you need to do, if you know the product, you know the answer already, because you designed it or you validated the design. It is, because before designing the app or before developing it, I took my data schema and did dozens of rounds on the infrastructure, the architecture, the information architecture, blah blah, and yeah, of course this should be the answer already. But if I'm gonna check the analytics, and if I need to understand supervised versus unsupervised for something specific, this is quite an easy question. Go and check the analytics tool, you will find it, write a query and find it, because I already did that.
Precision, Recall, F1
The book walks through why a single accuracy number can lie to you: a spam filter that always predicts "not spam" on a mailbox where 99% of mail is legitimate would score 99% accurate and let through every single piece of spam. "This forces you to ask the essential product question: what type of error is more harmful to our user?" (Playbook, ch. 9, p. 233). That question is what precision and recall are built to measure, and the book's own worked example is a churn model that answers it on purpose: "Primary Metric—Recall (Sensitivity): We will prioritize Recall to maximize the proportion of actual churners that our model correctly identifies, thereby minimizing costly false negatives... Precision: while less critical than Recall for this problem, we still need to be reasonably precise to avoid wasting too many resources on unnecessary interventions" (Playbook, ch. 9, p. 246).
I don't have a churn model, but I ran into a similar trade-off building something else entirely, an app that generates lessons. OK, in my product, for example, the micro learning app, I had this issue, but with AI. Generating a lesson is quite important, and I need to, you know, I cannot spend so much money on that, I don't have that much. But at the same time I cannot keep the tokens too low either, because it's related to context. So what happened to me, while I was checking the quality of the generated lessons, I realized that if I keep the context too short, the output is bad.
So what I did is, I made my agent understand the context and try to calculate the token range first, and then it starts creating the lessons. But this brought additional problems. What are those? If my agent says, okay, this lesson will take like 8,000 to 18,000 tokens, the agent will try to consume all of the tokens, all the time. So that's why I tried to optimize the process. And there's another one: if it takes too long, I let another agent review it, an agent as a reviewer, and when it checks, and when it says, alright, this is not a good way to do that, because you're mixing all the stuff and trying to bring in all the data, no, the lesson should be short. So that's the most important part, balancing between agents is quite important.
Overfitting
A model with a small k "is more prone to overfitting: it's overly sensitive to individual training examples" (Playbook, ch. 2, p. 33), and the implication for a PM is blunt: "High accuracy on training data, low accuracy on testing data... Your product will perform poorly in the real world, even if initial tests look promising" (Playbook, ch. 2, p. 32).
The book is direct about where the PM's job sits in this: "As a PM, you don't need to implement these techniques, but you should: Understand the Concepts... Ask the Right Questions: Inquire about how your data science team is addressing these issues" (Playbook, ch. 2, p. 33). That puts the PM on the noticing side: the one who catches when a great demo doesn't mean a great product, while the data science team handles the fix.
A million times, especially while I was working with games, you know, those products are quite fuzzy, you don't know if the user will like it or not, and the majority of my products, after five minutes, my players were becoming free users. They were free to do anything. So basically, you cannot control all the details. Instead, you put some milestones, quests, they try to pass those quests, they try to go there with your direction, but sometimes they can fail. Yes.
And the real world example, especially while you're building an application: the most important part for me is to make your product's five-minute experience the same for each user, then improve that, then make them free, and then follow their behavior, because it's easy, right, an application, it's not like a game. You can track with a heatmap, you can track with touches, you can track with screen time, or prompts, the number of prompts, prompt quality. This is the most important part. Check it.
Model Drift
"Launching the model isn't the end; it's the beginning of its life in production... think of it as conducting regular health checks," and detecting drift means "identifying when the model's performance degrades over time due to changes in real-world data patterns" (Playbook, ch. 10, p. 268-269). The book puts a number on why this is a PM's problem and not just an engineering alert: it recommends defining "the acceptable performance thresholds" yourself and setting concrete triggers, "Alert if Recall drops below 80% for 24 hours" is its own example (Playbook, ch. 10, p. 269).
I don't have experience in AI with this, because I always review and check the quality of the outputs, but in games, yes, a million times. I designed a feature, I asked developers to put it into the game, and then all of a sudden, retention goals dropped, playtime goals dropped. Those type of metrics are quite important, and if you're in charge of a product, you should take care of all of it and notice right away. You should check daily.
How Many of These Do You Need?
I don't think every PM needs to understand these five concepts, to be honest. Because the error is, I mean, if you don't have any information or knowledge about something, if you ask the right questions to your researcher agent, you'll understand that thing in a day. So don't get afraid of trying something and making mistakes. Make mistakes. Ask your agent, what was the mistake, where did I go wrong, what did I do, what should I learn, and research about it. Understand the concept, continue with it.
We have a limited memory in our brain, guys. If you try to keep all the literacy, all the terms, all the glossary of terms, your brain will blow up. You're not Lucy, the movie character, limitless perception, you can understand what's going on around you, but I don't know, maybe one percent of it will stay in your memory. So it's quite important to use your memory, your brain, efficiently.
If you're working in a huge team, those roles will be there, and they will share their insights with you. The most important part is, when a data engineer or a data scientist brings insights to you, I'm not saying you should cross-check their solutions, but please talk with them, find out if they know the product. I mean, if there's someone who was a data scientist on Spotify's recommendation algorithm, and you work with that person on a different product, like a vector database built to find similarities, of course a recommendation system has the same underlying thing, but in a different way. Let's say we're building an app similar to Tinder, that person would have to change their mind. That's the most important part, right? They need to understand how they generate insights, they can't just bring their legacy over to a new product. You should understand that.
The next layer past these five is its own conversation for another post: context windows, temperature, RAG, the LLM-specific vocabulary. A stricter one still, building an actual eval suite around them, is a different post again.
Nobody Knows Your Case
I mean, I have millions of things to say, honestly, but while I didn't have time to have different sessions with my agents, with AI, I got frustrated. Let's be honest, I was thinking, I'm missing the train, I need to jump in there. So I created my own job. I started developing apps. I used my PM skills to develop apps. I kept thinking, okay, if this happened to my team, what would I say, so I think about it that way. (The fuller version of that jump is in post 2.)
I give feedback all the time to my agents, and that makes me understand the AI's chain of thought, and also, you know, how I should work, how I should write my prompts, etc. The case is yours, no one knows your case. And therefore, if you want to stick to generic solutions or generic learning sessions, yeah, of course you should know the keywords for searching or for writing prompts, that's more important than the rest. But let's think about it this way: I have a specific issue about changing one element on a UI, because if I change that, everything will fall apart. So if I write, okay, remove this button, to my agent, when it removes it, the agent has to say, alright, I don't know if you want to remove this, it will affect these pages, are you sure? This reasoning, guys, your developers were doing the same thing, so the agent has to do the same thing. That's why you write a CLAUDE.md or something like that, you should be careful about that, you should have a reviewer check.
So yeah, just be yourself, and work, and develop, and understand the AI. And AI will understand you also. That's all.
