How AI is changing the way we build products: 5 lessons from Teresa Torres’ first AI product build
When Teresa Torres found herself sidelined by a severe ankle surgery, she didn't just catch up on her reading; she spent three months on her couch, "living on the edge" of AI development. The result was a live, production-grade AI Interview Coach.
In our latest webinar, the leading voice in product discovery and author of Continuous Discovery Habits shared her AI build story: what worked, what failed, and what she would do differently.
Her journey reveals that building with AI isn't just about adding a feature; it's about a fundamental shift in how we architect, synthesize, and define quality.
Here are our five favorite takeaways from the session.
1. Good enough is not enough
Many companies have been burned by the overpromise and underdelivery of AI, waiting for an ROI that they don’t see. Teresa’s answer to this is blunt: If AI is your differentiator, “good enough” is not enough.
While most teams fall into the trap of accepting “good enough” AI output, Teresa urges them to raise their quality standards: “When I look at the 80% output, this tells me there's promise here, but there's clearly more work to be done."
Quality assurance has become a new discovery practice in the age of AI. For Teresa, this translates into two different habits:
Evals: “We can put evals in place as guardrails to tell our models, ‘You cannot show this to a customer if it doesn't match this criteria.'”
Feedback loops: “As people move into agents, we need this reflective loop of ‘do something, evaluate it, do something, evaluate it.’ Can the agent manage those cycles itself so that you're improving the overall quality?”
“We need these two things to guarantee quality, even in non-deterministic products,” Teresa concludes.
High-quality input: Orchestration over complexity
Reaching this high-quality bar when building with AI requires actively improving your prompting skills beyond simple chat-based interactions.
A common mistake is to overload the LLM with information. "A lot of people want to give the LLM all the context they could possibly want. But that actually leads to context rot; it confuses the LLM. So how do we give it just the right context and nothing more?"
The key to building reliable AI products is recognizing that “LLMs perform better on simpler tasks and degrade on more complex tasks.”
Deconstructing tasks: Instead of writing “mega-prompts,” orchestrate a series of small tasks that build up to the big task.
Managing the context window: Use techniques like retrieval systems or small context files to ensure the model has "just enough information at the right time".
Teresa faced this challenge herself when building her AI product. “The Interview Coach started out as one prompt. We had four dimensions per rubric. The LLM would start to confuse the criteria of each dimension and so I had to move that into four different LLM calls.”
Think your product org is AI-forward? Take our 5-minute test and compare yourself to other product teams.
2. AI is an interview partner, not a replacement
While AI has become instrumental in product discovery, Teresa warns that it should be treated as a high-powered partner rather than a shortcut. Many teams are rushing to automate synthesis, but they are often falling into two critical traps.
Garbage in, garbage out
It all starts with the basics: “If you're not conducting a good interview, AI can't really save you in your synthesis.” But most of all, Teresa advises against ill-fitting tools. “When you use generic research tools, you’re missing a lot of the nuance and value from those conversations,” she argues. “In the future, each company will have to build out its own synthesis solution, or pick a vendor that uses their perspective.”
Lazy AI interview synthesis
“If you're going to take time out of your day to go and talk to a customer, do not outsource 100% of the synthesis to AI.”
Here’s how Teresa avoids this: “I do the synthesis myself. Then I ask Claude to do it. And then I compare what Claude found and what I found. Every single time I catch things Claude missed. Every single time Claude catches something I missed.”
AI shouldn’t be replacing humans in discovery interviews. It should be used “as another perspective to help get more out of a conversation.”
3. The product trio as we know it has collapsed
Teresa does see some truth when she reads, “Engineering is dead,” “Product management is dead,” or “The product trio is dead.” We all know that AI is reshaping the typical product team structure. But how?
“We’ll still need product expertise, design expertise, and engineering expertise. But smaller teams will be able to do more, and to me, that’s really empowering.”
In her podcast Just Now Possible, Teresa has met living examples of this. “I interview a lot of two-person founding teams, and they're building what used to feel like giant SaaS products,“ she says. “AI makes it super easy as long as you're good at building the right stuff. That's where the discovery piece becomes even more important.”
4. Product managers need to be builders, as well as thinkers
It’s not just the product team structure that’s being challenged; it’s also the essence of the product manager role. The rise of AI products surfaces a dilemma: Are product managers thinkers or builders?
Teresa’s take is that “all humans are builders. It’s part of our DNA. We like to make things.” She adds, “This technology is letting more of us tap into our maker mindset. I think that's a really good thing.”
Beyond the coding debate
While the industry has long debated whether product managers should learn to code, Teresa has "flip-flopped" on the issue throughout her career. While she was grateful that knowing how to code gave her a "language and a mental structure" to engage with engineers, she notes that the nature of building is shifting.
Developing a builder toolkit
“Everybody needs to invest in their builder mindset," she argues. “We have to get more comfortable with these technical building blocks.” This is not about mastering a specific programming language.
Proficiency now means understanding:
LLMs: Grasping things like how transformers function.
APIs: Understanding how to use them and how to design them for different contexts.
Context management: Knowing how to provide "just enough information at the right time" and using retrieval systems or small context files.
Understanding the "plumbing"
To truly build with AI, PMs must be willing to "see the plumbing" and "live on that edge.” Teresa recommends using tools like Claude Code to get a hands-on look at how the technology actually behaves.
Teresa has a word of encouragement for those who find this intimidating: "Now that we live in a world where literally an LLM can walk you through it while you do it, all this stuff is very learnable."
Clarity as the ultimate build skill
Ultimately, the product manager role is moving toward a highly skilled intent architect. "I think product managers in particular have the best skill that is required for building with AI," Teresa explains, "and that's really just being able to write really good specs and really strong clarity on what you want".
5. A great tool, even AI-powered, won’t always create the desired outcome
Teresa’s first AI build was a humbling experience of dismantling her most basic assumptions.
With Product Talk’s AI-powered Interview Coach, she aimed to offer product teams an effective way to consistently and continuously improve their discovery habits. “I had really hoped that by being able to get as much detailed and personalized feedback as you could consume in a three-month period, that people would use this every single day.”
The reality? “About 30% of our students use it and use it a lot, and 70% never use it once.”
This realization carried two reminders for Teresa:
AI-powered or not, there is no guarantee that your product will change users’ behavior the way you wish, especially when the format isn’t adequate. “What people don't tell you about on-demand courses is that people buy them, but they don't complete them.”
“Learning is hard. It's uncomfortable. If our behavior isn't changing, we're not learning.” I think our workplaces have taught us not to do these things. We've learned we're not allowed to not know, we shouldn't be uncomfortable, we should have the answers, we should know how to do this.” But learning is still the only way to keep up and stay ahead in a world where AI doesn’t wait.
The future of discovery
Teresa compares the AI revolution to “the internet, if not bigger.” Still, the core challenge for product managers remains the same: building the right thing. The speed of AI makes discovery more critical than ever, because building the wrong thing can now happen faster than ever.
"This technology is going to change everything... if we want to continue to build things, we have to be proficient in this and really understand its strengths and weaknesses."
Francisca Berger Cabral
Read also
Experience the new way of doing product management
Experience the new way of doing product management