- Open Source CEO by Bill Kerr
- Posts
- The Invisible Layer Separating Good AI From Great AI
The Invisible Layer Separating Good AI From Great AI
An interview with Tyler Phillips, AI Product Lead at Apollo.io. šš»
š Howdy to the 5,511 new legends who joined since our last edition! You are now part of a 306,215 strong tribe outperforming the competition together.
LATEST POSTS š
If youāre new, not yet a subscriber, or just plain missed it, here are some of our recent editions.
š“ Fundraising Fatigue, Hypergrowth & Selling Glovo. An interview with Oscar Pierre, Co-Founder & CEO at Glovo.
ā¤ļø How Athyna Makes Remote, Work. Tech, meetings, culture & comms. How to do it and maintain 90% engagement.
šš»āāļø Eight Very Cool Marketing Things. Insights I've found from around the web in the last weeks.

PARTNERS š«
Cash flow is the lifeblood of your business, but for many startups, it's also one of the hardest things to manage.
BILL's free Startup Cashflow Toolkit is full of tips, tools, and a few secrets for:
Stress testing your cash flow before problems hit
Improving spend visibility across your business
Reducing errors and costly mistakes
7 chapters. All free. Download it and keep your startup scaling.
Sell like a team of 10, as a team of 1. You don't have someone to keep your CRM clean, build reports, or tell you why deals are stalling. Lightfield gives you all three.
Every email, call, and meeting is captured automatically. Follow-ups surface before you forget. And when you need work doneā"build a pipeline report" or "draft follow-ups for every deal that went quiet"āit delivers. *3-month code: OPENSOURCE23
Interested in sponsoring these emails? See our partnership options here.

HOUSEKEEPING šØ
The internet is unbeatable. This meme, for example, is just too good not to share. And yes, itās political, and no, I donāt like Trump, but the point here is that someone found this cultural reference, right as we are neck deep in Dune lore, of an evil green pedophile, dragging everyone into war. You donāt need Trump Denangement Syndrome to appreciate the wonders of the Internet in this example.
As for everything else lately, my life could be summarized like so: work, train, Claude Cowork, train, workout, eat san choy bao, Project Hail Mary, Claude Worok, work, train, Claude Cowork. Going to break up the monotony by heading to a stand-up show tonight, and if we donāt get back, I might play around with some Claude Cowork before bed. Enjoy todayās piece!

INTERVIEW šļø
Tyler Phillips, AI Product Lead at Apollo.io
Tyler Phillips is a seasoned product leader, currently the Director of Product & Head of AI at Apollo.io, a fast-growing AI-powered go-to-market and sales engagement platform. In his role, he leads development of Apolloās AI Sales Assistant and broader AI platform initiatives designed to help businesses of all sizes discover, engage, and convert prospects more effectively. His work centers on building scalable AI solutions: he led Apollo AI product strategy and a 40+ person R&D organization, built the AI Assistant from zero to 21k WAU, and launched a research agent that reached 47M enrichments/month, driving $XM in add-on ARR, and grew Apollo AI's combined product suite to 50k WAU in 2024.
Before focusing on AI at Apollo, Tyler built a broad product career spanning B2B technology and growth roles, where he emphasized problem-solving, product quality, and user-centered design. At Apollo, he has been featured in industry conversations about what it really takes to deliver high-quality AI products in the go-to-market space, highlighting the importance of systematic evaluations and domain expertise in shaping successful AI outcomes.
What problem are you trying to solve with Apollo, and why?
Apollo is a go-to-market platform for SMBs and enterprises. The main problem weāre trying to solve is helping you find the right person at the right company at the right time, reach out to them, and book meetings. It sounds like an easy problem to solve, but itās actually very difficult. The second problem we focus on is helping you convert inbound traffic from your website into customers. And third, once youāve booked a meeting, how do you prepare for it? How do you execute the call effectively and eventually close the deal? So, weāre really thinking end-to-end across the entire sales funnel.
As for why this problem, if you can make someone $10, you can charge them $1. Itās a pretty self-evident problem to solve. I also find it fascinating because itās constantly changing. What worked last year wonāt work this year. You have to continually adapt your go-to-market strategy to find the edge and generate outsized returns. Thatās what makes it interesting to me.
Whatās hardest when going from early AI prototypes to a tool customers love?
The hardest part is getting to something thatās truly usable. Itās much easier to ship an initial version that people try once and then drop. The real challenge is building something reliable. For example, when we launched our Apollo AI assistant in Closed beta last June, the success rate was around 26%. That meant roughly three out of four people who used it didnāt get the outcome they wanted or didnāt have the right action taken. Itās very easy to stand up an initial system using base-level LLMs and some basic agent implementation. But getting to 80ā90% quality, where the conversation or agent consistently takes the correct action, is extremely difficult.

Source: Apollo.io.
The reason why itās so hard is that you donāt actually know what users are going to ask. You can guess, and thatās part of product management, but there are always edge cases you didnāt predict. You need to build a much more sophisticated system and fine-tune it to the userās needs. For vertical applications in particular, depth matters. You have to cover as many edge cases as possible, and that just takes time. Weāve worked on it for over a year, and itās still challenging to reach even a 90% success rate. But thatās where AI products are won or lost. Quality is everything.
What does your day-to-day look like leading AI?
My day-to-day is generally split across three areas. First is strategy. Iām constantly looking at our objectives, things like AI credit consumption or driving AI-assisted actions on the platform, and asking what bets we should be making to achieve our goals and help users succeed. That includes deciding whether to invest on the UX side, the quality side, or elsewhere, and then aligning the team around the right priorities.
Second is execution. Iām both an individual contributor, product manager, and a product leader, so Iām still very hands-on. I write PRDs, run evaluations, and review outputs. I spend a lot of time every week in spreadsheets going through AI outputs and running evals. Third is staying close to the customer. I meet with customers, review chat logs, and try to deeply understand what they need and how weāre positioning the product to serve them. I also spend some time hiring. Thatās generally how I divide my time.
Where is AI most effective in augmenting sales workflows, and where does it fall short?
I think a year ago, many people assumed AI would take over messaging. Now almost every seller uses AI for messaging, but itās proving to be less effective than expected because itās easy to tell when something is AI-generated. With tools like Gmailās AI previews, itās almost like AI reading AI, and AI is pretty good at detecting when the other side is also AI. It becomes a bit of a feedback loop.
Where AI has been most effective is in targeting. Itās very strong at researching accounts and contacts, at least for a first pass, by combining data from your CRM, your product, and the web.
For example, we worked with a translation company that used to manually visit websites to check for translation gaps between languages like Portuguese and English. With AI, they can now identify those gaps instantly. That makes qualification and research much faster and more scalable. | ![]() Source: Apollo.io. |
The second area thatās starting to emerge is analyzing go-to-market data to improve your understanding of your ideal customer profile. Itās about using the feedback loopāwho you message, who responds, who books meetingsāand feeding that information back into either a machine learning model or an LLM. With more data available, you can organize it and use it to refine your targeting and messaging strategy. Thatās the next big vector weāre seeing.
Will AI messaging eventually be indistinguishable from human?
I think itās conceivable. I donāt know if weāre that far away, but if youāre just using a one-shot prompt in something like ChatGPT or Claude, itāll be obvious. As systems become more advanced, they can truly understand your tone and voice, and capture rep feedback as they manually edit emails; I think theyāll get much better. Once you start building a system that learns from edits and adapts over time, the outputs can improve significantly.
The main blocker right now is having really strong context and tight alignment with your tone and voice. If those pieces are in place, I do think itās possible. Some companies are already doing a good job with this. But for now, itās still a challenge.


How do you decide whatās automated with AI vs. human-driven?
Iām a big believer in automating transactional, lower-level tasks. AI is well-suited for things like creating workflows to automate manual processes that users have already seen success with, building lookalike campaigns based on whatās worked before, or handling research and account updates. These are more menial tasks where a human isnāt uniquely positioned to add value.

Source: Apollo.
Where humans are still essential is in the actual human component: real conversations and understanding nuance on the other side. AI isnāt great at catching subtle signals in live interactions. Messaging, cold calling, and real-time communication still benefit from human judgment and review. I also think creativity should remain human-driven. Go-to-market today is about standing out. The best marketing ideas donāt typically come from an LLM. Youāre not going to ask one for breakthrough ideas and expect something truly original. Creative thinking is still in high demand, and thatās where humans have the edge.
How are you personally using AI in your own workflow?
I use different LLMs for different use cases. I use Perplexity for web research and searching; I donāt really use Google anymore. For deeper work, I use Claude as more of a coworker, especially its Projects feature. I use it for writing PRDs, shaping strategies, drafting OKRs, and things that require structured thinking. I find it particularly strong in writing.
Iāve started experimenting with tools like Cursor and Claude Code. I use Claude Code to go from zero to PRD and Prototype, as well as automate mundane tasks like weekly updates. I use Cursor to connect to Apolloās codebase and ship very basic PRs that my engineers review. I use OpenAI more for personal use cases, like cooking. At work, I use Glean to extract and summarize information from JIRA tickets, Notion, Slack, and other systems. I also use Slack AI to some extent. Overall, I tend to assign one core use case to each model based on what it does best, and thatās how I structure my workflow.
How do you think about leadership and culture?
A few things matter to me. First, create a culture of ownership. Itās easier said than done, but we set aggressive goals as a company and then assign one person to fully own each goal. That person operates almost like a GM for their area. Theyāre responsible for driving it forward, succeeding if possible, and if not, learning why we failed and how to improve. When people feel like owners instead of just executors, they tend to do exceptional work.
Second is incentives. I believe in a performance-based culture. If you perform well, you should be rewarded. Weāre not a family, weāre more like a sports team, similar to the Netflix philosophy. Of course, we treat people well, but performance matters, and incentives should reflect that.
Finally, itās about genuinely loving the customer. Itās easy for you to say youāre customer-first, but it has to be lived. Instead of just reading about how to build a campaign, go into Apollo and learn how to build one yourself. Instead of just shipping a feature, work directly with customers to get them using it and gather feedback. | ![]() |
That mindset should apply to everyone. When people are motivated by the same goals and rewarded accordingly, it creates a culture where theyāre excited about what theyāre building.
Is there any question about building product that you wish more people would ask?
One question I always like to ask is: how do you actually run evaluations? People focus too much on the product itself, but the real key to success lies in how you evaluate and improve it. The process you use to run evals and continuously raise quality is what ultimately determines whether you build a great AI product. In many ways, quality is the product.
Another important question is: how does context work in your product today? Context is a major area that has been underserved so far. We made a big bet last year to build in more context, and I still donāt think weāve done enough. The companies thinking furthest ahead about context are going to be the ones that win.
Can you explain the relevance of context in more detail?
We think about context from a few different angles. First is business context. That comes from your source of truth: your CRM, website, internal documents, and other systems that are constantly updated. That gives the AI an understanding of your company and whatās happening inside it. Second is user context, which is more like memory. If youāre interacting with an assistant, how does it learn who you are, what youāre trying to accomplish, and how you use the product based on your behavior?
Third is context from activity outside of Apollo. For example, we have a browser extension that lets us understand how users interact with prospects across email and other networks. That helps us see what theyāre actually trying to achieve, not just what theyāre doing inside Apollo. Fourth is skills-based context. Claude has a concept of skills, specific actions an LLM can take. In our case, we have domain expertise in go-to-market on our team, so we want to translate that into skills our AI can use. That means embedding go-to-market expertise directly into the system.
Finally, thereās what we think of as āa context graphā. This goes beyond just capturing activity. It looks at what led to a successful outcome, like a booked meeting. Which companies did you target? Which prospects? What emails did you send? What replies did you get? From there, you can build a decision tree that guides the AI toward sequences that have historically worked and suggests improvements, like refining the target list or adjusting messaging. Bringing all of that together is our goal for building a strong context layer at Apollo. Weāre well-positioned to do it because we already have access to this data through our product suite. Now itās about executing on it.
How do you get the best out of yourself personally and professionally?
I began writing weekly reviews while running my own company before Apollo. Every week, I look at the different areas of my lifeārelationships, career, finances, health, fitnessāand revisit what I want to achieve this year and how Iām progressing toward each goal. I think having a balanced life makes you a better professional. |
If you only focus on work, you miss the bigger picture. Paying attention to whether youāre healthy, eating well, and feeling connected to a community matters. Being well-rounded helps you show up better at work. I also like to take on challenges outside of work. For example, right now Iām training for the Murph challenge: 100 pull-ups, 200 push-ups, 300 squats, with a mile run before and after. Doing difficult things outside of work builds discipline and resilience. That carries over into how you operate professionally.
Extra reading / learning
The Secrets To Being A Great AI PM: Evals, Quality & More - March, 2025
What To Expect From Apollo Next with Tyler Phillips - February 2025
The Complicated Reality of AI Implementation in Engineering Teams - June, 2025

BRAIN FOOD š§

TOOLS WE RECOMMEND š ļø
Every week, we highlight tools we like and those we actually use inside our business and give them an honest review. Today, we are highlighting Metal*āa smarter way to search for investors, match by fit, and skip the endless cold-outreach guesswork.
See the full set of tools we use inside of Athyna & Open Source CEO here.

HOW I CAN HELP š„³
P.S. Want to work together?
Hiring global talent: If youāre hiring tech, business or ops talent and want to do it 80% less, check out my startup, Athyna. š
See my tech stack: Find our suite of tools & resources for both this newsletter and Athyna here. š§°
Reach an audience of tech leaders: Advertise with us if you want to get in front of founders, investors and leaders in tech. š
![]() |

















Reply