- Open Source CEO by Bill Kerr
- Posts
- Why The Next Interface Can Hear & See You
Why The Next Interface Can Hear & See You
An interview with Alex Smola, Co-Founder & CEO at Boson AI. 📞
👋 Howdy to the 1,626 new legends who joined since our last edition! You are now part of a 575,962-strong tribe outperforming the competition together.
LATEST POSTS 📚
If you’re new, not yet a subscriber, or just plain missed it, here are some of our recent editions.
✨ My Favorite Startmate W26 Companies. A look back at a handful of the best and brightest from the latest accelerator cohort.
⚡️ Zapier’s AI Offer: Valuemaxxing. How the smartest AI companies are spending less on AI.
🍀 Happenstance: 10x Your Luck. An interview with Alex Teichman, Founder & CEO at Happenstance.

PARTNERS 💫
Framer is the AI website builder for teams that care about every detail. Agents work directly in the canvas, helping design pages, manage CMS content, write copy, add SEO, and audit sites.
Build faster: Create, refine, and publish production-ready pages in minutes.
Scale confidently: Manage content, localization, and metadata across your site.
Stay in control: Review every change, collaborate in real time, and publish when ready.
Trusted by teams at Miro and Perplexity.

Everyone’s talking about getting back to the office. Candidates are still searching for remote.
Ashby’s data from 1,200+ startups show that remote roles draw 42% more applications and 9% higher offer acceptance rates, even as remote jobs slipped from ~80% to ~60%. For more insights, read our free State of Startup Hiring report.
Interested in sponsoring these emails? See our partnership options here.

HOUSEKEEPING 📨
When this photo was taken on Tuesday, I was attempting to snatch a 10-year best of 104kg (snatched 100kg before it), then Wednesday I found out I have a congenital heart defect that may mean I can’t ever train again, due to the risk of sudden cardiac death. The good news is I’ve lived to 40 with this genetic abnormality, training more than 95% of people, so whatever my biology has done to compensate so far seems to be working.

I missed it (dammit).
But cardio and heavy training are off the cards until I find out more, and it looks like they will be on ice unless I have open-heart surgery. I plan to write more about this and the startups that helped me come to this diagnosis in the coming weeks. Proactive healthcare is awesome, and I’m super lucky to have found this quirk in my body now rather than finding out the hard way down the line. Cheers, and enjoy today’s piece!

INTERVIEW 🎙️
Alex Smola, Co-Founder & CEO at Boson AI
Alex Smola is the Co-Founder & CEO of Boson AI, a Santa Clara company building the audio and audiovisual foundation models that voice agents run on. He spent seven years at Amazon Web Services as VP and Distinguished Scientist before leaving in 2023 to start Boson with Mu Li, his co-founder & CTO. Their Higgs family now covers the whole speech stack. Higgs TTS 3 shipped in June 2026 with more than 100 languages, zero-shot voice cloning, and inline emotion control. Higgs Realtime, the company's first speech-to-speech model, landed in August. Boson AI has raised $70M from backers including the venture arm of Temasek and Chinese entrepreneur Su Hua, and Alex's pitch to enterprise buyers is that his models run at roughly a tenth of what the competition charges.

That background is unusual even by AI lab standards. Before Amazon, Alex was a full professor at Carnegie Mellon, a principal research scientist at Yahoo Research, a visiting scientist at Google, and spent eight years in Australia at ANU and NICTA. He has written more than 200 papers and co-authored Learning with Kernels, one of the foundational texts on support vector machines. What makes him worth reading now is everything that came after. He runs a company of around 40 people where he has personally hauled out the office trash and racked servers, gives model weights away on Hugging Face while taking venture money that expects a return, and is betting that the next interface is not text but a machine that can hear you, see you, and read the room. He is also refreshingly unwilling to pretend that is purely a good thing.
Where would a real-time audio-and-video agent actually get used?
Anywhere you would currently have a concierge, or anywhere you would expect a human to be talking to you. Hotel receptions, entrances, and very high-value phone calls. Say you want to buy insurance, or buy a car. Basically any situation where it really matters that you not just get the data across, but also some emotional understanding—some EQ.
That's where this really matters, because there's a lot of back-channeling. If I smile, you're more likely to smile as well. That's not easy to learn, and it doesn't come easily. Or if I frown, you may not pick it up from my voice, but I'm a little bit skeptical now. But that bidirectional side channel gives you clarity about whether the person you're explaining something to actually gets it. You're nodding slightly, which means you're probably following me. If this were in Japanese, you'd be saying ‘hai, hai’, ‘sou desu ne’, or ‘wakatta’. Do that in English, going 'yeah, yeah, you're right, yeah,' and the other person would probably stop talking. In English, it's weird. In Japanese, if you don't do it, it's weird.
There's so much more to a conversation than 'the table is square.’ You want to exchange so much more. I think that's where the exciting development of the next two to five years will be. And the thing is, we're the only ones who can do both modalities in and both modalities out. Other demos have built systems that can do video and audio in and audio out, but doing both directions in real time is hard.
Why is voice becoming the next major computing interface?
Because this is how humans communicate. In fact, I'd argue it's probably not just voice; it's voice and video. Humans communicate best when they can hear and see each other. If we want to bring machine communication to humans, we need to—no pun intended—literally speak their language. Now, if you want to be very precise, for instance, if you want to write code, I probably wouldn't want to do that via voice. Some people have the amazing ability to do it, but I'm much more precise with a keyboard. But if I want to discuss something in a slightly more relaxed manner, then voice is much better.

Source: Boson AI.
That also means that we need to add EQ to how machines interact with humans. We need to learn how humans feel. You'd think that's very easy. Psychologists have written hundreds of books about it, and most of it is relatively small sample-size statistics. Then you could think about using movies, but I wouldn’t recommend you emulate what goes on in a movie. Take any Hugh Grant romantic comedy. If you actually did that in real life, you’d get arrested for stalking. So it's not a good strategy. We're right at the beginning of a very exciting journey there.
What problem is Boson AI actually solving with Higgs?
The problem we're solving is to really address voice conversations and voice agents. In many cases, you can instruct those systems just like a standard language agent. If you write the right prompt for Claude or ChatGPT, you can use the same prompt for our voice agents and, later, our audiovisual agents. You tell them what their personality and their character are, and how they're supposed to interact. Then you basically let those agents be themselves.
You can do things like switch between languages, and the instructions still carry through. It's similar to the instructions you'd give a human. You tell somebody, 'Hey, can you please start a conversation with this person? |
Here are the key talking points you want to get across.' You can instruct agents in the same way, and you get the reasoning and all of that too.
What do most people misunderstand about building voice AI?
One thing is that you need really low latency for things to feel natural. If I stopped mid-sentence, it would feel very strange, but that is often what models do. If I ask you a complicated question, you wouldn't necessarily just freeze, stare at me blankly for the next five seconds, and then spool the answer back. You'd think about it, and you'd indicate to me that you're thinking. You might go, 'Hmm, let me think, let me look it up.' You do all those things to indicate that you're in the process of generating an answer, so the entire human element is really important there. Latency and emotional competence matter both in terms of what you convey when you talk and in terms of understanding how humans feel when they speak to you.

Source: Boson AI.
You also want to address how humans interrupt and interact. As mentioned before, when you speak to a Japanese person, they may say, 'Yeah, yeah, you're right.' That's not an interruption. That signals they agree with you and that you should keep talking. Whereas on the other hand, if I tell you, 'Well, actually, you misspelled the third digit in my phone number, and it's a nine instead,' then you'd really want the model to stop what it's doing and say, 'Okay, yes, I got it, here's the new phone number.' That interaction and interruption need to be handled very differently.
Then there are also things like proactivity, and we're seeing some of the frontier models move in that direction by now. We did some research, published benchmarks, and so on, showing that's the case. But there's basically a lot more to communication than text in, text out. I'm not sure whether people are missing it, but it's just a really new area of research and development, and of amazing products that are going to happen.
Is that what makes real-time voice models so hard to ship, or is there more?
Some more things make those models really hard to ship. It's not that hard to ship a really expensive model. If you use a trillion tokens and you have unlimited compute, you can generate audio-in, audio-out models that are super intelligent. But what that means is that those models are probably going to use a lot of their intelligence to listen to you saying, 'Hmm, yeah, probably, maybe.' They're going to waste a lot of their compute. That's not just a shame, because you should never write inefficient models and inefficient code, but it also makes it really expensive. In the end, the customer pays for that compute.

Source: Boson AI.
What makes us different is that we actually spent quite some time engineering the models to be cost-effective. You have a reasonably inexpensive model that handles the front end of the conversation, and then for the more complicated things you use a more expensive model, with tool calls and delegations that get used whenever needed. Humans do this all the time. You may be in a coffee shop having a chit-chat with somebody, and the conversation is a little bit boring. In the back of your mind, you're thinking about what you're going to do afterwards, like what dinner you're going to cook. Your brain is also doing other useful things while the front end is carrying the conversation. On the other hand, if your conversation partner asks you a really complex question, or something really engaging, then you're fully immersed; you spend all of your brain power on it, and then maybe you relax a little bit again. Humans are pretty good at that, because the brain is expensive. It eats a lot of energy.
In the same way, you need to build conversational systems that do the same thing. For audiovisual, it's even harder. Streaming an avatar cost-effectively is not easy. Take any frontier video generation model and look at how long they take and how much they struggle with generating things in real time. It's not that they're doing something stupid. They're just trying to do an amazing job for anything that could happen. The solution is to think about how you can optimize it for the purpose at hand. That's where the hard work also went.
How is quality measured in a voice model?
There are a number of things you can do, and it's really many different aspects. You can measure latency (time to first response), intelligence, the ability to do tool calling, whether the voice sounds right or is emotionally expressive, whether it sounds like your voice, or someone else's voice, for any specific voice you want to clone. All of those parameters matter. So there isn't usually just a single number where you can say we're better because that one metric is better. It's really the whole package that you then need to optimize.

Source: Boson AI Blog.
And you also want to optimize for the cost-effectiveness of the model. You want excellent performance at a very good price. Then you have a lot of other side quests. Can I switch languages easily? I could tell the model, 'Sprechen wir Deutsch,' and I'd like it to switch to German. Then if I switch back to Spanish, I want it to answer me in Spanish. Humans take that for granted, but getting the model to actually do it really well is hard, but we're getting there.
How has being a CEO changed you since your AWS days?
The biggest difference is that if it doesn't work now, it's my fault. That means that initially, when we didn't have an office manager and trash was piling up, you grabbed a trash bag and took out the trash. Or you start putting servers into server racks because you don't have a proper SRE team yet that will do this for you. You're effectively always the person of last resort. If something doesn't work, it's your job, and it's very quickly your job to find somebody better than you at solving it.

Source: Boson AI Blog.
At the same time, it's a big responsibility, because if you screw up, you screw up not just for yourself but also for a lot of other people. It's a challenge. It's probably the most diverse job I've ever had. It's something else and something new every time. I may be in Toronto interviewing people for the site reliability team, and then you go back and work on a pitch deck, which is a completely different thing. And then the next thing is reviewing a paper we're currently writing. You have to try not to be terrible at any one of those things. But that's also what makes it exciting.


How do you run a company that's mostly researchers?
It’s exciting, and sometimes it feels a little like Christmas every day, because your team members surprise you with really delightful things. Someone has an idea, you tell them to build it, and it lands much faster than you had hoped. Other times it takes significant work to convince everybody to work on a given problem.

We started out doing a large amount of research, but now we have a product we're very proud of. Now comes the equally exciting, slightly different task of getting that product into our customers' hands. This is why we hired a CPO, and Lindsey has been quite instrumental in helping us with that. This is just what happens as the company matures. You start working with customers, selling to them, and making sure you deliver all while not losing your soul.
Where is enterprise voice AI heading?
I would say we're going to see quite a pervasive rollout into call centers. I think this will dramatically improve customer satisfaction. The only data point I need to give is: when was the last time you enjoyed calling your cable company? That was probably never. You get handed over from one person to the next, and to the next. And you have to plead your case to each of them, just for the next person to ask again for your name and what the actual problem is. So you get a lot of that context loss and context drop.
Being able to pull all this information up, see that a customer's connection has been flaky for the past few months and they've already called three times, and send out a technician, that's stuff a model can do in the background. It can fire off tool calls so that by the time you call and say, 'Hey, my modem's not working,' the model should already be able to say, 'Yeah, look, we've tried three times debugging; it's not working. When should I send out a technician?' Rather than you spending half an hour being very frustrated about it.
![]() Source: Boson AI. | ![]() Source: Boson AI. |
Those calls aren’t cheap for enterprises either, because they need to pay for people to provide the service. Call center instructions are often very narrow and restricted. So I think there's a lot of room for improvement in the customer service and support area. In administration, I’d say there's a lot of room for improvement too. For example, in Germany, to get dual citizenship, you need to write a letter to the Department of Administration. They'll then respond within 12 to 15 months and tell you whether they deem your application appropriate. Nobody's telling me that this is a situation where you shouldn't automate to save money. The people processing this are clearly overwhelmed. This is something where you could very easily provide better customer service with a fair amount of automation.
Furthermore, and I know this doesn't sound exactly like enterprise, I think we’ll see a move of a lot of those agents into the real world of companionship, entertainment, and so on. That sounds awfully dystopian, but think about an old age care home. If the relatives have time, they will spend a lot of time with them. But in many cases those people just sit there, and the only entertainment they get is from a TV, and maybe the nurse comes in fifteen to thirty minutes a day. That's a very sad existence. I'm not saying that an agentic solution is the perfect substitute for humans, but it's much better than having nothing.
What excites you the most about audio models right now?
What's really exciting is that they're moving from science fiction to reality. We were still quite far from that a couple of years ago. By now, this is actually starting to turn into something that you can talk to. Around three months ago, when we first started putting our system together where you can actually converse naturally and it responds quickly and all of that, that felt really great. Then we gave the model a face, and it felt even better. With the video now, it will still take a little bit of time until this is out as a product.

And at the same time, when I showed it to my daughter, who's 17 and wants to become an engineer, she was like, 'Yeah, isn't the system supposed to do that?' And I was like, 'No, this is the first thing ever.' She's just taking it for granted. To some extent, that also means that people may be quite willing to accept that this is what the future will hold. It actually makes you appreciate a lot of Hollywood movies from the past twenty to thirty years quite differently. They were much more on point than one would have thought. That's why this is a really exciting time. I think the next one or two years will be absolutely gorgeous.
Why does the open-versus-closed debate not matter to you?
I think we do a fair amount of both. We definitely benefit from open-weight models. We also contribute back to Hugging Face, where our models had 950,000 downloads in the past month. My co-founder and I have released other open-source models in the past. AutoGluon is from us. We released an entire textbook to teach deep learning. I've always been a big fan of contributing back to the community.
At the same time, you need to find a way to make it economically viable, because ultimately our investors expect us to show some return on their investment. It's a slightly delicate tightrope. You want to do the right thing by the community, because we benefited from them as well, so you need to contribute back. But you also want to do the right thing to the people who trusted us and made it possible to build these things. I think we've done quite well so far, but we need to live up to it every day.
How do you use AI to build AI products?
I use it a lot, but it goes in different ways. In the simplest case, if you need to decide which servers to buy, you describe the specs you want and ask it to find the right prices and so on. You can also use it to learn very quickly about a new area of research. You find out a certain area is becoming more popular, and you want to learn about it. A very common thing I do in this case is, 'Hey, I found out about this area. Here are some ideas I have. Find out who else has already done this, find me some other relevant work, provide me details.' And I get a 20-25-page dossier. I read that, and afterward I can go and read maybe two, three, four papers in depth. I'm able to learn a lot more quickly than I could otherwise.
That's more on the conceptual side. Then you can use the systems directly to derive math for you. In some cases it's about working on a proof technique. This can lead to very surprising results. It's definitely an incredible accelerator relative to what I could do by myself. And then in the end you can obviously say, 'Hey, now make my day, go and implement it.' If you think about having to deal with different GPU drivers from NVIDIA, with different CUDA versions, with different versions of PyTorch, with different versions of vLLM and SGLang and all of that, this is a task I'm incredibly glad to hand off.
Then you can also be a lot more ambitious and ask which experiments I should try out on something. And in some cases you can even automate this, so there's really a broad range of things. By now those systems have become sufficiently reliable that you can actually do this and not assume that things will blow up terribly. It doesn't always work, but it works more often now than even half a year ago.
How do you hire researchers who can also ship?
This is challenging, but here's the thing we've settled on so far. First, you don’t want to hire 20 clones from the same blueprint. It's way too easy to think that if you hired one good person with some attributes, now I want twenty of them. If they're good all around, that's great, but you want to have diversity in skills. | ![]() |
The one absolutely essential thing is that people have their own initiative. That's probably way more important in a startup than in a big company, because you get a lot of responsibility, but things also aren't as nicely settled as they would be in a big company. So in many cases it's, 'Okay, here's the problem, here's the basic idea, go and make it happen, go and figure it out.' That requires a certain level of autonomy that you may not need in a big company, but you also may not get. So it attracts different types of people. Coding ability tends to matter, but I would argue that by now it's more about algorithmic thinking than the sheer number of lines of code you can churn out.

One thing I insist on is doing in-person on-sites because otherwise you can get people who are artificially intelligent and not as naturally brilliant. I've definitely had cases where people breezed through the online interviews, but when they showed up, we had to send them home after the second interview because it was not going to work. We then tell them we're terribly sorry, but we don't think this is the right match, because I don't want to torture people if it's clear that it's not going to work out. Tools like Cluely are a really good way to get that onsite, but it's effectively wasting everybody's time. Maybe there are some companies where you can use that to trick your way into a job, but I don't think it's very effective.
What did running ML at AWS teach you about products?
One thing that Amazon is really good at is its release cadence. It has one hard deadline, that's re:Invent, and everything happens for that. So you plan, put the team together, and ship. That's good, but it's also not always the right way to do things when a year can be an eternity in AI. If you look back a year ago, coding agents were nowhere near where they are now. I would say they got about good enough in February this year, maybe March. And if you look at my GitHub record, I have a lot more commits now than before, because I just didn't use agentic coding that much. So this is something where the annual cycles can be challenging.

Source: DataLemur.
Definitely, making sure you talk to the customers, making sure you think your products through, that's very helpful. Then there are Amazon’s leadership principles, and funnily enough, we actually carried quite a few of them over into Boson. I still send out hiring and interviewing pre-briefs, just like we did before. I've made several optimizations to make this a lot less costly. We do the pre-briefs by email rather than having a 45-minute call before and a 45-minute call afterwards. I'm definitely focusing on a subset of them more than others: deliver results, learn and be curious, dive deep, earn trust, and customer obsession, obviously. Frugality, I guess everybody is reasonably frugal, but you also need to think about resources invested versus time saved, work versus speed. So the dial is at a slightly different point there.
Disagree and commit is really important in large organizations. In a startup, if you have a total of forty people, there's the overall vision, but beyond that you're kind of the captain of your own project, and you're going to make it succeed. So you should probably agree and commit with yourself. For our tokenizer, there's one scientist, and she builds this. That's her job. A larger company would have an entire team do the same thing, and I think she's doing a great job. There are some cases where I’d say I wish Amazon had been a little bit faster. We definitely had good internal plans for where things were going, but then you need to execute. Well, that's why you do a startup.
How do you get the best out of yourself?
I don't know if I'm getting the best out of myself right now, but I need to keep trying every day. You get up every day, and you make sure you do a good job.
You have to ask yourself if there’s anything you can automate. Is this the best use of my time? Make sure you don't burn out, because this is a marathon. So you make sure you keep your health in order. Short-term heroics will be very expensive in the medium term. | ![]() |
Try to learn new things too, and sometimes do it yourself rather than relying only on secondhand knowledge. Stay curious, and complacency tends to stay away on its own. When you realize you have a really big knowledge hole in something, go tackle it. The rest of the team probably has the same hole, so do a lunch-and-learn, sit down with a batch of papers, and come out able to explain it properly. Then just keep at it. You’re not going to be perfect. You try to be better today than you were yesterday. Does it always work? No. But tomorrow is coming, and you work on it tomorrow.
Extra reading
EX-Amazon VP: We’re Making Digital Humans Real - September, 2026
Introducing Higgs VoiceOver: AI voice-over in your editor - September, 2026
How Boson labeled millions of hours of data for its speech-to-speech model - August, 2026

BRAIN FOOD 🧠

TOOLS WE RECOMMEND 🛠️
Every week, we highlight tools we like and those we actually use inside our business and give them an honest review. Today, we are highlighting Granola*—an AI-powered notepad that takes, summarizes, and organizes meeting notes without using intrusive recording bots.
See the full set of tools we use inside Athyna & Open Source CEO here.

HOW I CAN HELP 🥳
P.S. Want to work together?
Hiring global talent: If you’re hiring tech, business or ops talent and want to do it 80% less, check out my startup, Athyna. 🌏
See my tech stack: Find our suite of tools & resources for both this newsletter and Athyna here. 🧰
Reach an audience of tech leaders: Advertise with us if you want to get in front of founders, investors, and leaders in tech. 👀
![]() |













Reply