Steven Mastandrea, Vibes’ Vice President of Engineering, has had a front-row seat to the power of LLMs and in shaping them to carefully yet aggressively bring them into his product development and engineering process.
Here, Vibes VP of Marketing Jay Hinman talks with him about his foundational approach to AI, his learnings along the way, and, ultimately, how his team’s innovations are already impacting the Vibes products he helps deliver to market.
Jay Hinman
Steven, you’ve been a strong advocate internally for some forward-looking usage of AI here at Vibes, both internally and in the products our customers use. You're also leading our efforts to keep Vibes’ human element and human-centric service approach front and center with our customers. Can you say some more about how you see things?
Steven Mastandrea
I think recently we have seen some incredibly powerful, productive uses of LLMs in development. My tagline that I've been using with my team, as we've been utilizing it and applying it where it’s effective, is that “nothing's changed, and yet everything has changed”. What I mean by this is that AI as we’re using it is an incredibly powerful tool. But as the adage goes, if your only tool is a hammer, then all your problems start to look like nails. I still think the amount of hype and over-expectation on the future power of it is astronomical, and there are a couple of reasons for that.
As I talk with my engineering colleagues, we say, “Remember back as an engineer, the first time you wrote code, and the first time you automated something, and you saw the power of code, and you said, “Wow! Like, holy cow!” After that, you're like, “I could do this, I could do this, I could do this…”.
Everybody's experiencing that now, right?
Jay Hinman
Even those of us in marketing!
Steven Mastandrea
You're able to experience democratization at a level that you've never been able to experience it before, because you always had to talk to an engineer, and they've done certain things, but you've never been able to do just what you wanted at the power of your fingertips. With every previous technological improvement, the interface was still technical in nature. LLMs are the first ones that cross the interface boundary and it’s incredibly powerful. And, with great power, comes great responsibility.
Every experienced engineer has had or seen the proverbial $100,000 error; you know, what did you do, or what did your team do that cost the company? If you haven't, you probably haven't been in a production environment or a production system long enough. It's just a cautionary tale. That doesn't mean that we don't try and move faster, that we don't use the tools, but if you automate and accelerate a bad process, you can make mistakes a hell of a lot faster than before. It's that duality that comes into play.
Jay Hinman
So how does that relate to the “everything has changed”?
Steven Mastandrea
Well, the other thing that has made AI so powerful for both engineers and non-engineers across the ecosystem is that there have been tools and ideas and things that have come into play in the software development space for decades. We've had unit testing and automation, CI/CD, automated deploys, and all these things that have optimized our processes, and that we've used. And now with LLMs, ensuring code quality, operational resilience, functional delivery are still needed (nothing has changed) and in an LLM / Agentic workflow, all of the techniques are changing.
LLMs are the first thing that have democratized the interface. Everything else you had to learn. If you wanted to query a tool like Looker, a reporting tool, there's powerful data behind it; you could go get it. But you always had to learn the basics of the query language, or the basics of SQL, and that's just enough of an inertia that you're like, “I'm not that guy”. LLMs have put that truly in the hands of everyone, for the first time.
Now you can go in and say, "pull what I need” and it can get pretty damn close. Now that “aha” moment is much closer, and the power and the vision and the opportunity are so much more impactful than anything else.
The flip side of it is, with great power comes great responsibility. If you're querying data out of the warehouse, go for it, all day long. The worst that you can do is maybe write a bad query that slows it down. If you have update permissions, then there’s real risk. Then there are all kinds of additional cautions, so if you're like, “I want to update this data with x-y-z", now what's the worst you can do? It can potentially be much worse.
Jay Hinman
So, I can see how quality is important, but how big is the quality aspect really?
Steven Mastandrea
Well, I think we’ve already seen several examples in the industry on AI-induced errors or security lapses. And that isn’t to say that AI itself is the cause, but more the reliance on reviews and quality controls that may not have been adapted.
From just a basic math perspective, if you think about a system’s existing error or defect rate (how many times per year an issue occurs), and let’s assume that the current rate is at the minimally acceptable tolerance level. Then, to accelerate delivery throughput by, say, 4x, you’d necessarily need to improve your entire quality/defect detection by 4x as well, Just to stay where you’re currently at!
Jay Hinman
Because it’s reading only what you've put into it.
Steven Mastandrea
Exactly. I think here at Vibes there are 3 main areas where AI comes into play. One of them is employees doing their work; this is where the most extensive engineering use is today. We are using Claude to write code, help do analysis, create specs, troubleshoot issues, and help us do other things to create better products for our customers.
There's a second aspect, which is that we're using AI in our backend software and in our offerings to do things. We're using AI to help vet a customer's text messaging program for compliance. We're using AI to do data analysis on traffic patterns. The third one is bringing powerful AI all the way to the end user, where we're letting the end user ask queries.
Each of these has different powers, uses and complexities. For Vibes engineering to use it for our own purposes is entirely within Vibes' control as far as risk factors go. When we bring it in on the back end and we start using it to, let's say, do traffic-shaping analysis and other things, again, it's still mostly within our control, but now we need customer data, we need customer traffic patterns, customer boundaries, so it gets a little bit more complex. Bringing an LLM all the way to the end user is, in my mind, the hardest complexity to get fully right.
The interesting thing about LLMs, is that they, by definition, always give you an “average” answer based on how they've been trained. So, if I'm using it for my engineers to write code, an average worldwide answer in written code is great, right? I think the further you get out to customer-facing, the more we're going to move into small language models: we're going to move in things that are trained specifically for us. Because if we're looking to evaluate a provisioning request, and we're vetting these requests, we're never going to get better than anybody else can get if we're using the same LLMs everyone else has. We are now working with this in our messaging apps to really help our customers.
Jay Hinman
You're going to have to decide where a human gets into the loop, and where a human is not in the loop. 5, 6 years ago, a human was in the loop for nearly everything that we did, and no one was using Claude code, because it didn't exist, but everything we did was human-centric. What does that term even mean now? Where do humans play into this?
Steven Mastandrea
One area is the data you're training the LLM on. A model that's trained on Vibes data, and Vibes terminology, and Vibes everything - a more closed and well-defined ecosystem, rather than all the information in the world. Humans absolutely play a part in shaping that.
The other part of it is, it's really contextual to what you're doing. So, one of the things that we use in the engineering realm is whenever you go to AI, the first thing you have to do ask is, “is it right?” Because it isn't always better. It's sometimes and occasionally better, and occasionally worse, but it’s always faster.
What other safeguards can you put in include or expand in addition to utilizing LLMs? We have automated testing. In marketing, you might have a style guide. We can actually write deterministic code that says, it has to admit this; it has to do that. We're experimenting with this, as we're building it into all of our processes. In our code reviews and our change reviews, the LLM's will do a first pass. It's going to enforce a bunch of rules for us, check a bunch of things, perform an initial level of risk and give that information to the human reviewer, if the risk level requires a human reviewer. This lets us manage by exception. The ideal is we want humans to spend the most time on the changes and additions that have the most risk, or that the LLMs might get wrong, while utilizing LLMs and Agents to take care of the more straightforward changes.
We – and our customers – are already beginning to reap the benefits of this accelerated but considered approach, which is realized both in the speed to market for new features and functionality, and even in how rapidly we can bring an RCS Studio or Connect customer through compliance review and provisioning. My team is excited about what we’re shaping here at Vibes and in using the democratization brought by LLMs to build messaging products that make a meaningful and positive difference in how businesses and consumers communicate.
Jay Hinman
Anything else to add?
Steven Mastandrea
Basically, to wrap it up – the same goals exist today as yesterday: understand and solve the real problems for our customers, help them achieve their goals and provide excellent value and service, and do it reliably, securely and efficiently. LLMs and Agentic development will require, enable and unlock entirely new tools, tactics and strategies to accomplish this.
