đ§đ The Hidden Layer of Every AI Agent | Tony Holdstock-Brown, Inngest
Using product tracing to build a low cost software factory, why you can't vibe code infrastructure, and building a dev tools company without a personal brand
Tony Holdstock-Brown is the co-founder and CEO of Inngest, the durable execution platform that quietly powers your favorite AI agents.
This was a long conversation on the hidden layer of AI infrastructure. Itâs the piece that secretly fails the most and increases LLM costs by 20x.
We get into why so many agents work in a demo but in production, how Inngest grew 35x after AWS and Cloudflare copied them, building their own cloud to get 20x lower cost structure, growing a dev tools company without a personal brand or Twitter account, why he thinks evals today are âbatshit insaneâ, and the thing they built to score 100% of your production agents without paying for LLM as a judge.
Support this Episodeâs Sponsors
Numeral: The end-to-end platform for sales tax and compliance.
Flex: Sign-up for Flex Elite with code TURNER, get $1,000 here.
Amplitude: AI analytics, all you have to do is ask.
Merge: Every model. One API. Total control. Check out Mergeâs Agent Handler.
Monaco: The revenue engine for startups.
Inquire about sponsoring future episodes here.
đ Stream on Spotify and Apple
Timestamps to jump in:
0:00 The hidden infra layer every AI agent runs on
1:46 Building complex chains of logic
3:31 Why agent SDK's don't go far enough
4:49 Healthcare was the original event-driven nightmare
6:32 Storing traces on your infrastructure enables self-improving loops
14:26 Why Inngest was already in the right place for AI
15:49 Score agents off product events, not LLM's
17:31 The OpenAI copy-paste signal
21:24 Swap in LLMs and cut costs 20x
23:44 How customers pulled the product forward
25:41 Orchestration belongs outside the sandbox
29:48 Building a neocloud to cut costs 20x
32:09 Most neoclouds just resell AWS
32:54 All AI infrastructure is converging
34:49 Why Claude can't just build your backend
36:44 How to build a software factory
39:12 Agents are a lottery you get addicted to
42:44 Loops must exist until AGI hits
45:38 If models keep getting better, why orchestrate?
48:28 When incumbents steal your features
52:30 Why you can't vibe code infrastructure
55:54 Why Tony has no personal brand
59:38 Dev tools GTM without Twitter
1:03:20 Lessons from the founder of DuckDuckGo
1:10:39 Truth as a company value
1:13:08 Taking too long adapting to AI
1:15:10 Startups are 100% R&D
1:17:19 Ali from Databricks
1:19:03 Writing his own code, Voice-to-text with local models
1:23:53 Evals are batshit insane
Referenced:
Find Tony on X / Twitter and LinkedIn
đ Stream on YouTube, Spotify, and Apple
Transcript
Find transcripts of all prior episodes here.
Turner Novak:
I think the best place to start. What is Inngest for people who donât know?
Tony Holdstock-Brown:
Yeah. So Inngest is a small SDK that gives you durable execution on any platform. The TL;DR is you build step functions, and they automatically retry. They will succeed every single time your functionâs run, and we abstract all of the infrastructure there.
So no queues, no state, no events. It just kinda works. So somebody who has never heard any of those words before... Yeah, yeah...
Did you just like, like
Turner Novak:
Why is that all important?
Tony Holdstock-Brown:
Yeah, totally. So the TL;DR of this is when youâre running something like agent, so youâre running some sort of process like, like order shipments, non-AI workloads...
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
You might be interfacing with API providers that fail. You might have systems that fail because, for example, your database is down. Mm. And, typically the way you get around this is building this crazy system of queues and events, and spending a lot of time, AKA months, on infrastructure.
When you use something like Inngest, you write some really basic code that says, âIn this step, we are going to call Anthropic, and this, in this other step, we are going to call a tool that Anthropicâs LLM wanted us to call. â And if Anthropic fails because of rate limits, we will automatically retry that step...
Turner Novak:
Mm.
Tony Holdstock-Brown:
Exactly where it is in the function without losing any of the previous context because we store all of its state, and you basically do zero work to get durability out of the box. And thatâs really cool because then any engineer who wants to create these step functions, whether thatâs an agent harness or if thatâs something like e-commerce orders or health insurance, you get all of this out of the box without spending any time on infrastructure.
Turner Novak:
Do people spend a lot of time on it if theyâre not... Yeah... Using
Tony Holdstock-Brown:
Inngest? Yeah.
Turner Novak:
Yeah, yeah,
Tony Holdstock-Brown:
Yeah. Okay. This is actually where Inngest comes from. I spent months rebuilding the same infrastructure on Kafka and then on queues, and it was like, just terrible.
I was gonna swear.
Turner Novak:
This, this was like in
Tony Holdstock-Brown:
2019-ish? Yeah You kind of... Yeah, it was a while back. Yeah, it was a long time ago.
It was a while back. And like, you know, queues still exist. Everyone uses... Classic example is SQS.
And, really, really complex to work with. Even harder with AI because youâre gonna be building really, really complex chains of logic for your agent harnesses, and if youâre not doing that, then when your agent harness fails, youâre essentially trying to retry in memory, and/or your entire agent is going to just fail, which, which sucks.
Turner Novak:
So how do people get around this today? Like, if Iâm not using something like Inngest and Iâm just like straight up, you know, shooting this all with my own code, building this all myself...
Tony Holdstock-Brown:
Yeah...
Turner Novak:
How do you do it?
Tony Holdstock-Brown:
Yeah. You either donât and you live with the fact that an agent is gonna get halfway through, die, and youâre gonna retry the entire thing, which is a ton of wasted effort and time, or youâre trying to string together this architecture yourself, building queues and so on. And that is gonna take you a ton of time and be relatively inflexible, and thatâs kind of the antithesis of AI. With AI...
Mm... You need to be able to update your code, your agents, your harnesses, your prompts, your models really quickly.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
Because models are gonna change relatively frequently. Youâre gonna learn a bunch as you run things in production, and youâre gonna take that learning and put it back in your application code to continue improving your harness. So in general, itâs sort of got to the point in which you need to develop using some sort of harness, and your harness is almost always going to be using durable execution so that it can generate the traces, retain state, retry correctly. Yeah.
Turner Novak:
And is this not something thatâs just like built into the AI products already? Like it just kinda sounds like... Itâs so table stakes. Yeah.
As somebody whoâs not in the weeds every day like you, Iâm just like, âOh, this is like that shouldâve been a solved problem,â âcause itâs so obvious.
Tony Holdstock-Brown:
It shouldâve been a solved problem, and itâs so obvious. I think like thereâs been some attempts at trying to fix this. For example, agent SDK, is a thing But it doesnât go as far as you need. It allows you to create some sort of harness yourself.
But it doesnât go as far as you need in order to give you the durability, the reliability, concurrency controls, constraints for each of your users. And this is also relatively new in that, you know, weâve been going for like four or so years. These are all relatively new things in the scope of infrastructure.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
So, I think a lot of people have also been trying to experiment with what an agent SDK could look like, like an agent framework itself. And some of those learnings are that each particular LLM call is in a step, and that LLM will return some data, which will maybe ask you to do some tool calls, and you run this in a loop up until you hit some goal TL;DR is that still needs steps and that still needs durability
Turner Novak:
Hmm. And to, to your point you mentioned earlier, when you first started Inngest, like, no one was using, like, this kind of capabilities. Like, it wasnât-- we didnât need to do this. Yeah, yeah.
So how did it kinda start, and then how did it evolve over time?
Tony Holdstock-Brown:
Yeah. So, Iâm gonna go through, like, how Inngest started, and then AI, and then the evolution post-AI. But real quickly, I used to run engineering for a healthcare company which is super event-driven âcause thereâs a good audit trail of all your events here. So what are some
Turner Novak:
Examples of stuff that you had to do?
Tony Holdstock-Brown:
Yeah, like when a patient filled out a treatment form or a doctor approved a treatment form or a patient asked for a prescription, you have to do, like, a bunch of different things, which is a series of steps. And, you also have to check whether or not specific things have happened in that flow. Like, for example, did a doctor approve a prescription within twenty-four hours? And if not, youâve gotta follow up on that particular request.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
That was literally one of the worst things to build, and it took us months to build these really basic flows. And, turns out that if we had Inngest with its step functions, it would have taken a day or two. Mm-hmm. So the APIs and the concrete ideas from Inngest around an event comes in, and it runs a step function.
You can pause that step function up until something happens in your product. And then it will automatically resume or time out, and it will automatically resume, and youâll see that that particular thing didnât happen. All of those came from trying to build this in the first place. And it turns out that step functions are really, really good for AI because it is a literal loop of steps that weâll call an AI LLM, and it will do something afterwards with that response over and over again until that goal has been achieved.
And so step functions are a great way to build this.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
So we, in some ways, got lucky that, AI needed our infrastructure. But then it turns out that our infrastructure is basically in the perfect place for running AI. And Iâll quickly talk about what that means. When youâre running AI and youâre running these step functions, I think everyone knows this by now, you wanna store all of those agent trajectories as traces so that you can see what your AI is doing.
If itâs doing the right thing, has it produced the right outcomes? And right now, a lot of people are doing this by storing traces separately from their infrastructure, which is interesting. Um...
Turner Novak:
Is that, is that a bad way to do it?
Tony Holdstock-Brown:
Yeah. I think, like, from a first principles point, absolutely. So this... Whatâs, whatâs so bad
Turner Novak:
About it?
Tony Holdstock-Brown:
W... Right. So a question, a question here is, like, what happens if you wanted to replay your twenty-step function and change step eighteen to see if that improved the outcome? If youâre just recording what happened in your step functions, and youâre not doing it this, this at the execution layer, itâs particularly hard to replay steps one to eighteen, change step eighteen.
And then see if it improved the outcome because you donât have that deterministic playback But step functions give you deterministic playback, which is really good for building self-improving agents and checking whether or not changes to those agent trajectories actually work. And so turns out that observability data is essentially a derivative of your infrastructure and your application code. And if your infrastructure/step functions can create these traces, then theyâre coupled directly to how the infrastructure runs these steps, and you get a ton more information, a ton more context, which allows you to build this self-improving loop way, way faster. So you essentially need the observability to be built into the infrastructure...
Basically... In order for the self-improving system to... Basically... Self-improve.
Exactly. As far, as far easier. And, and one like two, I suppose, examples here are Yeah, if you wanted to replay something and change step 15, much easier if youâve got something deterministic to play back steps one to 15. And then you can see if changes to step 15 improve the outcome.
Similarly, if you wanted to, for example, A/B test two individual harness changes without changing your infrastructure, that could be particularly difficult. But if you built this into the infrastructure such that you could A/B test different steps or different groups of steps, then everything is really, really simple, and it sum... Suddenly becomes really easy for you to change prompts or models or harness changes to, like, 1% of your users because the infrastructure is aware of whatâs happening. Hmm.
It can fork two different paths, two different steps or groups of steps. And then you can safely roll out, I donât know, two deltas between Anthropic and OpenAI to 50% or 20% of your users and check to see that the outcomes were the same but that the token costs were cheaper.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And so either way, youâre gonna be building this sort of stuff directly into your step functions and your flow of the harness. And it would be really nice if the infrastructure just gave you the tools to do that, âcause otherwise youâre gonna be rebuilding, rebuilding the same stuff over and over again.
Turner Novak:
So how would you do that if, like, it-- Inngest doesnât exist? Yeah. Like how-- Or I guess you guys do. You guys build it and then...
Yeah. Open it up to other people. But, like, so how, how do you, how do you actually make that possible? Or how would I do it if I wanted to, like, you know, Iâm gonna build this myself?
Tony Holdstock-Brown:
So this is a really interesting question. Right now, to be frank, when we speak to a bunch of our users and users that just push out AI... Yeah into production that donât use us...
Turner Novak:
Yep...
Tony Holdstock-Brown:
People do local evals to check, do the vibes seem good...
Turner Novak:
Yep...
Tony Holdstock-Brown:
With the current AI changes that Iâve made, with the current skill changes or prompt changes? This is, like, the current industry standard, right? Current, yeah. Do the vibes seem good?
Itâs like itâs just... Yeah. Am I, am I okay to push out these changes? And then youâre gonna, I donât know, change your prompt locally, change your harness, maybe run local evals on 100 test cases, see if that works.
If it looks good, YOLO it into production. Because LLM as a judge is relatively expensive, youâre probably not gonna be checking 100% of your agent production runs. I actually havenât met a single person that does that for 100% of your production runs.
Turner Novak:
You basically just have, like, hereâs the 100 most common things, and we just check those... Hereâs my golden test case... And
Tony Holdstock-Brown:
If theyâre good, then... Hereâs my golden test case of a absolutely stochastic kind of random system. Letâs hope that it looks good, and push it to production for 100% of users, and maybe Iâll spot check using, using either LLM as a judge or human review sample of what happens in production, and I hope that it works. Which is bananas.
Like as... Absolutely crazy. Literally, literally bananas.
Turner Novak:
So but isnât it probably, like, kind of okay, and weâve been able to get away with it... Okay because itâs like... Th... Th...
Thereâs like, these arenât necessarily life and death situations necessarily in a lot of cases so
Tony Holdstock-Brown:
Far In a lot of cases, yeah. In a lot of cases so far. In a lot of cases so far, if youâre building a vibe coding agent and youâre doing something in which, you know, processing a HTML, building a React app, youâre like, âOh, maybe this harness change is gonna improve things because weâre moving from TypeScript 5 to TypeScript 6, or 7 was just released today, you know, letâs move to that. â Fine.
But if youâre doing something in which youâre maybe, weâve got a bunch of users that use this, one of which is doing therapy, and mental health using AI, those circumstances are particularly important because you wanna make sure that youâre abiding by specific regulations. You also wanna make sure that any changes to models or prompts arenât inducing harm to patients. Mm-hmm. That sort of stuff is particularly important.
And also, generally speaking, if youâre, if youâre changing a prompt, there is some intended outcome that youâre looking for.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
Youâve probably changed it because youâve seen that the agent hasnât done the right thing in some particular use case.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
Thatâs when you bring something back into your local models. But you know that this is just, like, gonna happen over and over and over again. So itâd be nice to be able to test outcomes predictably and make sure that the changes youâre pushing to production actually do the right thing. Which is what weâve been doing in engineering for forever...
Yeah and weâre sort of lost with AI, which is craziness. You, you think we lost it? Um... It was just âcause the tools werenât there to continue it or?
Turner Novak:
Yeah,
Tony Holdstock-Brown:
I mean, I wouldnât necessarily know if weâve lost it or we just donât know how to do it with AI right now.
Turner Novak:
Hmm. Thatâs fair. And, and so how did it work then that Inngest was so well positioned to be able to do some of this? Like...
Yeah... Because thatâs usually not the case.
Tony Holdstock-Brown:
Totally. Yeah. Um... When we started Inngest, there was always this particularly interesting concept of, of us sandwiched in between your application code and the infrastructure.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And by that, I mean if youâve got a step function that has five steps, and you write this in TypeScript running on AWS, you could rewrite your function in Python and move to GCP, and your function will pick up where it left off, even if itâs halfway through. âcause we retain the state. We call your function in GCP instead of an AWS, and it picks up where it left off with all the state from the prior three steps, and it will finish running in Python. And you get, like, cloud migrations.
And this means that weâve abstracted compute, and suddenly computeâs completely fungible, including maybe the programming languages, which is really, really cool. And thatâs awesome because then we create this deterministic environment for running your application code, which is also really good for AI because it turns out that if youâre trying to run an LLM and a step, and an LLM and a step, which might be a tool call, that deterministic environment for running AI just automatically applies. So weâre in this particularly good position because the fundamentals just enabled that. Hmm.
And from there we can build really, really interesting primitives that do things like A/B testing variants, and then also using this tool that we have called Step. Waitforevent to wait for things to happen in your product that prove whether or not AI did the right or wrong thing. An example is like you build a coding agent or a PR review agent, and thereâs a PR. It reviews some code, tells you that thereâs a P0.
If you see that the PR was merged without any new commits being pushed, maybe that P0 was an encrypt flag. So you can listen to that webhook using Step. Waitforevent. And then you can take that webhook information and then automatically score your agentâs run as either good or bad based off of product events, which is really sick.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
No one can do that right now. And that allows you to grade outcomes on 100% of your production agents using the same primitives weâve had for like four years. And I think, like, thinking about things from first principles when we were originally building the system allows us to do a ton, whether or not youâre building with AI or itâs just, like, regular infrastructure. Just kind of all fit together, which was both lucky but also part of how we engineered the system for flexibility.
So I mean, it
Turner Novak:
S... It sounds like, I mean, since this is kind of like so important at this point Should you not be building some of the stuff yourself because you-- it helps you understand it better? Or, like, why would s... Why would somebody build this themselves or use something like Inngest to, to fix it?
Yeah. So, firstly, I think
Tony Holdstock-Brown:
A lot of people build their own jank eval harnesses, which is okay âcause evalâs really locally are just unit tests.
Turner Novak:
So th... Is that pretty much what people have been doing?
Tony Holdstock-Brown:
Pretty much what people have been doing. Is they, they kind of do... Yeah. Okay.
Itâs craziness, yeah. I donât think many companies have the ability to check whether or not their AI or agents are performing well based off of product outcomes.
Turner Novak:
How do you do that?
Tony Holdstock-Brown:
You have to consume basically every signal from your product. And then you have to specify whether or not the signals indicate that AI is doing the right or wrong thing. I
Turner Novak:
Mean, how do you even gauge that?
Tony Holdstock-Brown:
Yeah, yeah. So, so itâs... Like, what is being,
Turner Novak:
What is being measured?
Tony Holdstock-Brown:
Yeah, yeah. So there was this interesting tweet from OpenAI where someone was complaining about OpenAI recording whether or not you copied and pasted from chat.
Turner Novak:
Oh, interesting.
Tony Holdstock-Brown:
Yeah, crazy. And somebody was, like, furious at OpenAI on Twitter, just, like, tying, tearing into them. And then... So
Turner Novak:
H-how did... So how did that go? Like, this, this...
Tony Holdstock-Brown:
This PM replied saying like, âWell, dude, no one hits the thumbs up, thumbs down,â and so we have to take signals from the product, like are you copying and pasting some of our responses...
Turner Novak:
So then they know that
Tony Holdstock-Brown:
It worked... To figure out whether or not... Exactly, yeah. Mm-hmm.
Because if chat says something good and you copy and paste that, then the chances are thatâs a pretty good outcome.
Turner Novak:
Yeah.
Tony Holdstock-Brown:
And if you donât copy and paste anything, maybe it is, maybe it isnât, but itâs very ambiguous.
Turner Novak:
Yeah.
Tony Holdstock-Brown:
So OpenAI themselves use product signals to indicate whether or not their chat has done the right or wrong thing.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
And if youâre say, say for example something like Zapier do something similar, if you enable a workflow that was generated by AI, then AI did a good thing.
Turner Novak:
Yeah. Itâs almost, itâs kinda like if you churn out of your session, it was a success.
Tony Holdstock-Brown:
Yeah, in some ways. In some ways. And so depending on what your agent does in your own product, you can classify particular product signals as either good or bad markers.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
And then you can rate your agents using, honestly, like, super cheap events. Youâre not paying the crazy LLM as a judge on every single agent trajectory. Youâre just saying, like, there was a patient, patient came in, AI gave an answer through chat Did the patient follow up with an appointment? Did they not?
And based off of that particular outcome, maybe agent did a good thing or maybe it did a bad thing. Depending on your own product, youâll have signals that you can gather. And then you can start automatically rating, rating agents, which then allows you to do more advanced things in the future. Like for example, A/B testing to see if the same outcomes were generated with cheaper models or open-weights models.
And that way you can safely roll out new open-weight models in your products, knowing that you have sa... The same outcome and the same efficacy of your agents overall, but that the cost is like way cheaper. And this is interesting because this is like where the world is moving.
Turner Novak:
Yeah.
Tony Holdstock-Brown:
Like, this is like a big thing on Twitter. Iâm not the biggest Twitter user, but itâs still a big thing on Twitter. People were talking about how good GLM 52 is, which is this open-weights model. Everyone loves it.
And if youâre somebody building with agents right now and youâre paying super high token costs to one of the frontier labs, maybe internally you think, âI can take GLM 52 and swap it in for part of my agent loop,â and that would reduce costs dramatically... Mm. By like ten to 20x. And a question would be, how do you track the outcomes to make sure that itâs as effective?
Turner Novak:
Mm.
Tony Holdstock-Brown:
And using something like detecting whether or not your product is doing the right thing...
Turner Novak:
Mm...
Tony Holdstock-Brown:
Is one way of doing this at scale, cheaply for one hundred percent of your agent runs, which is cool. And it gets better because then you have all the traces and agent trajectories. You have product signals to know that you have known good trace trajectories and agent trajectories, which then you can take to post-training for open-weights models so that you create your own model thatâs relevant for your own sort of product...
Turner Novak:
Mm
Tony Holdstock-Brown:
And then swap that in using the same experimentation method. And then you get like better models than you would from frontier labs that are generic. Really, really, really good with a ton of intelligence, but super expensive...
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
To something thatâs much smaller but suited for your use case and also way cheaper and more efficient to run.
Turner Novak:
Mm. And so is there like a whole other, I donât know, like dashboard insights, and thereâs like an observiti... Observi... Yeah...
Observability layer of Inngest that people are also getting and using?
Tony Holdstock-Brown:
Yeah, for sure. For sure. Okay. And like we had to do this for just like step functions, durable execution anyway, because you need to see what steps are running.
So we had to have observability. So itâs like observing your
Turner Novak:
Own product performance?
Tony Holdstock-Brown:
Exactly, yeah. So we give you the same sort of observability youâd expect for AI. What steps are running? Which LLMs did we call?
What were the tokens? What was, what was the TTFB, the time to first byte? And how much did it cost? What were the total tokens consumed, input and output, and so on?
We give you all that information. And we also allow you to tag things with metadata, group things by session in case you have many runs but one single chat session... And all of this combines to give you basically a complete overview as to whatâs happening with your agents, using basic step functions. So itâs kind of out of the box.
Turner Novak:
Yeah. And then h... Like, how did you, how did the product evolve into this over time? Like, what kind of pull were you getting from customers to...
Tony Holdstock-Brown:
Yeah. So, again, like generically, we, we started as like a basic... Iâd say basic is really complex to build, but basically a over specific pieces of infrastructure queuing events. I
Turner Novak:
Mean, thatâs usually how infrastructure companies are initially started. Yeah. Itâs like some, like, really boring, kinda like lame, not... Yeah...
Not fancy thing.
Tony Holdstock-Brown:
Yeah, exactly. Exactly. And so, like, we allow you to build step functions but in a much nicer way than any company had previously done.
Turner Novak:
Mm.
Tony Holdstock-Brown:
And that was like, cool, people liked it. Since then, a lot of people started using us for AI, and the AI observability piece made sense. Using Inngest step functions, thereâs some really fancy things you can do that arenât possible. Like, Golang has this defer keyword that will queue up a function to execute when your parent function finishes, and we basically brought that to TypeScript.
So you can use defer to run a function to evaluate and judge your agent runs once the agent finishes So we saw like a bunch of different use cases, and we figured we already give you the observability data, and we already manage the process of calling LLMs, and we already orchestrate your entire function. Therefore, it would be super easy for us to build AB testing so that you can check two specific models, because we already do the orchestration. And if weâre doing AB testing, you need to be able to score particular variants to see which one won. And you can do LLM as a judge, and we can do that for you as well.
But thatâs expensive. An easy way of checking whether or not something did the right thing is just did the product do the right thing, which we can do using our previous primitive to step wait for event. So it all kind of came together in that customers just really needed this, our users really needed this, and itâs really, really, really hard to build yourself. And weâre seeing the same thing around like just general compute.
Like literally every one of our users right now that uses something like sandboxes has orchestration, and they either do orchestration out of the sandbox or in the sandbox, which is an absolute pain. And they also have to manage the sandbox lifecycle themselves, like starting, stopping, suspending. It would be really sick if you could just do step function, have something like group. Sandbox, and inside that particular sandbox, if you ever said, âI want to sleep or wait for this specific thing to happen in my product,â the sandbox automatically suspended.
You didnât pay for any compute because you werenât running anything. You get automatic durability, and that orchestration lives outside of the sandbox. So youâre not paying for like hella RAM that you donât need because youâre not putting an orchestrator in the sandbox for thousands of copies paying for like so much extra RAM. You just have one copy of the orchestrator that sits outside managing many sandboxes, which can be much lighter weight.
And you get like a much better experience and... The sandbox in this case is spin up an environment that then goes away when you donât need it anymore. Totally. Yeah, yeah.
Okay. Just running arbitrary code. So a bunch of our users do stuff like that for like code review. You wanna Git clone someoneâs PR.
So you Git clone their, their, their code, check out the PR. And then you have an agent that does a Git diff, looks at the code changes, and analyzes the code base to see whether or not it did the right thing. But thatâs like an ephemeral, ephemeral environment. You wanna do that in a VM that is completely isolated from other customers.
You might be doing code review for thousands of customers, and you donât wanna mix peopleâs code bases together.
Turner Novak:
So you have 1,000 different... 1,000. In theory, you might need 1,000 VMs, virtual machines, constantly on and constantly running... Yeah...
Which would cost much more than... Totally... Only using it when that customer is using the...
Tony Holdstock-Brown:
Totally. Exactly that. And youâre managing all of that yourself. Youâre managing the orchestration.
And so like if youâre building durable execution You can just build a much nicer primitive for this overall because you have the context of when your function starts and stops. You can start a sandbox using sort of like idempotent APIs built in without you having to manage the sandbox lifecycle, which you can suspend and resume. Like for example, if you did build a code review agent, create a sandbox, run some steps to review the code, use step. Wait for event to wait for the review to come back, be either approved or rejected or new code to be pushed.
But when you do step. Wait for event to wait for that feedback, sandbox automatically pauses. You donât pay for any active CPU, you donât pay for... You donât really need to do anything.
Thereâs like a ton of stuff that our users have asked for that weâre essentially building and releasing because it just makes sense. So weâre going both deeper on the infrastructure to give you compute sandboxes, Lambda runtimes, VMs, that sort of stuff. Based off of SDK, no Terraform, just write some basic application code using our SDK and it will auto-provision, plus up the stack for observability so that you can see exactly what your application is doing, get trajectories. And then build better agents.
And our view is that theyâre really combined because one is a derivative of the other, AKA observability is a derivative of what your product does, and if you get them both for free, then everything is really nice.
Turner Novak:
Mm. It, it sounds like itâs, youâre helping them build a better product, but also decreasing the cost that it takes. Totally. Yeah.
Like maybe, maybe thereâs faster speed in there too... Yeah, yeah,
Tony Holdstock-Brown:
Yeah...
Turner Novak:
If Iâm interpreting this all right.
Tony Holdstock-Brown:
Yeah, basically. Basically, like similar to, similar to what basically everyone will say in every infrastructure company. Our views on things are that if you have an SDK that defines what your code needs to do, AKA run step one, two, three, in a loop, then thatâs far better than you provisioning Kafka, provisioning queues, provisioning servers. And then managing everything yourself.
You write basically five lines of code. An agent, an AI can literally just like churn this out in one pass, and youâre, youâre, youâre ready to deploy on any infrastructure. Good to go. And we run like everywhere.
We donât care where you host your code. You can run it on our compute, soon, or you can run it on Railway or Render, like it doesnât really matter. We
Turner Novak:
Donât care. Interesting. And you, you mentioned that youâre, youâve got this bare metal bet. Yeah.
What does bare metal mean in this case for someone who doesnât know what that means?
Tony Holdstock-Brown:
Yeah. Okay. So, this is sort of nerdy and very deep on the infrastructure level. So firstly, Iâm gonna talk about how we work, and then Iâll talk about what we do and why, and why it wouldnât work on public clouds.
And then Iâll talk about what that means for the future of AI.
Turner Novak:
Mm.
Tony Holdstock-Brown:
The short story is You have a series of steps that run. If you imagine youâve got five LLM calls, first to classify, second to get some context, third to get some, like, information about whatever the userâs put in. Any one of those steps can fail. And so we have to save all of the information from step one, all of the information from step two.
So that if step three fails, we can restart your function.
Turner Novak:
From
Tony Holdstock-Brown:
Step three, pass all of that data. And
Turner Novak:
You can exactly-- Youâre at exactly the same point.
Tony Holdstock-Brown:
Super deterministic, yeah, ex... Like 100% determinism. But in order to do that, weâre saving a bunch of state every time a step runs, and we need to delete all of that state when your function finishes. And so weâre getting tons of data from you, which might be encrypted.
We donât care what the data is, but still itâs super bandwidth-heavy. And AWS, GCP, and all the other big clouds are extremely expensive when it comes to both compute and bandwidth. And so for us, itâs kind of untenable to make this really cheap for our customers on public clouds because of the amount that they charge. So the only way to build a really good durable execution company and step functions in general is to either sell it at a really high price because youâve gotta eat the public cloud cost or do it yourself on your own servers, which is bare metal.
So we run out of multiple DCs. Weâve got our own racks. We do everything from the switches to the firewalls to the machines. The only thing we donât do is the power and the connectivity to the internet.
And we, we run everything ourself, which is a lot of work for a smaller company. But the costs are, like, 20 times cheaper, which is insane, and the performance is way better. And that means that if you were to use, for example, our sandboxes or compute with us, you would be running on bare metal extremely close to where your workloads run, the queue, the, the, the execution, the step function. And also because we run our own machines and we manage our own connectivity really, super cheap, like super cheap, way cheaper than you get from anyone else, which is basically reselling public clouds And.
And so that just makes it. So.
Turner Novak:
If Iâm essentially using public clouds, thereâs like a baseline of what I, I just canât go below that price âcause I... Yeah I need to still make money if Iâm selling
Tony Holdstock-Brown:
It to you. Yeah, exactly that. Exactly that. Yeah.
Okay. So a lot of folks that are like the Neo clouds, I know Railway took an early bet on, on, on building on bare metal. So theyâre one of the few not to. But a lot of the other Neo clouds, specifically just resold AWS for a long time.
Mm. And that means their costs have to be higher than AWS because otherwise theyâd lose money.
Turner Novak:
Mm.
Tony Holdstock-Brown:
So thatâs, thatâs really, really tough. So we, we, we took an opposite bet, and that was our view since we started the company. We had to build on public clouds to begin with because, thatâs just how you get started. But we quickly moved off of that to bare metal.
And thatâs like, thatâs been really, really good for us.
Turner Novak:
How do you think the next year or so is gonna go in AI infrastructure? Like I, I donât really know what the best thing to ask your opinion on is, âcause you probably have opinions on a lot of it. But like how do you kinda think the next 6 to 12 months... Yeah will look like?
Tony Holdstock-Brown:
I think like itâs interesting, and multiple people have pointed out something very similar, which is everything is converging. Cloudflare and Vercel and Railway and all the other folks sort of look similar right now. And AWS released their own versions of step functions called durable functions, which looks very similar to ours recently, which is cool. Everyone is converging on the same sort of concept because it turns out that when youâre running code, step functions are a good way to do that.
They give you the observability, and itâs a really, really good abstraction so that you get all of these trajectories. I think it turns out that like, as youâve probably heard before, no one wants to mess around with infrastructure. Agents donât wanna do that either. You donât really wanna mess around with Terraform and have this really awful provisioning policy.
And so things are moving much more lightweight, like SDK first. And infrastructure for AI sort of looks like this self-reinforcing loop in which you get step functions, agent trajectories. And then you take all of that data that runs your own production systems Do post-training on lighter models so that you can get your cost down. And then you have this inference layer that will run your own models, which is particularly interesting.
I think a lot of AI infrastructure is basically moving to that direction of, of sort of inference hosting, to be super lightweight. And Neo clouds are in this really good position to take over, which is really, really, really cool.
Turner Novak:
To capture a lot of that
Tony Holdstock-Brown:
Inference value spend. To capture a lot of the inference spend and compute spend.
Turner Novak:
Mm.
Tony Holdstock-Brown:
And, and I think, like, this is also the case with, like, the proliferation of people that are becoming engineers, and I donât think many of them would like to deploy to EC2 by creating Terraform. I donât even know if many of them would know what a VPC is, whether or not youâre using public or private IPs in your VLAN. And so Neo clouds are in this really good position to capture this new wave of developers, so people, people donât wanna think about that really.
Turner Novak:
Yeah, âcause, like, in theory, you could say shouldnât AI... Like, couldnât you just say, âHey,â like, âClaude, just do the back end for meâ? Yeah, yeah. Like, itâll do it.
Like... Itâll do... Is that not the case or? I mean,
Tony Holdstock-Brown:
It, it might do it, and youâll probably approve it, and youâll probably be like, âCool, that looks good. â Whether or not it firstly does the right thing, secondly introduces a bunch of complexity, and thirdly massively increases cost is a huge question. And in some ways, I think basically throughout engineering weâve learned that reducing complexity is good, and the delta between having all of that work And then just writing six lines of code and have it automatically work is huge.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
Because then thereâs far, far, far lower chance of mistakes if all of this is handled for you...
Turner Novak:
Mm
Tony Holdstock-Brown:
Than there is if youâre writing a Terraform policy that has EC2 and then youâre attaching security groups and youâre managing IM And then you hope that, you hope that whatever model youâre using knows how to run all that and deploy it all correctly. So I think, like, generally speaking, yeah, models both get smarter. But you also wanna focus specifically on your differentiator, which is running the product and building the product rather than managing said infrastructure, which kinda sucks.
Turner Novak:
Hmm. Yeah, I, I guess thereâs, like, the, how specialized is the thing that youâre doing, and, like, should you do it yourself versus if somebody... If, like, if itâs a shared problem by everyone else, thereâs probably, like, a shared provider. Yeah.
And I think if you donât get any ex... Any specific edge or differentiation, like, as a, as a company thatâs using one of these... Yeah like, you should just-- Theyâll fix the problem for you. Like, donât waste your time solving this when itâs already solved.
Tony Holdstock-Brown:
Yeah, yeah,
Turner Novak:
Yeah. Totally. Yeah. Like, go, go figure out other things that
Tony Holdstock-Brown:
No one else has solved yet. Yeah, totally. Totally. And, and, like, one question that you maybe have as well is the future of AI.
I know recently everyone talks about loops and software factories.
Turner Novak:
Yeah.
Tony Holdstock-Brown:
And one question... Thatâs
Turner Novak:
What, like, a step function is, right?
Tony Holdstock-Brown:
Like, youâre just running loops. Basically just doing the same... You guys have been doing loops for four years. Yeah, yeah.
Exactly. And, like, one question youâll have if youâve got a loop or a s... Or a software factory, and youâre like, âI wanna fix this bug,â is, like, how would I do that? You...
Some interpretations are like, âUse Fable. Use this great model, and it will read the code, and it will analyze everything, and it will use extra high thinking. And then youâll understand exactly whatâs happening, and maybe youâll fix the bug. â Hmm.
And, like, no joke, sometimes it will just come up with a bunch of crap, and it wonât be the right issue.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
The right way to do things would be to take a look at the logs, take a look at the errors, trace that code path to fix the bug. And then have a better understanding of exactly what went wrong.
Turner Novak:
Couldnât AI do that?
Tony Holdstock-Brown:
In order to do that, you need everything to be set up correctly, right? Like... Hmm... Youâd need to be taking logs from your EC2 service and from the applications, and make sure youâve got the right log train sorted, and youâve gotta make sure that all of the steps to, for all of this observability was properly tracked.
Or you could just use something that allows you to YOLO six lines of code, and you get all of that out of the box. So if there was a failure, it was reported. You see exactly what step failed. You have an API endpoint for listing those errors, and then you can mark them as resolved really easily.
Your agent would know how to do that âcause skills in MCP is a thing, and all it has to do is make one query to get all of the errors. And then it can tie that directly into the code context because it will see which function failed and why. And that process of fixing things in your self-reinforcing loop of software factory becomes super simple. And so I think thereâs, like, a also sort of where the puck is moving with agents and infrastructure, that you want things to be as simple as possible.
So you can get the right outcome in as few steps as possible with agents. Because the fewer steps you have, the less chance there is for failure overall.
Turner Novak:
Hmm. Can you expand on that a little bit
Tony Holdstock-Brown:
More? Yeah, I think like if youâve got an issue in your code base and youâre like, âHey, there was this race condition and these two things happened,â you could have your coding agents using maybe Cloud Code or OpenCode or whatever you want, go ham, read the code base, spin up a ton of sub-agents to read each particular package, hope to figure it out.
Turner Novak:
This is like kind of what people do, right? Itâs
Tony Holdstock-Brown:
Kind of what people do, yeah. Thatâs, thatâs kind of software factories overall. And thatâs like super expensive, and youâre hoping that that agent, your models have enough knowledge to piece together things through the context that it generates to find the right issue. Mm-hmm.
And youâre like hoping that that brings
Turner Novak:
Success. And itâs basically just like spin up to a ton of things, just get as much context as you possibly can... Mm-hmm. And then like...
Tony Holdstock-Brown:
Mm-hmm.
Turner Novak:
Thread, like thatâll get you the answer... Thread the needle... âcause youâve just...
Tony Holdstock-Brown:
Thread the
Turner Novak:
Needle... Youâve just got all this stuff.
Tony Holdstock-Brown:
Yeah. So you pay for super expensive models that have a bunch of knowledge, and you hope that it threads the needle to get the answer, which is cool.
Turner Novak:
And it, it usually does kind of work sometimes.
Tony Holdstock-Brown:
Usually, yeah. I mean, it works. Or sometimes. Okay.
And then all these new models are getting better and better, so that definitely works, but itâs also like hella expensive. Mm-hmm. Crazy expensive. And this is the whole thing about Fable right now, everyoneâs like complaining that Fableâs gonna cost a bunch of money, and itâs token use, and itâs not included in my subscription, all that craziness.
Yeah.
Turner Novak:
I still think itâs crazy that we went through this era of token maxing. Like... Yeah... I mean, I get it, like learn it, see what it does, but also like who, whoâs paying for all this?
Yeah. Like, is... I mean, that was just like crazy, those headlines. Like, I think it was Uber spent like a billion in a quarter or whatever an hour.
Itâs crazy. So Iâm like... Itâs crazy... Thatâs nuts.
Itâs crazy, man. Like
Tony Holdstock-Brown:
Absolutely
Turner Novak:
Insane... Yeah, yeah... That that happened. I mean, I...
Itâs probably good for them. Like Iâm su... I mean, thatâs a drop in the bucket for them, whatever. Like theyâre, theyâre fine spending a billion...
Yeah... And then they probably learned a lot, whatever, however they wanna spin this. Yeah, yeah. But Iâm also like, how is that...
Thatâs just like classic like, like top of market type behavior. Like money doesnât matter. Like... Yeah...
Just spend it as much as you can.
Tony Holdstock-Brown:
Dude, itâs crazy. Itâs crazy. And I think like the, the kind of overview of like how agents work and everyone building crazy agents that like go out and paralyze on 10 different tasks and just YOLO merging stuff. And then youâre paying for agents to review the agentâs code is like, cool.
Maybe weâll probably get there someday. Right now itâs like super expensive to do that, and it works fairly often, which is good. Think, think itâs kind of like playing the lottery, you know, and you get addicted to the win. Like, oh my God...
Mm-hmm. This PR landed, it shipped. I didnât have to do anything, and I just gave it a little bit of guidance. Yeah.
And some PRs youâre like, âOh my God, this thing is absolute-â dumpster fire. Which is terrible. And so like, yeah, itâs kind of like playing the lottery, which people love, you know? Same, yeah.
Like that sort of gambling aspect of will it do the right thing or will it not? I feel like itâs part of it though when youâre,
Turner Novak:
Like when youâre using like Claude. Mm-hmm. You got that little, the ink blot thatâs like... Mm-hmm.
Expanding and now itâs like, âOoh, whatâs it gonna do? â Yeah. Like, howâs it gonna work? Yeah, yeah.
What am I gonna get out of this?
Tony Holdstock-Brown:
Yeah, yeah,
Turner Novak:
Yeah. And then, and then the first line will be like, âI just analyzed the information, now Iâm doing this thing. â Okay, cool. Itâs on the next step.
Yeah, yeah. Itâs like, all right, whatâs the next output gonna be? Yeah. And itâs like, now I just did this.
And youâre like, âOh, yeah, itâs getting closer,â like what are we gonna get out of this? Yeah, yeah,
Tony Holdstock-Brown:
Yeah. Exactly, exactly.
Turner Novak:
And then youâre like, itâs kind of like that dopamine hit after a couple minutes of like... It is, it is okay,
Tony Holdstock-Brown:
Itâs circling... Youâre just waiting. Youâre like, âOh, this is actually pretty good. â Itâs also like why I, I canât wait.
I was talking about this with somebody from Nebius and like, I donât know, the future of AI and the future of coding agents, the future of all these harnesses is super interesting. Especially if you consider like diffusion models, which give you like 1,000 token a second performance. âCause like youâve got this waiting and youâre running all these agents, and youâve got this crazy loop, and itâs doing all this post-processing. And if you can just like spew out the entire response in a second...
Really quick or sub-second, like thatâs craziness. Yeah. And the efficiency you get from that is going to like completely change how we build things. Mm.
And itâs gonna completely change how fast, how often you can check, a modelâs output. So Iâm like super interested in where things go, both like with traditional LLMs, with diffusion models, with harnesses, with this engineering. But it all sort of revolves around the same thing, which is like how can we track whether or not itâs doing the right thing overall? How can we track outcomes in your product?
I think this is also why like coding models are so easy. You know, youâve got like unit tests. Did it or did it not do the right thing? Youâre saying coding...
Itâs
Turner Novak:
Very
Tony Holdstock-Brown:
Easy... Because itâs just determine...
Turner Novak:
Like itâs very like rule-based. So easy. Itâs just like, yes, this works. So easy
Tony Holdstock-Brown:
In comparison to see whether or not agents are doing the right thing or your modelâs doing the right thing. And itâs much easier to train than it is doing... Mm something thatâs very hard to gauge. And so like, yeah, Iâm super interested in the future and about where things go.
All of that said, I think overall, you know, harnesses, loops, all this sort of stuff will ultimately remain âcause the only option to not have these loops is to have a one-shot prompt that gives you this golden answer, which is literally AGI. And if you donât have AGI, youâre going to have to massage and control the LLM to generate thought, tools, context, and so on. And thatâs always gonna be some sort of loop. So durable execution, step functions, and workflows like must exist up until AGI hits, and at that point like all bets are off.
Turner Novak:
Yeah. Iâve kind of... I mean, in terms of the whole like AGI discussion... Iâve always just been like Is that, like, is it even a thing?
Is it even a thing? I donât know. Yeah. âCause it, with, with self-driving cars, I feel like it was, like, 10 years of everyone like, âItâs two years away.
â Dude. It was never here, and then all of a sudden itâs here.
Tony Holdstock-Brown:
Yeah, yeah,
Turner Novak:
Yeah. And it doesnât do 100% of it, though. Like, you still, like... Itâs not 100%.
Itâs not 100%. Like, driving. In
Tony Holdstock-Brown:
SF, all Waymoâs stopped because of the fireworks recently. Oh, did they? Like, on July 4th. Yeah, theyâre like...
Oh, I didnât know that... Tons are being towed. It just, like, completely failed. Wow, thatâs crazy.
Which is super interesting. Yeah, super interesting.
Turner Novak:
Yeah. So itâs like, and the same thing, like, when... Is, are we 50 years away from, like, true AGI? I donât know.
Yeah,
Tony Holdstock-Brown:
Yeah.
Turner Novak:
But then on the other side, itâs like you kinda... When I, when I think back three years ago, four years ago when this entered the discourse, like you need to be able to raise money. You need to be able to convince people to come work for you.
Tony Holdstock-Brown:
Mm-hmm.
Turner Novak:
Are you... Who youâre gonna use as your AI provider? Yeah, yeah. This like stodgy old 25-year-old company or this like, you know, weâre building AGI.
Like youâre gonna, youâre gonna use... Yeah, yeah... The new cool AI. So itâs like I get why it was done, but Iâm also like I kinda hate that all the disc...
Like weâve been 18 months away from AGI for a couple years now.
Tony Holdstock-Brown:
Forever. Forever. Forever. And we got, we got, we got Fable, which is marginally better than Opus, you know?
Like itâs pretty cool. Itâs pretty cool. Itâs good for coding. But like, yeah, I, I, I both agree and also think that you are still, when youâre building products, going to need to provide context.
Youâre going to need to give the model more information when it asks for it. Youâre going to need to build some sort of loop, some sort of harness over what the model needs.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
Youâre going to need code to interface with other parts of your system to give the agent what it needs, and youâre going to need to track that so that you can guarantee itâs doing the right thing, whether or not like the model is extremely good or whether or not the model is like...
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
A year old from now, you know? Like gimme, so.
Turner Novak:
Well, be... âcause you could make the argument to your point of right now people just kinda like give it to the model. It solves it. Itâs good enough.
It costs a lot of money. But if we say if the models continue to get better, you can just keep doing that, and the will it get cheaper and go down? Like thereâs no point of using all this like custom orchestration that you get with Inngest or... âCause you, that could be another argument is that like the models keep getting better and like thereâs
Tony Holdstock-Brown:
No
Turner Novak:
Point.
Tony Holdstock-Brown:
Models keep getting better, right? But like even if youâre using something like, I donât know, that thereâs the new OpenAI model coming out. Thereâs, thereâs Fable. You run it.
You use Claude Code locally, and you can imagine Claude Code is basically a harness, which it is, and that harness is gonna do something very similar to if you were building your own harness in your product. Youâre gonna throw a prompt in. Itâs gonna generate some answer. Itâs gonna think a little bit more.
Itâs gonna call itself to get more context. Itâs gonna maybe spawn a sub-agent, which is another function call or another series of steps, and itâs more than likely going to request that you run some tools to get context about the code that youâve written. Like for example, I wanna set some stuff. I wanna read some files.
I wanna write a unit test to see if this particular bug exists, and I wanna run that unit test, and I wanna feed the results of that unit test back into my context. And no matter how good the models get, I think the concept of one shot is just never enough. One shot is never enough, Eminem. Mm-hmm.
And so... So
Turner Novak:
Even when we have AGI, itâs still gonna need to be self-improving
Tony Holdstock-Brown:
Still need self-improving... Mm... Still need contexts. Um...
âCause like fundamentally, I think itâs impossible to answer a question without more context. Like, this is the same with humans as well.
Turner Novak:
Yeah. So even if itâs... Itâs just... Like a PhD level expert on something, theyâre like, âOkay, can you like clarify the question a little bit more?
â
Tony Holdstock-Brown:
Theyâll, theyâll probably ask more questions than a dumber model, you know? Yeah. Fair. PhD level is gonna be like, âI need to know so many specifics about the question youâre asking to give you the right answer.
â
Turner Novak:
Yeah,
Tony Holdstock-Brown:
Thatâs right. And so youâre gonna have more and more loops and more and more context so that it can narrow down on the right thing and link the relevant concepts. That is true. And that happens now.
That happens with Fable. That happens with like every time models get better, Iâve noticed... Mm. Like, Iâve noticed like we just run more models.
Turner Novak:
Yeah. Which is crazy. Well, and thatâs the thing too, is like when I think of like three years ago, four years ago, you like use ChatGPT back in like, you know, December 2022, and you say... Itâs just like, it gives you an answer and itâs like kind of wrong.
Tony Holdstock-Brown:
Yeah.
Turner Novak:
And youâre just like... Yeah... Whatever. But, but now today, you know, it...
Theyâll ask like three qualifying questions and youâre like, âOh, shit, I forgot to tell you that. â Yeah. Like, yeah. Yeah, yeah.
That, that is, that is helpful just for me to give you.
Tony Holdstock-Brown:
Yeah, yeah. Yeah, exactly. Theyâre like, âOh, that was actually ambiguous and I was gonna give you a completely misleading answer, but instead I-â Yeah... âthought enough to clarify this.
â
Turner Novak:
Yeah.
Tony Holdstock-Brown:
And so the whole thinking thing that theyâve done is pretty cool because thinking will like di... Direct the model in specific ways as well, and yeah, I, I just canât see that we canât live without this sort of harness-like system, which must exist if youâre building with AI. You know, and itâs also like particularly good if youâre, if youâre not building with AI or because a bunch of people use this for deterministic workflows, just like e-commerce, insurance, all that sort of stuff as well.
Turner Novak:
Hmm. You mentioned that AWS launched... Step functions. I think youâve also had another, like, massive public company...
Tony Holdstock-Brown:
Oh, yeah
Turner Novak:
Sort of like took your product as, like, inspiration there. So you have like, how does that typically go? Like, when youâre running a startup, you guys have raised, I think the public number is like 20, $25 million bucks. I, I donât know.
Like, you have like 1 millionth the resources of... Yeah... An AWS. Like, what happens when a big public company, like one of the biggest companies in the world, is like, âOh, this is cool.
Like, weâre gonna do it too. Weâre taking your product and running with itâ? Yeah,
Tony Holdstock-Brown:
Yeah,
Turner Novak:
Yeah. The difference between that happening now and the first time it happened is
Tony Holdstock-Brown:
Huge. First time it happened when Cloudflare copied us, like, I wonât go into the process of how they copied us or why, or the communication that we had with them. But the first time they copied us, I was like pretty annoyed... Mm...
You know, personally. âCause Iâd spent a lot of time building and developing the SDKs, with the team, and the team have done a great job on what weâve built and defining the DX. And then to see somebody just like rip it off was like, really annoying, especially somebody that we work closely with. And that happened a few times.
So, on a personal level, if youâre a founder, it can be a little frustrating. But itâs also like really, really good. Now weâre really happy. When AWS copied this, copied our stuff.
Used it, I wonât say copied. When they used similar DX approaches, I think thatâs a fair way to say, it actually made us really happy. And I was talking to somebody from AWS last week at AIE conference, and they were like, âOh, like durable functions. â And I was like, âYes, which is very similar to our API that we made four years ago.
â And they were like, âI had no idea that we did that. â Mm-hmm. Like, itâs just kind of funny. Itâs nice.
Itâs validation that weâve done the right thing, and itâs validation that this DX is really, really nice for engineers. At the end of the day, like if youâre a founder though, getting caught up on what your competition does is maybe worst case. I think like weâd mentioned it before when you were like, how was Inngest in a good position to run AI to begin with? Mm-hmm.
And it turns out that because weâd thought about the engineering from first principles and the way that the SDK worked and the way that we work with particular, systems, and that you can rewrite things from Go to TypeScript, and itâll pick up where it left off. That sort of first principle thinking allowed us to build a really flexible platform that continued into AI.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And if you just think about what your competitors are doing and try and copy your competitors or poke holes in it, maybe you donât have the same first thinking principles, and maybe youâre just like not doing the right thing for your own product, really. So I donât care what our competitors do. I donât care what other people in the ecosystem do. I donât even look anymore.
Donât care. I just think about what our product can do, what are the use cases for it, what are our customers doing, and what would make that process better. And if other people copy us, like, cool, thatâs actually great. They donât know how hard it is to build these super high read, high write, high delete systems at scale.
They donât know how hard it is to, to build the infrastructure to make all of this work. They donât know all of the pitfalls that weâve ran into. They donât know the future of what weâre doing, which Iâll honestly just broadcast in podcasts like this, AKA traces, agent steering, trajectories, sessions, taking all that information so that you can do post-training on open weights models so that you can take your production data, generate your own custom models. And then run them so that you can free yourself of the dependency of frontier labs and run the same sort of AI quality for like one twentieth of the cost.
Like, all of that stuff might not be in their roadmap, and if they listen to this and it is, like, cool, good. Doesnât matter. I donât really care because we have a job to do, and weâll, weâll get it done.
Turner Novak:
Um... So, this is quite good actually. I, I quite like it. Yeah.
So, so then w... Like, whatâs so hard about doing all those things? âCause couldnât I just, like, take the transcript from this episode and, like, throw it in a cloud code and be like... You know, build this, like, no mistakes.
Yeah. Like, you know, especially the models
Tony Holdstock-Brown:
Are getting
Turner Novak:
Better.
Tony Holdstock-Brown:
Yeah, yeah. Iâm pretty sure that somebody has, you know? Iâm pretty sure that somebody out there has a system thatâs, like, Inngest for AI, built AI specifically... Yeah and also donât care.
Also donât care. Like, you, you might be able to. You might be able to do that for your own personal stuff.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
Maybe... Honestly, maybe you can have a personal version of Inngest that you spin up and you have an agent work on. And then you work on your products on the side. But the question would be, do you wanna do both?
Probably not. Because if youâre working on the infrastructure, youâre not working on your product, and that means other people that just build the product are gonna do a better job than you. Mm. So then youâve got opportunity costs, and, and youâre gonna lose when youâre building your product.
And secondly, if youâre doing that to build infrastructure, then I donât, I, I donât think it will scale. Mm. The challenges are so extreme when youâre building at the scale of, like, millions of events per second, doing so much QPS on your data stores, that just itâll get, itâll be tough. So is, is the issue that you...
I could
Turner Novak:
Technically vibe code this, but then thereâd be a lot of issues Iâd have to f... Probably, like, fix manually, and Iâd have to go in and really repair s... Like fix... Figure out how to, how to serve these millions of events per second to the point of, like, it canât just be, like, a vibe code thing.
Yeah. You need, like, m... Like a big team thatâs constantly... Yeah,
Tony Holdstock-Brown:
Yeah. You know? Basically. Basically.
I think youâve got two paths. You vibe code this for your own product, and that means youâre scratching the surface of what we give you for free. Mm-hmm. And youâre not gonna get the same self-reinforcing loop at the same speed.
Oh, so I can use you
Turner Novak:
For free? Yeah,
Tony Holdstock-Brown:
Yeah. Oh, okay. Exactly. Yeah.
So thereâs
Turner Novak:
Probably, like... In, in that case you could say, like, why would you vibe code it...
Tony Holdstock-Brown:
Exactly... When Inngest already has it built for you? Why would... Yeah, exactly.
Why would you vibe code anyway? Just use, use this for free. Secondly, like, youâre not gonna get the entire self-reinforcing loop at the s... To the same degree that we would give you for free.
Thatâs just gonna be really hard for you to do. And if you do decide to build your own infrastructure to, to compete with us, thatâs super cool, just, just very difficult. So,
Turner Novak:
You know. I think you, you still, like, grew, I think the number... Youâve grown, like, 35x. Probably.
Maybe itâs more than that.
Tony Holdstock-Brown:
More than that.
Turner Novak:
Since... Yeah, how, how much have you grown since the very first time you got copied?
Tony Holdstock-Brown:
Oh. Thousands. Like... Thousands...
Thousand. Yeah, yeah. And thatâs, like, great. At first I was worried because I was like, âOh, this, this-â Is the end of the world for us, you know?
Yeah. The sky is falling and so on. Um...
Turner Novak:
Whatâs interesting, like...
Tony Holdstock-Brown:
We kept growing...
Turner Novak:
One of the biggest companies in the world is doing the same thing as you.
Tony Holdstock-Brown:
Mm-hmm.
Turner Novak:
They... Thereâs a pretty high bar for them to, like, do something. Mm-hmm. So if they do it, it means that thereâs a really big opportunity...
Mm-hmm. Stack ranked along all... Yeah the other things that theyâre doing. Like, this was worth...
Tony Holdstock-Brown:
Yeah...
Turner Novak:
Pursuing.
Tony Holdstock-Brown:
Yeah. Totally.
Turner Novak:
So it almost, like, validates the, the
Tony Holdstock-Brown:
Market is
Turner Novak:
There.
Tony Holdstock-Brown:
It does. It does. Yeah, it does. And in order to compete with us to some degree, they can do the basics.
But itâs gonna be extremely hard for them to go deep on all of the things that we do.
Turner Novak:
Mm.
Tony Holdstock-Brown:
For example, as of yet, no one else does AB testing of different steps within your step function so that you can safely roll out changes. Nobody else does tracking the outcomes of those particular changes. Nobody else has things like deferring so that you can run cleanups or do sagas or score based off of outcomes of that particular agent run. People are, like, still relatively far behind...
Mm... But itâs, like, total validation that weâre doing the right thing, which is cool.
Turner Novak:
I think one thing thatâs super interesting about, like, Tony the person I... Especially being a dev tool, like infrastructure founder, you like just donât have a personal brand. Like, most people... Like, itâs like, âHey, Iâm starting a dev tools product.
Like, I need to start building up my Twitter persona... Yeah... Or something to, like, start getting customers. â So why have you not done it?
Because itâs, like, the most common... The most common thing... Like, you should, you should do it. Like, you...
The most common instantly you should be doing this. You and other investors of mine tell me that I should do this every day. I mean, Iâm like, whatever you wanna do. Yeah, yeah, I appreciate you.
Whatever your, whatever your strength is... Thatâs what you should do. But I feel like probably every day people are like, âOh, when are you gonna start tweeting? â All the time.
All
Tony Holdstock-Brown:
The time. Yeah, yeah. And when I tell people that I donât use social media, really, their initial reaction is, âThatâs insane. Thatâs crazy.
Why? You should do this. â Yeah. So...
I am, I, I have pretty strong views on social media from before I started my company, that it just wasnât good for people
Turner Novak:
As in like a, the addiction, the health benefit theory
Tony Holdstock-Brown:
The addiction, the health, the comparison, that sort of stuff.
Turner Novak:
Mm.
Tony Holdstock-Brown:
I, I much preferred talking to my friends to catch up with what theyâve done than look at their IG stories... Mm... Overall, back when I had IG. And I guess thereâs still technically a profile that Iâve locked myself out with two-factor, that I would like Facebook to delete.
So if youâre listening, please delete it.
Turner Novak:
Oh, like theyâre trying to force you to open your Instagram to log into things?
Tony Holdstock-Brown:
Two, I could... Is that two-factor?... I could technically ask them to recover my two-factor. Um...
Oh, yeah. Got it... But any... Like, besides that, like I just, I much prefer talking to people and catching up, and I just viewed social media as, as sort of generally bad for peopleâs mental health, and I didnât like it.
Turner Novak:
Mm.
Tony Holdstock-Brown:
And I also viewed things as like sort of, um... Thereâs some really, really good social media. Thereâs some really, really good things you can learn, and thatâs really cool. I appreciate that from a bunch of people, but because of my previous views, I just didnât really wanna do it.
Mm. And also, like I much prefer learning, building, doing, than I do talking about that sort of stuff...
Turner Novak:
Mm
Tony Holdstock-Brown:
You know? So like we have this entire FoundationDB backend, which is super sick, and itâs actually up to the engineers to talk about, and weâll probably talk about this in detail about how we built this FoundationDB backend. And I much prefer talking about that rather than how we did it and why. But I know that thatâs important, and this is what other people in the team are for.
Mm. So Iâm really grateful that they do it. I just spend like all of my effort on what to build, why, how, where weâre going, and making sure that we do the right thing, and my co-founder, Dan, is also way better at tweeting some banger tweets than I am,
Turner Novak:
You know? Yeah, I was gonna say, like, thereâs like probably three other people on the team I can think of off the top of my head that like have a bigger social, personal brand...
Tony Holdstock-Brown:
Yeah.
Turner Novak:
Yeah, yeah. For sure... Persona than you. Yeah
Tony Holdstock-Brown:
For sure. Yeah. But then like I, I sort of know that itâs partly required. Like you take a look at G from Vercel, and heâs just like,
Turner Novak:
Just lives on Twitter.
Tony Holdstock-Brown:
Craziness. Yeah. And thatâs cool. Then you take a look at Ali from Databricks.
I donât even know if heâs got Twitter. Maybe he does. He probably tweets a little bit, but I havenât seen him. Also...
Iâve seen
Turner Novak:
Some tweets. Yeah
Tony Holdstock-Brown:
Youâve seen some tweets?
Turner Novak:
But
Tony Holdstock-Brown:
Maybe.
Turner Novak:
Yeah, Iâve seen... Not a lot.
Tony Holdstock-Brown:
Yeah, not a lot. Not a lot. Not
Turner Novak:
To the level of
Tony Holdstock-Brown:
Like G... But theyâre probably written by someone else. Yeah, yeah, exactly. Exactly.
And so like, yeah, thereâs a die cost me. We probably need to do it. My investors will ask me to do more. I will probably give my Twitter handle to some folks in the team so that they can do stuff, on my behalf, which sucks, but probably.
Or they will eventually convince me, and I will have time to do it. Mm. But as it stands, like thereâs so much to do. And we already have enough growth that like that sort of distribution of a personal founder level just isnât like was on top of mind.
Turner Novak:
Mm.
Tony Holdstock-Brown:
Like it was doing the thing. Yeah. I much, much, much preferred doing the thing.
Turner Novak:
So if the primary You know, GTM strategy for dev tools is like Twitter. Yeah. You donât have it. Yeah.
What, like, what, what did you do? How did you approach just, like, getting people to use the product if you werenât doing that?
Tony Holdstock-Brown:
Yeah. Oh, man. Wow. So way back in the start, we noticed this missing piece, AKA queues could not run on serverless.
You couldnât use Kafka in a serverless function. The only way would be to create this jerry-rigged SQS, SNS Lambda type deal, which is extremely ugly, very hard to build, zero local testing. Itâs like pure, acronyms, the thing that you just
Turner Novak:
Described. Right, right. Which tells
Tony Holdstock-Brown:
Me itâs like an insane setup. Itâs like, itâs like an incantation, you know? Yeah. Like youâre casting some spell, but youâre doing it with like crazy code and terraform.
Itâs just like awful. Itâs the worst thing. And I, I remember like people trying to do that, and people had these like crazy architecture diagrams because youâre an AWS consultant. Mm-hmm.
And I was like, âThese people are fucking insane. This is nuts. â
Turner Novak:
So itâs AWS consultants coming to people and like explaining how to do this. Basically, yeah. Mm-hmm. You need to know the infrastructure very
Tony Holdstock-Brown:
Deeply in order to tie these specific things together, and itâs just like the worst thing. And so, that realization was like, well, thereâs a wedge here. We can build some sort of event-driven queue, some sort of step function, and we can make it run... Mm-hmm.
On serverless, and thatâd be sick because then we give it to everybody.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And serverless is like both really good, also the lowest common denominator of running functions, you know. Hmm. Like stateless, kind of crappy short-lived functions, but super good at scaling and zero DevOps. And thatâs cool because thereâs this unmet need of people trying to ship stuff fast.
Sometimes YOLO out in API endpoints, and we give them durability out the box. And meeting that unmet need was good because then organically we would grow. Weâd talk about it. Weâd, like, work with different frameworks that people could, could, could write their code in.
Weâd work with different cloud platforms that people could deploy to. And... And you discover it through the, like, marketplace on... Yeah...
Like AWS or? So itâs mostly zero integration, zero marketplace. People would just find out about it. Weâd, like, maybe do some, like, guerrilla marketing, and weâd, like, talk about it on Reddit or Twitter or stuff like that, like my co-founder would.
But we would just, like, grow, which is cool. And then, yeah, partnerships, marketplace is a good, um... Eventually, like, the idea was just so obvious that people realized this is something they need to do. And then we do lots of blogs, lots of content.
Mm-hmm. I think, like, one underrated thing in every GTM strategy, or not e-even underrated, everyone knows, just content. Write content. Write content...
Yeah... For blogs, write content for SEO, write content for AI to, to scrape up. Like, just content, all the things. So you
Turner Novak:
Have to do all of it. Uh,
Tony Holdstock-Brown:
Yeah.
Turner Novak:
Itâs
Tony Holdstock-Brown:
Not just AI, SEO. Itâs like... Yeah, yeah. Mm.
Blogs are so, so good. Just write more blogs. Everyone should write a blog.
Turner Novak:
Mm.
Tony Holdstock-Brown:
âcause thereâs so much useful information, and thereâs so much information about why you have solved a particular problem a specific way, which is really int-interesting. Mm. And so, we, we did a ton of that, and we still do. We have so much to write.
Thereâs so much good stuff that weâve learned in the company...
Turner Novak:
Mm...
Tony Holdstock-Brown:
Infrastructure-wise that would be really cool to talk about.
Turner Novak:
So how do you decide whatâs worth, talking about? Like, if Iâm listening to this as a founder... Talk about everything Iâm like, how do I come up with ideas? âCause, like, a lot of people, they donât, they donât know what to say.
Tony Holdstock-Brown:
Yeah. Talk about, talk about everything. Like, literally everything... Oh, okay...
And then see what works. Thatâs the best thing. Talk about everything from an infrastructure point of view, to how you built it, to what you can do, to case studies, to examples. Just talk about everything and see what sticks.
Like, thatâs the only way, really.
Turner Novak:
Mm. â
Tony Holdstock-Brown:
Cause your audience, every audience is different. Yeah. You could build an exact replica of our company, but maybe you target a different demographic. Mm.
And maybe that means you canât talk about infrastructure. Maybe that means you need to talk about really, really, really basic things because theyâre not necessarily the most technical people.
Turner Novak:
Mm.
Tony Holdstock-Brown:
And thatâs okay. So just talk about everything.
Turner Novak:
Yeah. And you, I know one book, itâs called Traction, that you got a lot out of, of... Yes. Like, what, like, so what did you learn from that?
Tony Holdstock-Brown:
Oh, yeah, the DDG founder, I canât remember his name now, wrote this book called Traction, and itâs basically a, an experimentation framework where it talks about different GTM strategies. Hey, like, which is very similar. Just write about everything and see what sticks.
Turner Novak:
Mm.
Tony Holdstock-Brown:
Do a bunch of different GTM strategies like talk, Twitter, socials, blog posts, webinars. Pick a few, test them over the course of a few weeks, see if they work, see if they resonate, and if they do, keep on doing it. And if they donât, drop it temporarily. Donât drop it forever.
Yeah. Drop it temporarily. Come back to it at some point in the future because...
Turner Novak:
So itâs almost like running loops of, like, your strategy.
Tony Holdstock-Brown:
Loops of marketing. Yeah, yeah. Loops for everything. Oh.
So yeah. My whole Twitter stuff is like, I mean But itâs good. Itâs noisy, but itâs good. So I can understand why individual DevTools founders would prefer a presence on there.
Totally get it.
Turner Novak:
Yeah.
Tony Holdstock-Brown:
And you, you do get pretty quick
Turner Novak:
Validation of, like, letâs say you get a million views on your thing and, like, 100 replies of people discussing. Like, itâs pretty hard to argue that that didnât... Totally really quickly have a tangible...
Tony Holdstock-Brown:
Totally...
Turner Novak:
You know, value provided to your product or your brand... Totally. Yeah, totally... Or whatever itâs like.
Totally. Itâs, itâs there. Totally. Versus, like, if you do a webinar, someone might convert two months later.
Tony Holdstock-Brown:
Yeah, totally.
Turner Novak:
Versus on Twitter, you, like, see it. Just
Tony Holdstock-Brown:
Spam all the stuff as fast as possible. Yeah, totally. And also, like, thereâs the whole 1% contributes to 99% lurker type deal, you know? Yeah.
So you, like, you get 100 people looking. But thereâs thousands of people that have read that that are thinking something, either good or bad, about your product and what youâve said. And so thatâs really, really good. So I can totally see the value.
I can totally see why people would like to do that. Maybe, maybe I do tweet. Maybe, maybe...
Turner Novak:
Maybe you should.
Tony Holdstock-Brown:
Maybe I should. Maybe I should get on there and craft.
Turner Novak:
I mean, âcause, like, the interesting thing is, like, you probably... Thereâs probably a lot of things that you, like, write in Slack to the team or email. Yeah. Like, you explain something, some new framework.
And, like, you make a blog post. Yeah, yeah. And you put it on the blog, and then, you know, it just sits on the blog. Yeah.
Versus you could literally copy and paste the chunk of it... That
Tony Holdstock-Brown:
Is good
Turner Novak:
Advice... Put it in a tweet, and just tweet it. Yeah, yeah. Like, itâs the same thing.
It literally took you an extra minute maybe. Yeah, yeah,
Tony Holdstock-Brown:
Yeah.
Turner Novak:
Versus you already put an hour... Yeah... Maybe five hours... Yeah...
Or whatever into writing this.
Tony Holdstock-Brown:
Exactly,
Turner Novak:
Exactly. So I donât know. Thatâs, thatâs how I tell people is a good way to just get started if youâre kind of like, âI donât have time to do this. â Yeah.
Itâs like, well, you had time to write this two hours of, of thing that you made. Yeah. Like, it doesnât need that much... Yeah...
More... Yeah... To, to get it out there.
Tony Holdstock-Brown:
Yeah, true, true.
Turner Novak:
So.
Tony Holdstock-Brown:
Yeah, yeah.
Turner Novak:
And then itâs like, whatâs the upside? Whatâs the downside? Like, downside is just, like, no one read it, whatever. Like...
Whatever you didnât spend that much time on it. You already made the thing.
Tony Holdstock-Brown:
Yeah,
Turner Novak:
Yeah. And then the upside is, I donât know, like, you know, Jeff Bezos, âcause he uses Twitter, comes across it... Reads it, and heâs like, âOh, Inngest, this is pretty cool. Like, weâll use it for Prometheus, our new AI thing that we just raised, you know, \$12 billion for.
â And, like, you know, youâre a board-level vendor for this... Yeah, yeah... For Jeff Bezosâ company. Yeah.
And, you know, theyâre gonna pay you, you know, \$100 million to be a customer. Yeah, yeah. Like, thatâs pretty good outcome... Mm-hmm.
From literally copy and pasting a Slack message and putting it in the... Copy and paste on Twitter.
Tony Holdstock-Brown:
You know what? Maybe
Turner Novak:
Iâll just
Tony Holdstock-Brown:
Get an agent to write my tweets for me in a loop. That would be it. Just create some banging tweets.
Turner Novak:
I mean, the sad part is, like, a lot of people,
Tony Holdstock-Brown:
A lot of people do use AI. Dude, itâs just banging slop, man. Itâs slop. So.
Itâs slop everywhere.
Turner Novak:
Yeah. And I, I think, like, the Then you just need to make sure is youâre, youâre using it for, like, idea generation... Mm-hmm. But not necessarily Like the whole thing...
Tony Holdstock-Brown:
Yeah, totally... Is
Turner Novak:
AI.
Tony Holdstock-Brown:
Totally. There was this really, really, really good post from somebody at Microsoft, maybe the director of AI at Microsoft. Also canât remember their name. Hmm.
He was talking about self-reinforcing agents, learning feedback loops, and so on. Really, really, really good tweet. Hmm. And it contained a lot of the same principles that we built Inngest on, which is observability, determinism, reinforcement, based off of the production trajectories using that particular framework to run agents.
Super good. Everyone loved it. It was like huge tweet. Mm-hmm.
And I copied and pasted it, well, the link to the tweet in Slack, because I was like, âThis is exactly what weâve already said we were working on. â Hmm. âThis is complete validation of our entire system. â
Turner Novak:
Hmm. â
Tony Holdstock-Brown:
And so, like we should, we should just do the same thing. We should do the same thing. â
Turner Novak:
And that happens a lot, I feel like, where, you know, you... I mean, âcause that was probably like some kind of memo that was written internally...
Tony Holdstock-Brown:
Yeah...
Turner Novak:
Or something. Yeah. And then they posted it publicly. And if, I mean, if itâs good, like people will share it...
Yeah... And use it. Yeah. So itâs like youâre almost like really kneecapping yourself by not...
Tony Holdstock-Brown:
Mm-hmm.
Turner Novak:
Sharing. I mean, you wanna be careful, like, âHereâs our roadmap. â Ah. Hereâs like maybe you donât give a shit.
I donât give, I donât care. Like maybe it doesnât matter.
Tony Holdstock-Brown:
I donât care.
Turner Novak:
Yeah. But like thereâs, thereâs definitely like a, thereâs definitely like a you donât wanna share everything, but like... Sure... Thereâs a lot you probably could.
Tony Holdstock-Brown:
Yeah, for sure. For sure. And again, like we mentioned previously that all roads are converging. Many infrastructure companies are looking very similar.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
You know? And because of that convergence, the roadmap is fairly easy to predict.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
You know? Like compute is a thing. Running Inngest functions it f... Is a thing because itâs bananas that right now you have to choose to host Inngest functions on some other provider.
Weâll just let you do that for cheaper than other providers because we already run the bare metal.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And we can do better sandbox DX because itâs integrated into the durable execution framework inside your already existing harness. So like the roadmap is, itâs pretty easy. Itâs pretty easy for people to, to, to guess where weâre going. I think itâs pretty difficult for people to understand the nuance of what we do.
Like for example, I think it wouldâve been pretty difficult for people to have guessed that we were gonna release something that allows you to score 100% of your production agent runs using product events. But thatâs, thatâs, thatâs like something thatâs so special and unique to us that nobody else could have thought to do that anyway.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And weâre like very unique in that only we can give that to everyone. I donât know. Youâre right. I donât care if people know.
Yeah. You know, weâre, weâre all, weâre all working on very similar problems at this point. Mm-hmm. And a lot of the stuff that weâve done right now and that, that, that weâre working on is also ground thatâs been trodden on before.
You know, whether or not you use Firecracker or use cloud hypervisor to provision your VMs, itâs all the same stuff that people have been doing for 10 years.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And a lot of the infrastructure that already exists has been paved by a bunch of other prior companies and clouds doing the same stuff in open source. Hmm. So like doesnât matter. Everyone right now is doing microVMs because microVMs for sandbox is like the, the new harness.
Mm-hmm. So itâs all the same.
Turner Novak:
So it sounds like if Iâm a customer trying to decide who I should go with, the reason I would consider Inngest or the reason that I, that I should go with you guys is if I like appreciate that like you will launch new features that... Make my life easier... Yeah,
Tony Holdstock-Brown:
So... And make
Turner Novak:
Me, save me money, make
Tony Holdstock-Brown:
My product better our, our entire thing is, like, allow you to safely build reliable AI or good products... Hmm... Without worrying about infrastructure. Hmm.
And that could be without worrying about compute, without worrying about queues, events, observability...
Turner Novak:
Hmm...
Tony Holdstock-Brown:
Tracking that everything worked correctly, and making sure that your product does the right thing, and you can do that using a few lines of code. Hmm. Everything else is completely abstracted. That means you can focus on what specifically you need to do instead of building anything else.
Zero infrastructure required. Hmm. Thatâs our entire anti-infra, infra thing. Hmm.
Like, just donât worry about the infrastructure at all.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
What are some of the values you guys have as a company? Hmm. Yeah. Firstly, truth.
I think, like, have you, have you read Principles by Ray Dalio?
Turner Novak:
Oh, no, Iâve not read it.
Tony Holdstock-Brown:
Oh, cool.
Turner Novak:
Really dry, really dry book.
Tony Holdstock-Brown:
Okay. But really good. Okay. So, like, both good and bad at the same time.
In it he talks about a few different things, and heâs got a few principles, and one of them is truth, and I... Hmm really, really agree with this one.
Turner Novak:
So why truth?
Tony Holdstock-Brown:
If youâre building the wrong thing and you donât respect the truth, and the people are telling you that itâs wrong and theyâre not using it correctly, then youâre gonna be misguided and youâre gonna continue down the wrong path. Hmm. If you built the right thing but the world changes around you with AI, and you donât understand and appreciate the truth that the world has changed around you and you continue down that path, youâre inevitably gonna be building the wrong thing, because the truth of the situation is that a lot of stuff has changed... Hmm and you need to change with the times.
And so if you can understand the truth, are we building the right thing? Are we on the right path? Do our users appreciate what we are doing, or is it actually wrong? Then we can basically get to the, to the, as close to the right answer as possible.
Hmm. And it also keeps us from, like... It keeps us, in some ways, like, free of ego, from thinking like, âOh, we had durable execution and we built this step function SDK, and the APIs that everybody else copied, and we must, we, we... This must be correct, âcause everyone else has copied us.
â Like, if we just agree that finding the right thing, finding truth is the right thing to do, then, like, turns out in five yearsâ time thatâs wrong...
Turner Novak:
Hmm
Tony Holdstock-Brown:
Then, then no one is too attached in the way that weâve built things. Hmm. Weâre really okay to change everything that weâve ever done... Hmm...
Because weâve learned more information. So truth is, like, super important. Context, openness, is also super, super important, important for us as well, like context, nuance, that sort of stuff of... Hmm of what weâre doing.
But yeah, I think, like, truth is such a fundamental thing that if you avoid something thatâs not true, it inevitably comes back to, to screw you over. Has that happened to you before? Probably accidentally.
Turner Novak:
Mm.
Tony Holdstock-Brown:
Never deliberately. Weâve never avoided something because it was true by thinking like, âOh, we know best. â I canât necessarily think of anything off the top of my head. I was gonna try and make something up around like AI.
Turner Novak:
But like, we were like, âOh, shit, this is pretty cool. We should just, just to do this. â Well, did, did
Tony Holdstock-Brown:
You...
Turner Novak:
Did, did it take you a
Tony Holdstock-Brown:
Like... And probably took us too long... A couple weeks? Yeah, it probably took us too long to really adapt to the whole AI thing.
Turner Novak:
So assuming like the day ChatGPT came out, you shouldâve like instantly gone.
Tony Holdstock-Brown:
Yeah,
Turner Novak:
That wouldâve been great. So, so why donât you think you did? Like, why did you not instantly change everything?
Tony Holdstock-Brown:
I mean, like sometimes itâs hard to know whatâs a fad or whatâs real, you know? Like, sometimes... Yeah... Itâs very hard to predict the future.
And so you, youâre forced to make bets when you run a company, like youâre forced to make bets on, on the complete direction of the company. Are we gonna go all in on AI, or are we not? Are we gonna go semi-in on AI? And you could argue that like, you know, one is better than the other, and thatâs very true.
One is always better than the other, but you never really know. And so we had a bunch of people start using us for AI, and it turns out that that was really, really good. And so we became much more AI first. Hence, all the eval stuff that we just released.
Turner Novak:
Mm.
Tony Holdstock-Brown:
But at the same time, being general purpose infrastructure, thereâs this big component of ours that you can use this to build whatever product you want. And that dichotomy is like hard to toe the line on. But foundationally, I think, I think we, yeah, probably couldâve done it faster.
Turner Novak:
Mm. Well, I feel like we were just coming off literally like a month prior, FTX had failed, and we had gone through this whole like Web3... Yeah... Like...
Yeah... Era. I donât know... Exactly.
Yeah, it was pretty cool... Where it was like, you know, every podcast you listen to, itâs gonna like mint an NFT that like other people might wanna trade your, your podcast, you know, listened NFTs. Forget about NFTs. Right?
Forget about NFTs. Like,
Tony Holdstock-Brown:
I mean... The most ridiculous thing.
Turner Novak:
Yeah, it was, it was crazy. And like, it was all anyone was talking about. Like, NFTs are the future of like the economy, and youâre just like, âWhat? â So then like this new thing comes up...
Yeah... And youâre like, âI donât know. Iâm c... Yeah like, whatever.
You guys just... Yeah... You guys just told me that like... Yeah...
The whole world was gonna be on the blockchain... Yeah... And that was like wrong.
Tony Holdstock-Brown:
Yeah, yeah.
Turner Novak:
And then now, like thereâs this new AI thing. I donât know. Itâs just like you guys are just crazy.
Tony Holdstock-Brown:
Yeah.
Turner Novak:
I feel like, personally, I probably got a little bit like thrown off by that initially. Yeah,
Tony Holdstock-Brown:
Yeah.
Turner Novak:
Like where... But I feel like itâs sort of how the industry works, where like you go all in on things. Um... Mm-hmm.
And you see if it works, and if not, like... Yeah, which is like good and bad. Like, when you think about like the reason like a startup can be successful is like a startup raises \$20 million. Like, all of that capital is basically R&D.
Mm-hmm. Theyâre like a tax problem. Mm-hmm. When you think of like there might be a competitor youâre competing with...
Yeah... That has a billion dollars in revenue or something like that, but then what do they actually spend on true R&D?
Tony Holdstock-Brown:
Mm-hmm.
Turner Novak:
Like, like deep tech, quote unquote, like very much like high risk, you know, capital is being work, being put to play. It might be zero. Yeah. Like, they might not actually be doing any R&D.
Tony Holdstock-Brown:
Yeah, yeah,
Turner Novak:
Yeah. So like as a startup, your point, the reason you exist is to like go after this like pretty... Highly risky, probably potentially wonât work. Like, youâre going on an, an adventure.
Like, itâs like adventure capital kind of thing... Yeah where itâs like weâre on an adventure, and like... Yeah... Itâs like this really crazy thing that weâre trying to, like, fix...
Yeah... And solve this problem.
Tony Holdstock-Brown:
Yeah,
Turner Novak:
Yeah. And so I donât know, itâs like, it kind of should be a little bit crazy.
Tony Holdstock-Brown:
It should be a little crazy. Like... Yeah, totally. I think, like, also the infrastructure that we built just didnât exist before we built it, you know?
Turner Novak:
Hmm.
Tony Holdstock-Brown:
And whether or not it was for AI or not, itâs foundationally the same infrastructure. We never really pivoted. Hmm. We never really did anything different.
It was always that infrastructure, but it just straight up didnât really exist in the way that we built it... Hmm... Which is cool. And it turns out that thatâs actually really, really effective and will help people build this new wave of products that must exist with AI.
Yeah. And so, like, yeah, really, really, really fortunate that, that thatâs the case.
Turner Novak:
Yeah.
Tony Holdstock-Brown:
But yeah, everything is, everythingâs a gamble.
Turner Novak:
Well, and even too, when you think of, like, Nvidia, Jensen didnât start it in 1993 saying, like, âLLM, we, we, like, need to build, like, the infrastructure. â I was like, AI, like, I donât know if it was a w... It was barely a word. Yeah, yeah.
Like, artificial intelligence was, like, science fiction. Yeah. And it was basically, like, gaming graphics cards or whatever.
Tony Holdstock-Brown:
Gaming graphics cards, numbers, matrix mu... Multiplications.
Turner Novak:
Yeah, and e... Itâs just
Tony Holdstock-Brown:
Like...
Turner Novak:
And even then, when you think of back in, like, 2021... Mm-hmm. I mean, it was basically, like, a crypto company. Majority of Nvidiaâs...
Yeah... Revenue was, like, Bitcoin mining.
Tony Holdstock-Brown:
Yeah, yeah. CUDA, man, CUDA, the best bet Nvidia ever made. CUDA is the best.
Turner Novak:
So what is CUDA for somebody who doesnât know? Thatâs, like, their software
Tony Holdstock-Brown:
Platform. Yeah, yeah. So, like, how
Turner Novak:
Do you describe CUDA to people?
Tony Holdstock-Brown:
Not very well. Not very well... âcause I donât do too much CUDA.
Turner Novak:
Okay.
Tony Holdstock-Brown:
Well, programming language to run things on your, on your Nvidia, kind of on your Nvidia graphics cards.
Turner Novak:
And it basically locks you in. Yeah, thatâs pretty cool. Yeah, yeah. Do you have a favorite, like, CEO or founder or company just, like, throughout history, whether itâs, like...
Ooh still around or, like, currently operating? Like, who do you... Oh... Do you have anyone that youâve gotten a lot of inspiration from?
Tony Holdstock-Brown:
I guess the reason I brought up Ali from Databricks is I really rate Ali from Databricks overall. I think Ali from Databricks is, uh... I donât know why I call him Ali from Databricks. I could just call him Ali from now on.
I think that Ali from Databricks is great. Heâs great. Heâs a, heâs a really, really good person. Heâs really, really ruthless but good at what he does.
If you take a look at the team that heâs got around him, everyone has sticked for such a long time, which must speak to a lot of, his work and the way that he operates his company. Donât they have seven co-founders?
Turner Novak:
They have a lot of co-founders, right?
Tony Holdstock-Brown:
Yeah, itâs, itâs, itâs an interesting company, for sure. Um... Itâs
Turner Novak:
A lot of academics. Yeah, yeah. Itâs all academic co-founders, which you donât... Yeah...
See as much.
Tony Holdstock-Brown:
Yeah. And then I, I, I think that Ali is, in general with that company is, is, is interesting and cool. Hmm. So, theyâve, theyâve done a, a really, really, really good job.
And also, like, interestingly, theyâre not, like, s... So big on Twitter, like you were saying. You know, theyâve done... Yeah...
A lot of really, really, really good things. And they built this really interesting technology thatâs pretty cool. And the way they operate is, is quite good as a company. So I, I, I really respect that.
I really respect that. I really like the way that Cloudflare was built. Copying and that entire thing aside, itâs an interesting way to attack the problem. And then basically get all this leverage by running so much bandwidth through all of your pops that...
Mm that you can do a ton. So I think thatâs particularly es... Interesting from, like, just, just a company building perspective, which is cool. Personal favorite, like I, I also love the folks from both PlanetScale and Railway.
You know, Sam is great, Jake is great. Theyâre both really, really, really good people as well. So huge fans of them personally as people, as well as their companies.
Turner Novak:
Hmm.
Tony Holdstock-Brown:
Yeah.
Turner Novak:
Do you have a favorite, like, new AI tool that you use? Or like what does your, like, stack look like, I guess? Oh,
Tony Holdstock-Brown:
Man. All right. So, coding, I was the biggest holdout in the company. I was just, like, writing manual code myself in Neovim, and I still use Neovim for everything.
Turner Novak:
So you donât, you donât use AI to code?
Tony Holdstock-Brown:
Iâve started using a little bit of AI. Iâve started using a little bit of Codex and Claude, but I manually review every single chunk that comes out of them like a, like a crazy person. Um... Why do you
Turner Novak:
Do that?
Tony Holdstock-Brown:
I... Every time I look at it Iâm like, âSomething was wrong, we can simplify this. â Not every time. Like 60, 40, 60% of the time Iâm like, âLetâs change this, and letâs improve the way that this particular thing works.
â sometimes itâs pretty good. Sometimes I donât care about the problem, like front end, for some internal tool, and Iâm like... But if itâs in our code base, Iâm, like, reviewing each chunk, manually. Uh...
You gotta be training the code base on your changes. Yeah, yeah.
Turner Novak:
Like, training the AI
Tony Holdstock-Brown:
On
Turner Novak:
What you changed.
Tony Holdstock-Brown:
Yeah, exactly. Exactly. Other than that, I think, like, Iâve just started to use, voice-to-text... Hmm...
Especially for PRDs, especially for communication about what we do, and our vision... Yeah and why things need to exist. Mm. Context around the company, context around what weâre building and why, and I YOLO voice-to-text in Notion like a crazy man.
Itâs the best.
Turner Novak:
Do you, do you get it to where it will, like... Youâll just talk for 10 minutes, then it will, like, just succinctly rephrase what you said, or?
Tony Holdstock-Brown:
I, I just... I, I do paragraph by paragraph in text. Okay. And I have a, have a hotkey combo that will start.
Iâll speak maybe, like, eight sentences, and Iâll stop, and then Iâll do a quick check to make sure they look good, make sure... Mm... It reads okay, and then Iâll continue.
Turner Novak:
Mm.
Tony Holdstock-Brown:
So, thatâs actually pretty good.
Turner Novak:
Um... So do you use Notionâs built-in AI?
Tony Holdstock-Brown:
No, no. I use something on the Mac. I use local models... Mm...
On the Mac for that.
Turner Novak:
Mm.
Tony Holdstock-Brown:
I think Nvidia has this Parakeet model, which is pretty cool, so I use that with some, some, some local stuff to make it work. Itâs pretty good.
Turner Novak:
Mm.
Tony Holdstock-Brown:
And thatâs not because of our new principle. I just was mucking around, and I was like, âThis is cool. â So it just stuck, you know? Mm.
I think, like, there are some really, really good tools that if I were back 100% engineering full-time, I would love to use, you know? I would love to use more, sort of like reinforcement loops. I would love to use more feedback from errors or stack traces or production stuff directly into the code base, and Iâd like to have this entire setup that would be Great to use with multiple agents that I know some people in the company have. But Iâm, s...
Unable to do that day-to-day unfortunately.
Turner Novak:
Is that âcause youâre just on calls with customers, recruiting
Tony Holdstock-Brown:
People... Calls
Turner Novak:
With
Tony Holdstock-Brown:
Customers... Like... Yeah, calls with customers, talking to people, talking to marketing, talking to sales, talking to product, taking a look at the bare metal builds that we have and infrastructure stuff. Just like day-to-day so much changes.
Mm. And so, I wish I were more knee-deep in stuff, you know... Mm... âcause the world has changed so much, and I look at it, Iâm like, âThis is
Turner Novak:
Greatness. â Yeah. How do you, how do you stay on top of it then if youâre, like, just knowing what direction to go with, with how fast things are moving?
Tony Holdstock-Brown:
Yeah. So even if Iâm not building the product day-to-day, Iâm still pretty close to what happens. Well, Iâm like super close to what happens in our product, what we need to build, why. Iâm really close to our usersâ feedback that we get.
Iâm really close to what our users are doing, and why, theyâre doing it in specific ways. Also, what other things theyâre doing and why. And also, like, pretty plainly thinking about the problems we have in the company, and how we can improve overall as a company and where we need to be. Mm.
And also, like, just looking at the trajectory of things in the past year, and what that implies for the next year... Mm... So that we can continue to build for where things will be in 6 months and 12 monthsâ time. Mm.
Otherwise we, presumably might be doing something wrong.
Turner Novak:
Mm.
Tony Holdstock-Brown:
So all of this means that, like, often, the, the work that Iâm doing isnât like day-to-day engineering, even though I do that sometimes. Itâs a lot of, a lot of that context, to, to propel us in the right direction. And thatâs like honestly more frequent than ever because so much changes so consistently with AI. Mm.
But think, like, it would be, it would be much slower, and the world was moving at a much slower pace even five years ago than it is now...
Turner Novak:
Mm
Tony Holdstock-Brown:
In the tech world at least. Yeah. And so, you need to really keep on top of things and think about where youâre going and how you adapt to the world, which comes back to that truth principle that we have. Mm.
Turner Novak:
In terms of, like, building observability into the product... Thereâs this thing called evals we hit on a little bit. Can you r... Maybe really quick give us a slightly more in-depth explanation of kinda how that works, and w...
Why you think itâs kinda crazy the way people do it?
Tony Holdstock-Brown:
Yeah, okay, cool. This is maybe a hot take. I think the way that we do evals is absolutely batshit insane. Not in that itâs terrible.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
I think itâs, like, a pretty good first step. Yep. But I also think itâs absolutely craziness, in that youâre basically asking LLM, âDid you do the right thing? â Youâre asking the criminal, did they commit the crime, you know?
Yeah. And thatâs also super expensive with the context you need to pass in. If youâve got this agent trajectory thatâs taken, like, 10 sub-agents and 100 steps, and youâre passing all that context to say, âDid it get the right answer? â Itâs, like, hella expensive.
Mm-hmm. And so agent evals are essentially unit tests over input. Did it give me the right output? And thatâs cool.
You can do that both in code programmatically, and you can say, like, âI expect the output to be, you know, 95 cents given this particular input,â and you can do that using LLM as a judge, which is the insane part. But also, I can totally understand why itâs necessary.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
All of these things mean that itâs really hard for you to ch... Test whether or not AI is doing the right thing in production because LLM as a judge is insanely expensive. Youâve gotta put all of that context in to another LLM to ask it to evaluate whether or not it thinks the previous calls were correct. Mm-hmm.
And thatâs crazy because token costs are really expensive. Thatâs slow, and thatâs also really expensive. So most people are not doing production evals, and if they are, theyâre doing it on a sample of their production stuff, and if not, theyâre using humans to review some of their agent trajectories by doing sampling too.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
And the idea of not knowing what your crazy black box thatâs cu... Super non-deterministic is doing and whether or not itâs doing the right thing in production is insane. And so our views on things were, like, specifically how can we make this as close to deterministic as possible? How can we, how can we make this work for every agent run in production?
And that means using product signals, which is why we ended up building the whole product signal part of our eval, alongside allowing you to do LLM as a judge when you think itâs necessary. Like for example, the code review thing.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
If you have a code reviewing agent and you wait for the PR to be rejected, you might want to use a product signal to wait for that rejection. And then use LLM as a judge to rate the feedback that somebody gave when they rejected that particular PR. Cool. All makes sense.
But youâre using product signals to derive whether or not you end up asking AI for more information about that rejection.
Turner Novak:
Mm-hmm.
Tony Holdstock-Brown:
And that way you can basically sample 100% of your production tra-trajectories and use cases. You can get all of these product signals, which are basically free because events are cheap, and weâve been doing that for decades. And then you can sparingly use the LLM as a judge to get more information and score agents in greater detail when you think itâs necessary, rather than doing it at the sampled rate, which is super expensive. And you can make your LLM as a judge much more specific to give you much more detail about whether or not it did the right thing overall, which is super cool.
So overall, I think that LLM as a judge is necessary. But I think that we can improve the way that we track production AI and that we must do that if we want to tr-track every agent trajectory and then use that to build self-reinforcing agents that get better because that dataset must be good for you to take that and do post-training. I think thatâs really hard for people right now in general, you know? And, and thatâs not to say that every eval company is wrong.
Thatâs to say that the way that we do it is a superset of the way that other observability companies do it because we can do the same unit testing and eval stuff, checking the outputs, plus listen to product events and so on, which, which gives you a lot more capability.
Turner Novak:
Well, cool. This has been a lot of fun. Thanks for, thanks for coming on the show.
Tony Holdstock-Brown:
Thanks for having me. Itâs been really, really great to talk.
Find transcripts of all other episodes here.
