I’m going to anchor this on one thing. The Flight Data Recorder. It’s there for a very critical reason on every commercial plane you fly. And just like planes inflight, your agents operating in the wild need a flight data recorder.
AI observability is not simply viewing from afar. It’s not tower observation from the taxiway, loose communications scattered across radio channels, or a rudimentary transponder signal telling you little more than identity, position and altitude. It’s certainly not an engineer parked in the cockpit with a collection of bespoke instruments.
The AI industry is tossing out words like telemetry, observability, and transparency all the while yelling about how much it hates the black box of AI.
In the old days of AI, you really didn’t need a flight data recorder. And I’m not talking about AI a decade ago or the early days of machine learning. I’m talking about AI in 2025, 2024, and 2023, when most of the GenAI craze was really just productivity tooling and chatting with data.
A flight data recorder wasn’t that essential to begin experimenting with LLMs. One could argue you just needed a gateway to clamp down on chatbot ingress and egress.
However, today’s agentic world is entirely different, and we’re now getting serious about how AI transforms the enterprise.
Anyone coming from the world of RPA will know exactly what we are up against. Agents are actually doing things with your systems and data. They are retrieving documents. They are updating records. They are calling APIs and triggering workflows. Breaches are happening. Transactions are executing. And enterprises are running almost completely blind, albeit with fragmented logging spread across an expanding Shadow AI environment.
This is why a Flight Data Recorder is so necessary right now.
Situational Awareness
The challenge is that almost everyone is missing the point about AI observability. It’s not about logs. It’s not about telemetry. And even your concepts of workflows and data are rooted in the wrong framing.
In the physical world, when you step on a plane, when it takes off, as it navigates the airways, and as it lands, the critical instrumentation is being recorded. Sure, there is telemetry. That’s necessary. But it’s not just about those discrete measurements. The Flight Data Recorder is providing situational awareness.
And that is the fundamental key. It’s capturing air pressure, hydraulic pressure, control-surface positions and countless other indicators in relation to each other throughout the flight path. It’s temporal. There are dependencies. One signal gives context to another.
You don’t just need telemetry and logs. You need situational awareness.
Telemetry from SDK tooling can give you measurements, but it doesn’t give you the awareness required to investigate and act. Neither does bespoke tooling.
We’ve learned a lot from the physical world, and we need to apply those lessons. Airplanes, boats and cars capture enormous amounts of operational data. They also have control systems and standardized communications buses that move those signals to where they are needed. That bus is vital. It gets the telemetry out of individual components and into a record where situational awareness can actually be constructed.
This is why the Control Plane is needed if you want any meaningful level of control and observability across your agentic systems.
Cause and Effect
Do not forget that the flight data recorder is not just about an investigation after a crash. It is an absolute wealth of information on how a plane flies as well as how the operators flew it. It’s a gold mine for maintenance, optimization, safety, and predictive action. And when something goes wrong, it’s the most crucial piece of evidence for understanding cause and effect.
That’s the reality of telemetry. It’s not a static measurement. It’s a potential causal indicator, and as a causal indicator it needs to carry all the necessary metadata to understand the situation around it.
Now this is the other thing people are missing with Agents. The world will not be ruled by “God” agents, the mythical single agent designed to do it all. It’s going to be agents and micro agents working collaboratively as part of a business workflow. Just as a plane has engines, fuel systems, generators, actuators, environmental controls and miles of electronics, the enterprise consists of a maze of componentry.
Now keep in mind that I said business workflow.
These workflows are runtime and operational, working directly in and on your enterprise systems. They are not a pipeline nor a script stuffed into a god agent or even an agent. A real and true business workflow involves tens of agents performing thousands of tasks, thousands of times over.
Now try to reconstruct that through application logs and pretty dashboards.
It’s a different world from the sanitized and experimental environments where agents operate in demos and trial systems. The real world inside the enterprise is eventually going to look like the crowded airways, with thousands of live flights, millions of telemetric signals and all propped up with probabilistic models that are reasoning and adapting in real time.
In this environment, you most definitely need the Flight Data Recorder for true Situational Awareness on your AI.
The Sandbox vs Live Wire
But let’s talk about another item that is commonly misunderstood in the AI and agentic hype cycle. Data.
That might seem odd in the context of observability, but hear me out. Data used in model training is not the same as data used in development. It’s certainly not the same as data used in testing. And it’s wildly different from the data you’ll encounter in production. Production data is in flight and has consequences attached to it. It’s not just the fuel that agentic processing runs on. It is the crown jewels of the corporation.
Get it wrong in production and there is immediate risk, breach and liability.
In training, you are typically dealing with curated data. You’ve spent time getting it ready for the model. It has already been pre-processed by the time it reaches the training cycle, and then you go through reams of testing. Get it wrong there and you burn some compute cycles and cost getting it right.
If something goes wrong in production with your live, in-flight data, you have a completely different problem.
And guess what? With millions of tasks across thousands of workflows and hundreds of agents processing data, performing transactions and making decisions, it will be the situational awareness from the Flight Data Recorder that becomes mission critical.
Live-wire, in-flight data has real-time security requirements, real risks, liabilities and fines. It is varied, dynamic, volatile, siloed, and frequently unstructured. It requires definitive policy enforcement around AML, KYC, SOX and the list goes on. That is well beyond the controlled environment of a sandbox.
In production, your data has to be bound to a Control Plane that can provide evidence of when, how and where it was used. There is no sanitized lab in production, and observability becomes inseparable from your architecture..
The Landing
Do not equate an SDK, emitting telemetry or collecting logs with observability. And certainly do not equate a pretty dashboard with giving you the information you need to understand agentic AI.
And please, skip the “God” agent thinking. It will set you up for unrecoverable technical debt.
If you are touching live-wire enterprise data and you don’t have a Flight Data Recorder capturing what happens across a Control Plane and its policy enforcement points, you are just waiting for the crash.
Your reputation will take the hit. The liability will mount. And you will be left with a team of forensic scientists and forward-deployed engineers endlessly scouring debris and stitching logs together just to make sense of what happened.
This is why you need a Forensic Control Plane.
The flight recorder analogy works for me, with one caveat. A trace reconstructs what the instrumentation captured, but it does not automatically prove the record is complete or that the external action succeeded. That needs evidence from the system where the action was meant to occur. I split those evidence sources out in The Three Witnesses to a Run: https://petermccannstrain.substack.com/p/the-three-witnesses-to-a-run
So I still want the recorder. I just would not let it be the only witness.