AI infrastructure is becoming increasingly dynamic. Workloads can move between cloud, edge and devices. Compute resources become congested or available. Network paths deteriorate and recover, and the best place to execute a workload at one moment may not be the best place moments later.
Yet the application still has to perform. If a workload requires a response within a defined time window, a certain level of continuity or predictable communication behaviour, those service objectives still need to be met even when the infrastructure beneath it changes.
This raises an important question for distributed AI: What should the infrastructure actually optimise for?
Much of today’s AI conversation understandably focuses on individual components: more capable models, faster accelerators, greater inference throughput and lower latency. All of these matter. But as AI applications increasingly span multiple compute environments and communication paths, the performance of any one component tells us less about whether the complete application is operating as required.
In other words, component success does not automatically mean system success.
For distributed AI, the next infrastructure challenge may therefore be less about maximising every individual component and more about ensuring that a changing system continues to operate within the requirements of the workload. One way to think about this is as a performance contract between the application and the infrastructure supporting it: a clear definition of the operating conditions that need to be maintained for that workload to perform as intended.
Physical AI makes this issue particularly clear because infrastructure decisions ultimately affect what happens in the physical world.
Consider a mobile robot. Perception, navigation, reasoning and physical action can involve very different computational requirements and time constraints. Some functions may need to remain onboard, while others may benefit from more capable edge or cloud infrastructure. There is no universally optimal answer to where that computation should take place.
Microsoft Research demonstrated these trade-offs in a 2026 measurement study of mobile robotic manipulation across onboard, edge and cloud GPU platforms. The researchers found that the full workload stack was impractical on smaller onboard GPUs, while more capable onboard GPUs came with increased energy demands. Offloading alleviated some of those constraints but introduced others: additional network latency affected task accuracy, while bandwidth requirements could make straightforward cloud offloading impractical.
The significance is not that cloud is better than edge, or that more compute is necessarily better. It is that the performance of the application emerges from the interaction between several parts of the system.
A faster processor may improve one part of the workflow. Lower network latency may improve another. Greater energy efficiency may extend operating time. None of these, in isolation, tells us whether the robot will complete its task within the conditions required.
This is not limited to robotics. Production AI workloads already exhibit very different service requirements. Microsoft Research has examined large-scale LLM serving across workloads ranging from latency-sensitive interactions to tasks with much more relaxed timing requirements, requiring coordinated decisions across request routing, compute allocation and model placement.
The applications may be very different, but the underlying observation is similar: different workloads place different demands on the infrastructure beneath them.
This is why there is unlikely to be a single definition of “real-time AI”.
A conversational service may prioritise responsiveness. An industrial vision system may require consistent processing within a defined time window. An autonomous machine may keep safety-critical functions local because they cannot depend on external connectivity, while a complex reasoning workload may tolerate more delay in exchange for access to more capable compute.
The more useful starting point is therefore not simply to ask how low latency can go, but to ask what the workload actually requires.
Depending on the application, those requirements may include response time, tail latency, bandwidth, continuity, resilience, recovery behaviour or geographic constraints. Once understood, they can be translated into measurable service-level objectives.
Research is already moving in this direction. A 2026 study published in IEEE Transactions on Services Computing examined edge LLM workloads with differentiated objectives for time to first token, time per output token and end-to-end latency. The researchers found that prioritising throughput alone could result in poor attainment of workload-specific objectives, and instead proposed scheduling around differentiated SLO requirements.
The broader principle matters more than any individual metric: maximum component performance is not necessarily the same as the performance the application requires.
For infrastructure leaders, this changes the design logic. The starting point becomes the operating conditions the application needs. From there, the questions move to where the workload should execute, how the required data reaches it, how performance should be measured across the full execution path, and how the system should respond as real-world conditions change.
The infrastructure may be dynamic. The active service objective still has to be met.
In the previous article in this series, we explored how network and compute decisions are becoming increasingly interdependent, and why this points towards a more intelligent network layer.
Recent developments reinforce that direction.
In October 2026, the IETF published RFC 10053 and RFC 10054 on Computing-Aware Traffic Steering (CATS). The framework considers computing-resource state alongside network conditions when directing traffic towards service instances, rather than relying on connectivity information alone. Significantly, RFC 10054 includes distributed AI inference and training among its use cases.
Commercial infrastructure is moving in a similar direction. NVIDIA has described geographically distributed “AI grids” extending accelerated computing across centralised, regional and edge infrastructure, with workload placement and routing increasingly influenced by performance requirements.
These developments suggest that the question is no longer simply whether network and compute decisions should become more coordinated. The more important question is what that coordination should ultimately optimise for.
Our view is that the answer increasingly needs to come from the workload itself.
The lowest-latency network path, the least-loaded compute location or the connection with the greatest available bandwidth may all be useful inputs. None necessarily represents the objective. The objective is to help maintain the operating conditions the application requires.
This gives network intelligence a clearer purpose.
An application-aware communications layer needs visibility into current infrastructure conditions, but it also needs an explicit expression of what the workload requires. Visibility tells the system what is happening; application intent tells it what matters; runtime control provides the ability to respond.
Application-aware does not mean that the network somehow understands the application by itself. Service requirements need to be expressed through mechanisms such as policies, service objectives, orchestration or workload metadata. Once those requirements can be considered alongside real-time infrastructure conditions, however, routing, resilience and transport decisions can become increasingly responsive to what the application actually needs.
The principle is straightforward: understand what the infrastructure is delivering, understand what the workload requires, and adapt when the two begin to diverge.
This is where Intelligent Data Movement becomes more consequential. It is no longer only about transporting information efficiently or finding a better network path. It is increasingly about helping maintain the communications conditions under which distributed intelligence can continue to operate.
As intelligence becomes distributed across devices, edge infrastructure and cloud resources, communication is no longer simply about connecting two endpoints. Data, context and decisions may need to move continuously between different points of intelligence, each operating under different timing and reliability requirements.
At Caton, this is a problem we recognise from years of working on dependable real-time data movement. Professional live media provided an unusually unforgiving proving ground: sustained high-bandwidth data had to travel across networks and geographies, in real time, with very little tolerance for interruption.
Live media became an important proving ground for that underlying capability, while the infrastructure challenge itself extends well beyond any one application.
Distributed and Physical AI present a broader and more complex challenge. Compute placement becomes part of the architecture, communication may be bidirectional, and individual workloads can have very different service requirements. Technology developed for live media cannot simply be transferred unchanged, but the underlying infrastructure discipline remains relevant: understand what the application requires, observe what the communications environment is actually delivering, and adapt when the two begin to diverge.
As AI moves further into the physical world, that discipline becomes increasingly important. Models determine how intelligence is created and compute determines where it can be processed, but distributed intelligence also depends on whether data, context and decisions can move reliably between those locations within the conditions the application requires.
This is where we believe the communications layer needs to evolve next: from reliable transport towards increasingly intelligent and application-aware data movement.
The infrastructure will continue to change. Network conditions will fluctuate, compute loads will shift and workloads will move. The service objective still has to be met.
For distributed intelligence, the challenge is therefore not simply to build smarter endpoints, but to make the connections between them dependable enough for the system to work as one.