AI infrastructure has spent the past several years optimising where intelligence is computed.
The next challenge is how that intelligence is connected.
As inference becomes distributed across cloud, edge and devices, application performance increasingly depends on the interaction between compute placement, network conditions and data movement. These are no longer separate design decisions.
We are already seeing this convergence in inference-aware routing, distributed serving and edge-cloud orchestration.
We believe the result is the emergence of a new infrastructure requirement: a network layer that is aware of both application intent and real-time network conditions and can adapt accordingly.
This article explores why that shift matters and why the network is becoming part of the AI stack.
For most of the Internet’s history, the division of labour was straightforward. Applications determined what needed to happen, compute processed the request, and the network provided the connection between them.
That separation worked because the underlying communication model was relatively stable. Applications could largely treat the network as transport infrastructure: essential, but distinct from the logic of the application itself.
Distributed AI begins to challenge that assumption.
As inference spreads across cloud, edge and devices, application performance increasingly depends on where compute is available, where data resides, how quickly information can move between locations, and how the network responds when conditions change.
The role of the network is therefore beginning to shift from simple connectivity towards active coordination.
Much of the first wave of AI infrastructure investment centred on training. That was understandable. Training frontier models requires enormous concentrations of compute, making large GPU clusters and centralised AI factories a natural architectural choice.
Inference behaves differently because it happens whenever a model is used. As AI becomes embedded in customer interactions, enterprise workflows, search, recommendation, automation and autonomous systems, inference moves from an occasional compute task into an operational workload that must respond consistently under real-world conditions.
Agentic applications make this even more pronounced. A single request may involve retrieval, planning, reasoning, policy checks, multiple model calls and interactions with external tools or other agents. Each step may depend on the one before it, which means latency accumulates across the chain rather than appearing as a single isolated delay.
Production data is already showing the impact. In Akamai’s 2026 survey of technical leaders running AI inference, 82% said their most important use cases required end-to-end response times of 500 milliseconds or less, while 64% required 250 milliseconds or less. The same study found that maintaining acceptable latency under peak load was one of the most difficult scaling challenges for half of the respondents.
The implication is significant. AI infrastructure can no longer be designed only around where the most compute is available. It must also consider where the decision needs to happen, what data is required and how quickly the two can be brought together.
This does not mean that every AI workload should move to the edge. Centralised infrastructure will remain essential for training, fine-tuning and compute-intensive inference, where large GPU clusters provide significant performance and economic advantages.
What is changing is that inference introduces a broader spectrum of placement requirements. Some workloads benefit from being close to users because responsiveness matters. Others need to remain close to data because transmitting large volumes of information across long distances is inefficient. Some require specialised accelerators that may only exist in particular locations, while others can run effectively on smaller distributed infrastructure.
Physical AI makes the distinction particularly clear. A robot may execute immediate control functions locally, use nearby edge infrastructure for perception or trajectory planning, and rely on cloud systems for higher-level reasoning or long-horizon planning. The end, edge and cloud do not replace one another; they perform different roles within the same distributed system.
Industry practice is beginning to reflect this logic, although deployment is still catching up with the architectural requirement. Akamai’s research found that 60% of respondents considered proximity to users and decision points important or critical, yet 46% still operated inference in a single centralised cloud region.
That gap is revealing. The direction is becoming clearer before the infrastructure has fully evolved to support it.
Once compute is distributed across multiple locations, the problem becomes more complex. Selecting the right destination is no longer simply a matter of identifying the nearest or least-loaded server.
AI workloads introduce additional state. Does a particular worker already have the required model loaded? Does it contain reusable context in its cache? Is the appropriate accelerator available? How saturated is the GPU? Would another location improve latency sufficiently to justify the move?
These considerations are already appearing in production infrastructure.
The Kubernetes Gateway API Inference Extension, for example, adds inference-specific routing capabilities on top of traditional gateway functions. Instead of routing only according to standard HTTP or infrastructure signals, it can use model identity, request criticality and live serving metrics to select an appropriate inference endpoint.
NVIDIA Dynamo takes a similar idea into distributed model serving. Its router can consider KV-cache overlap and current worker load so that requests are directed towards compute that can execute them more efficiently.
These systems operate at the inference-serving layer rather than the transport network itself, but they illustrate an important architectural shift: routing decisions are increasingly informed by the workload being routed.
That is a meaningful departure from traditional static or purely topology-driven routing.
The relationship becomes even tighter when inference itself is disaggregated.
Large-language-model serving can separate the prefill phase, where the prompt is processed, from the decode phase, where tokens are generated. These phases can run on different pools of GPUs because they have different compute characteristics.
This can improve utilisation, but it introduces a new dependency. The intermediate model state generated during prefill must be transferred to the decode worker efficiently enough that the benefit of distributing the computation is not lost in the handoff.
NVIDIA’s Dynamo architecture treats efficient KV-cache transfer as an important requirement of disaggregated serving. This is a useful illustration of a broader principle: once compute is distributed, the movement of data between compute resources becomes part of overall compute efficiency.
Emerging standards work is beginning to examine the same problem from the networking side. A July 2026 IETF Internet-Draft on transport considerations for large-scale distributed inference discusses issues such as KV-cache transfer, path-load awareness, steering and differentiated reliability. The work is still exploratory rather than an established standard, but it signals growing recognition that distributed inference creates requirements that span both compute and transport.
The boundary between compute orchestration and network orchestration is therefore becoming less distinct.
Traditional networks primarily answer a connectivity question: can one endpoint reach another with sufficient capacity and acceptable performance?
Distributed AI adds a second question: given what the application requires and what the infrastructure is experiencing at this moment, how should that communication happen?
The distinction matters because both compute and network conditions are dynamic. A path may become congested, an inference region may approach capacity, a wireless connection may degrade, or the most useful model state may already exist at another location.
Today, operators address these situations through a combination of load balancing, traffic steering, retries, fallback paths, observability systems and operational intervention. Akamai’s 2026 survey found that 64% of respondents regarded traffic steering as very important or critical to production inference, with more mature operators placing even greater emphasis on runtime control.
Emerging IETF work is also beginning to examine closer coordination between compute and networking. Recent Internet-Drafts on distributed inference discuss edge-cloud collaboration, intelligent scheduling, predictable latency and combined compute-network visibility. These are works in progress rather than established standards, but the direction they explore is notable: compute placement and network behaviour are increasingly being treated as interdependent decisions.
We believe these developments point towards the emergence of what we describe as an Intelligent Network Layer.
This is not a new protocol, nor does it replace IP networking, SD-WAN, cloud infrastructure, inference gateways or edge platforms. It is better understood as a coordination layer that brings together two forms of awareness: what the application needs and what the underlying infrastructure can currently provide.
On the network side, that may include latency, packet loss, congestion, path availability and geographic conditions. On the application and compute side, it may include workload priority, model location, accelerator availability, cache state or service-level requirements.
The purpose is not for the network to make the application’s decisions. The application still determines what it wants to accomplish. The role of an Intelligent Network Layer is to help ensure that the necessary data reaches the appropriate compute resource under the conditions required for the application to perform.
That distinction is important. We are not arguing that networks suddenly become autonomous AI systems. We are arguing that networking increasingly needs enough context, visibility and runtime control to support distributed AI effectively.
This evolution also changes how reliability should be understood.
Traditional infrastructure is often evaluated through aggregate measures such as availability, average latency and total throughput. These metrics remain important, but real-time AI places greater emphasis on consistency and tail behaviour.
The question is not only whether a service is available over the course of a month. It is whether communication remains stable at the moment a distributed application needs to make a decision.
This is why production AI operators increasingly monitor p95 and p99 latency, build failover paths and invest in traffic steering and faster recovery. Akamai’s research shows that organisations running inference in production are placing greater emphasis on high-percentile performance and operational resilience rather than average performance alone.
For a real-time intelligent system, reliability therefore becomes something that must be maintained continuously while the application is operating. Observability, path diversity, recovery speed and runtime adaptation become part of application performance rather than separate networking concerns.
None of this diminishes the importance of models or accelerators. Compute remains the foundation of modern AI.
But compute alone does not create a distributed intelligent system.
Models must reach data. Inference must reach users. Agents must interact with applications and with each other. Edge systems must coordinate with cloud platforms, while Physical AI systems must continuously exchange information between digital intelligence and the real world.
As those interactions become more distributed, the infrastructure connecting them becomes increasingly important.
This is why we believe the traditional boundary between the AI stack and the network stack will become harder to define. Networking will continue to provide connectivity, but it will also play a growing role in placement, routing, resilience and coordination.
The network is becoming part of the AI stack not because it performs inference itself, but because distributed inference increasingly depends on the network behaving as part of the system rather than merely underneath it.
In the first article in this series, we introduced Intelligent Data Movement as an emerging capability: ensuring that the right information reaches the right destination, at the right moment, over the most appropriate path.
The Intelligent Network Layer is one way that the idea begins to take architectural form.
Intelligent Data Movement describes the outcome, while the Intelligent Network Layer describes the infrastructure capabilities required to support it: continuous visibility, context-aware routing, orchestration and adaptation across distributed environments.
We can already see elements of this architecture emerging independently across inference gateways, cache-aware routing, distributed serving, compute-aware traffic steering and edge-cloud orchestration.
The terminology will continue to evolve, and the architecture is far from settled. But the underlying direction is becoming easier to see.
As intelligence becomes distributed, the infrastructure connecting it must become more aware of both the application and the network conditions around it.
For more than a decade, Caton has been solving a specialised version of this problem through mission-critical real-time media.
Live media presented an unusual networking challenge: large volumes of data had to move continuously across unpredictable public networks with very little tolerance for interruption, while operators required real-time visibility into network behaviour.
Meeting those requirements led to capabilities around continuous observability, intelligent path orchestration, active traffic management and resilient transport.
Distributed AI is a different application domain, but some of the underlying infrastructure questions are becoming familiar. How should traffic move when several paths are available? How quickly can the system detect degradation? Can communication continue when one path, region or provider deteriorates? And can network behaviour adapt before the application itself is affected?
We believe these questions will become increasingly important as inference spreads across cloud, edge and the physical world.
The future network will not replace the model, the cloud or the edge. But as intelligence becomes distributed across all three, the network will increasingly become the layer that allows them to operate as one system.
The next generation of AI will depend not only on where intelligence is computed, but on how intelligently that computation is connected.