Generative AI is quickly transitioning from experimental novelty to foundational technology across all industries. From healthcare and finance to manufacturing and retail, AI-powered applications are beginning to shape how we think, work, and interact. But behind every chat prompt, predictive model, or intelligent agent is a massive, evolving digital infrastructure. As demand for AI grows, the network has become a critical enabler – not just of performance, but of possibility. The shift is no longer about delivering content faster; it’s about delivering artificial intelligence instantly, reliably, and at scale.
To explore how networks are being reimagined for this new era, I sat down with Rodney Dellinger, CTO of Webscale at Nokia, and Alan Weckel, co-founder and principal analyst of 650 group, for a roundtable conversation on what it will take to support AI at scale. From the rise of distributed data centers and edge inference to the growing role of optical transport and automation, the discussion made one thing abundantly clear: the future of AI is inseparable from the future of interconnection. Here are five key takeaways from our conversation.
1. “AI is reshaping network infrastructure at unprecedented speed”
The scale and speed at which AI is transforming digital infrastructure is unlike anything the industry has seen before. As enterprises rush to embed generative AI into everything from customer support to product development, data centers are undergoing a radical redesign – and the network is being pushed to its limits. Estimates suggest more than $2 million per minute is being spent on AI infrastructure, with data center port deployments happening at a rate of 300-400 per minute. “From an infrastructure perspective, we're going to spend over a trillion dollars on AI,” said Alan Weckel, co-founder of 650 Group. “That equates to roughly 400 ports per minute being installed into the data center.” This is not an incremental evolution. It’s a wholesale reinvention of the digital backbone that powers our lives, driven by a surge in east-west traffic and the insatiable bandwidth demands of GPU-intensive workloads.
What makes this moment so striking is the speed of transition. Just a few years ago, traditional compute dominated the data center landscape. Now, AI infrastructure is poised to take the lead, with AI-related connectivity growing north of 100% year-over-year. “We’ve never really seen anything quite like this,” said Weckel. “In the last couple of years, we’ve started shifting from a traditional compute market to one where AI infrastructure will be the dominant form of technology inside the data center.” If current trends continue, more than half of all data center bandwidth will be dedicated to AI by the end of the decade. The pressure is on operators to rethink everything – from network topology to capacity planning – in order to keep pace with a new generation of applications that demand not just high throughput, but ultra-low latency and consistent performance at scale.
2. “Training and inference have distinct, evolving infrastructure needs”
Behind every AI-generated output lies one of two processes: training or inference, and each places very different demands on network infrastructure. Training large language models (LLMs) requires enormous volumes of data and high-density GPU clusters with extremely low-latency interconnects. GPU clusters which build these large language models today need to be within the data center because of latency requirements. These clusters must operate in tightly coupled environments where delay between GPUs can make the difference between viable and cost-prohibitive AI training. For now, this means training must happen as close to the data and compute resources as possible – often within a single facility or metro region.
Inference, by contrast, is what happens after the model is trained – and increasingly, it’s happening much closer to the user. Latency can’t get in the way of users querying models, and the “first time to token” is a great metric by which to measure to it. “If you imagine using something like ChatGPT, tokens are like syllables,” explained Rodney Dellinger, CTO of Webscale at Nokia. “Ideally, you’d want tokens to be generated every 2 to 500 milliseconds for it to feel natural to a human reading the information.” As inference becomes more complex and conversational, the performance envelope will tighten even further. Delivering on those expectations will require not just faster compute, but smarter, more distributed networks that bring AI closer to where people – and decisions – are.
3. “Distributed AI workloads are redefining interconnection”
As demand for AI accelerates, one of the biggest challenges facing infrastructure providers is the availability of power. In many metro areas, the grid simply can’t support the energy needs of next-gen AI data centers. “We’re hitting power limits in single data centers,” noted Dellinger. “That means AI training is starting to happen over distance – with predictable, deterministic latency becoming the new priority.” The old model of warehouse-scale training centers is giving way to geographically dispersed GPU clusters, connected by ultra-low-latency optical links. It’s a redefinition of what east-west traffic means – no longer confined to a single facility, but stretched across kilometers or even entire regions.
If it wasn’t before, this new architecture makes the role of interconnection mission-critical. We will see this concept of distributed GPU clusters become a reality, and Internet Exchanges (IXs) are a great place to combine and amplify the potential of these different data centers. By aggregating capacity and minimizing latency between sites, IXs are uniquely positioned to act as neutral, high-performance meeting points for distributed AI workloads. And it’s not just about performance – it’s also about user experience. “Time to first token is really the AI-era equivalent of speeding up search,” said Weckel. “Users now expect instant responses, just like they do when they Google something. That’s why extended distance between data centers while keeping latency low is so valuable.”
4. “The edge will be critical to AI user experience”
AI doesn’t just need to be powerful; it needs to be fast. As consumers come to expect more frictionless experiences, latency is becoming a dealbreaker. That’s why inference – the process of extracting output from AI models – is moving ever closer to the edge. Edge means that the computational power, the application, and the data, are very close to the user. We all hate the loading wheel when it pops up. Nowadays, when you scroll through Instagram or TikTok, videos just start – it’s a completely different experience. And to make that happen, you need to make sure the content is as close to the user as possible.
Cloud providers such as Cloudflare and Azure are already deploying LLMs at the edge, often within the same city or region as users. As AI evolves into a set of real-time, interactive agents, this kind of proximity becomes essential. As Dellinger put it, “As we see more and more model-as-a-service applications emerging, they need to be closer to the enterprises that use them, otherwise latency will cascade and lead to a bad customer experience.” In a post-AI world, the edge isn’t just a performance enhancement – it's a prerequisite.
5. “Optics and automation will become the backbone of scalable AI networks”
The burst of AI applications we’re witnessing is putting unprecedented pressure on network infrastructure, and optics are doing the heavy lifting. From data center fabrics to long-haul interconnects, new generations of optical technology are constantly being deployed to keep pace with performance demands. “Linear pluggable optics are a big topic of conversation,” said Dellinger. “You need the optics closer and closer to either the compute chip or the switching ASIC, and we’re already seeing RFPs for 3.2 terabits.”
Dellinger continued, “The 800G ZR Plus can go 2000 kilometers at that data rate, which is shocking for a pluggable that size. That’s the real gamechanger – the distance it can cover.” Technologies like linear pluggable optics and coherent modules are moving closer to the compute itself, helping eliminate unnecessary conversions and minimize latency. We are pushing very hard for data to stay as long on the optical part as possible. Ideally, only one transformation should happen in the router – that’s how you reduce latency, by removing unnecessary conversions.
But optics alone aren’t enough. As networks grow more distributed, more layered, and more complex, automation is the only way to scale effectively. “Without automation, it simply won’t work,” said Dellinger. “You need some sort of real network-as-a-service with a ubiquitous data model that can handle a disaggregated, multi-platform network.” AI-driven workloads demand dynamic, deterministic connectivity, not just static pipes. APIs, self-service portals, and intelligent orchestration will be critical in delivering the kind of AI services that operate with speed, consistency, and scale.
As fast as artificial intelligence is evolving, the network must evolve faster. From edge inference to long-haul optics, the future of connectivity is what will determine the difference between AI as a concept and AI as a platform on which the future can realistically be built.