AI & Data Infrastructure | Fundamentals #3

AI infrastructure begins with something microscopic and ends with something enormous.

At one end of the system are advanced semiconductor devices measured in nanometers.

At the other are massive data centers containing thousands of servers, complex cooling systems and power infrastructure connected to regional electricity grids.

Between those two points sits an increasingly integrated industrial ecosystem.

This is what makes AI infrastructure different from a simple technology supply chain.

A chip does not create useful AI capacity on its own.

Neither does a server.

Neither does a data center.

AI computing emerges only when all of these layers operate together.

That is the core idea behind the AI infrastructure ecosystem.

From a Chip to a Computing System

The first step begins with the semiconductor.

Modern AI systems rely on advanced processors designed to execute enormous numbers of calculations in parallel.

These processors may include:

• GPUs
• AI accelerators
• CPUs
• custom ASICs

But a processor cannot operate alone.

It must be connected to memory, integrated through advanced packaging, installed on computing boards and assembled into a larger system.

This means that what begins as a semiconductor eventually becomes something much larger:

AI Processor

↓

Processor + HBM

↓

Advanced Package

↓

Compute Board

↓

AI Server

↓

Rack-Scale System

↓

AI Cluster

The physical scale increases at every step.

And with that increase comes greater complexity.

1. The Chip Layer: Creating AI Compute

At the lowest level of the ecosystem sits the AI processor.

Its job is to execute the calculations required for AI training and inference.

Modern AI accelerators can contain enormous numbers of computing units operating in parallel.

But processor performance alone is not enough.

AI workloads also require large amounts of data to move continuously between compute and memory.

This is why technologies such as HBM have become so important.

The processor provides the compute capability.

Memory keeps that compute supplied with data.

Together, they form the fundamental engine of modern AI systems.

2. Packaging: Turning Components Into an AI Accelerator

The processor and memory must then be physically connected.

This is where advanced packaging becomes important.

Modern AI processors often operate alongside multiple HBM stacks.

The distance between these components matters because data must move extremely quickly.

Advanced packaging technologies help place logic and memory closer together while supporting very high interconnect density.

This changes the role of semiconductor packaging.

Historically, packaging was often viewed as one of the final steps in semiconductor manufacturing.

In AI systems, however, it increasingly influences:

• system performance
• memory bandwidth
• power efficiency
• thermal behavior
• overall chip size and architecture

Packaging is therefore becoming part of the core design of AI computing systems.

3. From Accelerators to AI Servers

Once processors and memory are integrated, they must become usable computing systems.

That is the role of the AI server.

A modern AI server includes much more than processors.

It may contain:

• multiple AI accelerators
• CPUs
• system memory
• storage
• network interfaces
• power supplies
• thermal-management systems

At this point, the infrastructure challenge begins to move beyond semiconductor engineering.

Mechanical design, electrical design, networking and cooling become equally important.

This is the first major transition in the ecosystem:

AI infrastructure moves from chip-level engineering to system-level engineering.

4. The Rack Is Becoming the Computer

Traditional computing architecture often treated the individual server as the main building block.

High-end AI infrastructure is moving beyond that model.

Increasingly, the rack itself is becoming an integrated computing platform.

NVIDIA’s GB200 NVL72, for example, combines 36 Grace CPUs and 72 Blackwell GPUs within a single rack-scale architecture. The GPUs are connected through NVLink, while the overall rack also incorporates networking, power-delivery equipment and liquid-cooling infrastructure. NVIDIA

This represents an important shift.

Instead of designing:

processor → server

the industry increasingly designs:

processor → server → rack

as one coordinated system.

That changes everything from networking to cooling.

Why Rack-Scale Computing Matters

As AI models become larger, individual servers cannot provide enough compute capacity.

More processors must work together.

But simply adding more processors creates new problems.

They need to communicate.

They need power.

They generate heat.

They must be maintained reliably.

Rack-scale systems attempt to solve these problems at the system level.

NVIDIA’s DGX Grace Blackwell rack architecture, for example, includes compute trays, NVLink switch trays, power shelves, top-of-rack networking and liquid-cooling manifolds as parts of one integrated architecture. NVIDIA Docs

The rack is therefore no longer just a cabinet holding independent servers.

It is becoming part of the computer itself.

5. Networking: From One Rack to an AI Cluster

Even a powerful rack is not enough for the largest AI workloads.

Multiple racks must be connected.

This creates an AI cluster.

Networking allows thousands of processors to behave as parts of a much larger computing system.

There are broadly two communication problems.

Scale-Up

Scale-up networking connects processors within a tightly integrated computing domain.

The objective is to provide extremely high bandwidth and very low latency.

Technologies such as NVLink play an important role here.

Scale-Out

Scale-out networking connects separate systems or racks.

This can involve:

• Ethernet
• InfiniBand
• network switches
• NICs
• optical transceivers
• fiber networks

As clusters become larger, networking efficiency becomes increasingly important.

A processor that spends time waiting for data from another processor is expensive infrastructure sitting idle.

This is why AI performance depends not only on how fast processors compute, but also on how efficiently they communicate.

6. The Data Center Becomes Part of the Computer

Once thousands of processors, servers and racks are connected together, the next layer is the data center.

At first glance, a data center may look like little more than a building containing servers.

For AI infrastructure, that is increasingly misleading.

The facility itself directly affects computing performance.

An AI data center must support:

• high-density electrical distribution
• large network fabrics
• liquid-cooling systems
• backup power
• high floor loads
• large heat-rejection systems
• extremely high reliability

This means the boundary between IT equipment and building infrastructure is becoming less clear.

The processor affects the rack.

The rack affects the cooling architecture.

Cooling affects the building.

The building affects the electrical system.

And the electrical system determines how much compute can ultimately operate.

In this sense, the entire data center is increasingly becoming part of the computing system.

7. Power Connects AI to the Physical Economy

AI computing may happen digitally, but it ultimately depends on electricity.

That creates another major transition in the ecosystem.

The AI infrastructure chain expands beyond the technology industry into the energy industry.

A large data center may require:

• utility connections
• transmission and distribution capacity
• substations
• transformers
• switchgear
• UPS systems
• backup generators
• battery energy storage systems

These components can take much longer to build than servers.

The International Energy Agency notes that a data center can potentially be built and brought online within only a few years, while energy infrastructure often requires significantly longer planning and construction periods. IEA

This difference in timelines is becoming one of the defining challenges of AI infrastructure expansion.

When Power Becomes a Compute Constraint

Consider two identical AI data centers.

Both can obtain the same processors.

Both can obtain the same servers.

Both can build the same cooling systems.

But one location can secure 500 MW of electricity while another can obtain only 100 MW.

The first location can potentially deploy far more computing capacity.

This shows why power is becoming inseparable from AI infrastructure.

Compute capacity is increasingly constrained by:

Chip Supply × Facility Capacity × Power Availability

Improving only one of these factors does not guarantee additional usable AI compute.

The whole system must scale together.

8. Cooling Connects Compute to Mechanical Infrastructure

Electricity flowing into processors ultimately produces heat.

As AI systems become denser, managing that heat becomes more difficult.

This is where AI infrastructure connects to another industrial domain:

thermal management.

Cooling equipment may include:

• cold plates
• coolant distribution units
• pumps
• manifolds
• heat exchangers
• chillers
• cooling towers
• facility water systems

Current rack-scale systems show how deeply cooling is becoming integrated with computing hardware.

NVIDIA’s GB rack architecture includes liquid-cooling manifolds as part of the rack itself rather than treating cooling solely as separate facility infrastructure. NVIDIA Docs

This is a significant change.

The data center can no longer simply choose its cooling system after the computing equipment has been selected.

Increasingly, compute architecture and cooling architecture must be designed together.

The AI Infrastructure Ecosystem in One View

We can now connect all the layers.

Semiconductor

Advanced processors are manufactured.

↓

Memory & Packaging

HBM and advanced packaging enable high-performance accelerators.

↓

AI Servers

Processors are integrated into complete computing systems.

↓

Rack-Scale Computing

Multiple servers and processors operate as one coordinated system.

↓

Networking

Racks are connected into massive AI clusters.

↓

Data Center

The facility provides the physical environment for the cluster.

↓

Power Infrastructure

Electricity is delivered and distributed.

↓

Cooling Infrastructure

Heat is removed from the computing system.

↓

AI Compute Capacity

The entire physical system becomes usable computing infrastructure.

This is why the final product of the ecosystem is not really the GPU.

It is not even the server.

The final product is:

Reliable, scalable AI compute capacity.

The Industry Is Moving From Components to Systems

This shift has major implications for how the AI industry should be analyzed.

In the past, individual components could often be studied separately.

A semiconductor analyst could focus on chips.

A data-center analyst could focus on facilities.

An energy analyst could focus on electricity demand.

AI infrastructure increasingly connects all three.

A faster GPU may increase:

• HBM requirements
• rack power density
• networking demand
• cooling demand
• data-center power demand

One technology improvement can create downstream changes across multiple industries.

This means the unit of competition is also becoming larger.

Companies are increasingly competing not simply on components, but on their ability to participate in an integrated AI infrastructure platform.

From Technology Stack to Industrial Ecosystem

This also explains why such different companies are becoming connected to the same growth trend.

Semiconductor companies

Provide compute and memory.

Server manufacturers

Turn chips into systems.

Networking companies

Connect those systems.

Data-center operators

Provide physical capacity.

Electrical-equipment companies

Deliver and control power.

Utilities and energy companies

Supply electricity.

Cooling companies

Remove heat.

Each company may operate in a different industry.

But AI demand links them together.

That is why AI infrastructure should increasingly be understood as an ecosystem rather than a single market.

Why the Ecosystem Is Getting Larger

The scale of the system is also expanding quickly.

According to the IEA, global data-center electricity consumption is projected to reach around 945 TWh by 2030 in its base case, roughly twice 2024 levels. Accelerated servers driven largely by AI are expected to account for almost half of the net increase in data-center electricity consumption. IEA

More recent IEA analysis keeps its central 2030 projection near that level while noting that AI-focused data-center electricity consumption is growing significantly faster than overall data-center demand. IEA

This scale matters because every additional unit of AI compute must ultimately be supported by physical infrastructure.

More compute requires more:

silicon

memory

networking

land

electricity

cooling

and capital.

AI growth therefore translates directly into industrial investment.

Where Bottlenecks Can Appear

Because the ecosystem is interconnected, limitations can emerge almost anywhere.

Potential bottlenecks include:

Semiconductor Layer

• advanced process capacity
• HBM
• packaging

System Layer

• server manufacturing
• rack integration
• networking

Facility Layer

• suitable data-center sites
• construction capacity
• cooling equipment

Energy Layer

• grid connections
• transformers
• substations
• power generation

This creates an important principle:

The AI ecosystem can grow only as fast as its slowest critical layer.

A breakthrough in processor performance does not eliminate infrastructure constraints.

Sometimes it makes them more severe.

Why This Matters

The first two articles in this series established two ideas.

The first showed that AI is becoming a physical infrastructure industry.

The second mapped the value chain from semiconductors to data centers, power and cooling.

This article adds the next piece:

those layers increasingly operate as one system.

That distinction matters.

A value chain explains where the parts sit.

An ecosystem explains how the parts affect one another.

And that is increasingly how modern AI infrastructure needs to be understood.

The Bigger Picture

AI infrastructure begins with a chip.

But it does not end there.

The chip becomes part of a package.

The package becomes part of a server.

The server becomes part of a rack.

The rack becomes part of a cluster.

The cluster becomes part of a data center.

The data center becomes part of the electricity system.

And all of it depends on cooling, networking and physical infrastructure working together.

That is the AI infrastructure ecosystem.

And as AI expands, the companies capable of understanding — and solving — these connections may become just as important as the companies designing the processors themselves.

AI & Data Infrastructure Learning Path

Previous Article

The AI Infrastructure Value Chain: From Chips to Data Centers

Next Article

Why AI Infrastructure Is Becoming a Strategic Industry

References

• NVIDIA — GB200 NVL72
  Overview of NVIDIA’s rack-scale AI architecture integrating 72 Blackwell GPUs, 36 Grace CPUs, NVLink and liquid cooling.
• NVIDIA — DGX GB Rack Scale Systems User Guide
  Technical documentation covering compute trays, NVLink switching, top-of-rack networking, power shelves and cooling infrastructure within rack-scale AI systems.
• NVIDIA — DGX GB200
  Reference architecture illustrating how multiple rack-scale systems can be connected into much larger AI computing clusters.
• International Energy Agency — Energy and AI: Energy Demand from AI
  Analysis of data-center electricity consumption, accelerated servers and the mismatch between fast data-center deployment and longer energy-infrastructure development timelines.
• International Energy Agency — Key Questions on Energy and AI
  Updated analysis of AI-focused data-center electricity demand and infrastructure bottlenecks across chips and energy systems.

ℹ️ About iAtlas

iAtlas is an independent publication covering batteries, semiconductors, OLED, advanced materials, AI, and global industrial trends.

We transform complex industrial developments into clear, reliable, and easy-to-understand insights.

Whether you’re following today’s industry news or building long-term expertise, iAtlas helps you understand not only what happened, but why it matters.

Technology creates change.
Insight creates opportunity.
— iAtlas