Ubuntu APT Mirrors Configuration on AWS EC2

ubuntuawsaptec2mirrorscloud-initdevops

Bottom line

Ubuntu EC2 instances use region-specific APT mirrors at <region>.ec2.archive.ubuntu.com, auto-configured by cloud-init at boot. These mirrors are not hosted on AWS infrastructure - the S3-backed mirror feature that Canonical introduced in 2012 was discontinued around 2022-2023 (community-reported, no official announcement).

In practice, apt traffic from an EC2 instance must traverse the public internet, which has three consequences. Private-subnet instances need a NAT Gateway (~$32.40/month/AZ + $0.045/GB), apt operations can suffer latency spikes or regional outages, and there is no VPC-endpoint-based solution. This contrasts unfavorably with both Amazon Linux 2023 (S3-backed repos reachable via VPC Gateway Endpoint) and Debian (CloudFront CDN, AWS-sponsored). For fleets of Ubuntu instances, the practical mitigation options are. Accept the NAT Gateway cost, deploy a local apt-mirror (300-550 GB), run apt-cacher-ng as a caching proxy, or switch to a CDN mirror like CloudFront-backed community mirrors.

Key findings

  • Finding: Ubuntu EC2 AMIs default to <region>.ec2.archive.ubuntu.com mirrors, configured automatically by cloud-init's DataSourceEC2 module. This is industry-standard behavior and works transparently for public-subnet instances. Why it matters: Understanding this default is the foundation for any customization - if you want to change mirrors, you must override cloud-init's behavior or the AMI-baked sources.List.

  • Finding: The S3-backed mirror feature (<region>.ec2.archive.ubuntu.com.s3.amazonaws.com) that previously allowed VPC-Endpoint-based apt access is no longer available. Why it matters: This removes the cleanest architectural pattern for private-subnet Ubuntu instances; every apt operation now requires internet egress or local infrastructure.

  • Finding: Debian and Amazon Linux both provide AWS-native package delivery (CloudFront CDN and S3 respectively) while Ubuntu doesn't. Why it matters: This is a tangible competitive gap for Ubuntu on AWS that affects operational cost and architectural simplicity, especially in security-sensitive private-subnet deployments.

Background

Canonical has maintained region-specific APT mirrors for AWS EC2 since at least 2012, when "Regional S3 Backed EC2 Mirrors" were announced for testing. The mirror naming convention is <aws-region>.ec2.archive.ubuntu.com (e.G., us-east-1.ec2.archive.ubuntu.com, ap-south-1.ec2.archive.ubuntu.com). These mirrors were initially backed by S3, making them accessible via S3 VPC Gateway Endpoints - an elegant architecture for instances in private subnets. At some point between ~2015 and 2023, S3 backing was removed, and the mirrors moved to non-AWS infrastructure.

Cloud-init, the industry-standard instance initialization tool, is responsible for configuring these mirrors at first boot. Its EC2 DataSource detects the AWS region from instance metadata and substitutes it into the mirror URL template. Cloud-init also supports explicit mirror override via the apt cloud-config key, offering uri, search (fallback list), and search_dns (DNS-based discovery) options.

Current state (as of May 2026)

  • cloud-init 26.1 is the current documentation version and maintains the same mirror-selection behavior.
  • Ubuntu 24.04+ uses deb822-format .sources files in /etc/apt/sources.list.d/ instead of legacy sources.list.
  • The ec2.archive.ubuntu.com mirrors use unencrypted HTTP (APT verifies package integrity via GPG signatures, making encryption less critical for content but relevant for privacy/metadata).
  • Canonical's main archive experienced a significant outage on September 5, 2025 (36 minutes downtime on archive.ubuntu.com and security.ubuntu.com), with mirror synchronization issues persisting for ~2 days. Notably, EC2-specific mirrors were reported to remain functional during this outage.
  • The full Ubuntu package archive is about 3.6 TB as of September 2025.

Technical details

Mirror selection mechanism

Cloud-init resolves the apt mirror in this priority order:

  1. Explicit uri or search list in cloud-config user-data
  2. search_dns - looks for <distro>-mirror DNS entry in instance FQDN, localdomain, then search domains
  3. DataSource-provided mirror - on EC2: <region>.ec2.archive.ubuntu.com (constructed from instance metadata region)
  4. Distro default - http://archive.ubuntu.com/ubuntu

The security mirror is resolved similarly, using <distro>-security-mirror for DNS search and http://security.ubuntu.com/ubuntu as the default.

cloud-init apt configuration example

#cloud-config
apt:
  preserve_sources_list: false
  primary:
    - arches: [default]
      uri: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/
  security:
    - arches: [default]
      uri: http://security.ubuntu.com/ubuntu/

Or using search with fallback:

#cloud-config
apt:
  primary:
    - arches: [default]
      search:
        - http://local-mirror.internal/ubuntu
        - http://archive.ubuntu.com/ubuntu

AMI-level vs cloud-init-level configuration

Mirror selection happens at two levels: AMI build time (sources.List is customized per region during image publication) and cloud-init runtime. For Ansible-based configuration, setting preserve_sources_list: true in cloud-init and managing sources via Ansible's apt_repository or direct file management is a common pattern.

Evidence, comparisons, and related context

Comparison: Debian on AWS

Debian uses deb.debian.org which resolves via SRV DNS records to both Fastly and Amazon CloudFront (sponsored by AWS). The dedicated endpoint cdn-aws.deb.debian.org serves packages entirely through AWS's CDN. In practice, Debian instances on EC2 download packages from within AWS's network - no internet egress, no NAT Gateway cost. HTTPS is also supported.

Comparison: Amazon Linux 2023

AL2023 hosts its package repositories on S3, served through CloudFront. Instances in private subnets can reach them via an S3 VPC Gateway Endpoint (free, except for data processing at standard S3 rates). No NAT Gateway is required for yum/dnf operations. This is documented as a first-class AWS feature.

Local mirror (apt-mirror)

A full local mirror requires 300-350 GB initially (Ubuntu 24.04 amd64) plus 150-200 GB over the release lifetime. Syncs run every 4 hours via cron. For fleets of 10+ Ubuntu hosts, this economically pays for itself quickly by eliminating per-instance internet egress.

Caching proxy (apt-cacher-ng)

Apt-cacher-ng acts as a caching HTTP proxy for APT traffic. First download of a package hits the internet; subsequent requests from other instances are served from local cache. Much lighter than a full mirror (no need to sync entire archive - only packages actually used are cached).

Ansible integration patterns

For the ansible-role-ubuntu-2604 use case, three Ansible-native approaches exist:

  1. Inline sed/user-data: Replace mirror in /etc/apt/sources.list or /etc/apt/sources.list.d/ubuntu.sources
  2. cloud-init user-data: Set apt.primary.uri to the desired mirror in the instance launch configuration
  3. Local infrastructure: Deploy apt-cacher-ng or apt-mirror within the VPC and point all instances at the internal endpoint

Limitations and critiques

  1. No VPC-endpoint path: Unlike Amazon Linux 2023 and Debian, Ubuntu has no AWS-native delivery mechanism for packages. This is a genuine architectural gap that forces either internet access, NAT Gateway cost, or self-managed mirror infrastructure.

  2. HTTP-only transport: All default Ubuntu APT mirrors use unencrypted HTTP. While package integrity is verified via GPG signatures, metadata leakage (which packages an instance is downloading) and potential for downgrade/blocking attacks exist. Debian's CloudFront mirrors support HTTPS.

  3. S3 discontinuation undocumented: No official Canonical announcement or documentation was found regarding the removal of S3 backing. The only evidence is a March 2023 Server Fault comment. This lack of transparency makes it difficult for operators to plan migrations.

  4. Mirror performance variability: Community reports (2015-2025) document ec2.archive.ubuntu.com speeds varying from 6 MB/s to as low as 157 bytes/second. This can cause autoscaling health-check failures if instances can't apt-update within grace periods.

  5. New AWS region lag: There is no documented guarantee that new AWS regions immediately receive ec2.archive.ubuntu.com mirrors. Instances may fall back to archive.ubuntu.com if the region-specific DNS doesn't resolve.

  6. Mirror sync inconsistency during outages: The September 2025 incident showed that when the primary archive has issues, mirrors that sync during the incident window can serve broken content for hours afterward.

Open questions

  • What is the exact current (2026) backend infrastructure for ec2.archive.ubuntu.com? (S3 is confirmed gone; is it CloudFront, direct EC2 instances, or something else?)
  • When exactly did Canonical discontinue S3 backing, and was it announced anywhere?
  • Does Canonical have plans to offer a VPC-endpoint-compatible or CDN-backed mirror solution for AWS?
  • What is the SLA or availability commitment for ec2.archive.ubuntu.com mirrors?
  • Do newly launched AWS regions get ec2.archive.ubuntu.com mirrors immediately, and what is the fallback behavior?

Practical takeaways

  • For public-subnet instances: The default ec2.archive.ubuntu.com mirrors work well with no configuration needed. Consider mirror://mirrors.ubuntu.com/mirrors.txt as a fallback for resilience.

  • For private-subnet instances: You have three options. (A) run a NAT Gateway and accept the cost, (b) deploy an internal apt-mirror or apt-cacher-ng, or (c) switch to a CDN-based mirror that supports VPC-endpoint routing if one emerges. The self-managed mirror is the most cost-effective for fleets of 10+ instances.

  • For Ansible roles: Set apt.primary.uri in cloud-init user-data to point at your chosen mirror. If using an internal mirror, also explicitly set apt.security.uri to avoid instances reaching out to security.ubuntu.com directly. For Ubuntu 24.04+, manage /etc/apt/sources.list.d/ubuntu.sources in deb822 format rather than legacy sources.list.

  • For cost-sensitive deployments: Calculate whether NAT_Gateway_cost > mirror_infrastructure_cost. At ~$32/month per AZ plus data charges, a t3.Small mirror instance with attached EBS volume may break even at surprisingly small fleet sizes.

  • Monitor the mirror landscape: Canonical's mirror infrastructure is evolving. Watch status.canonical.com and the ubuntu-mirrors-announce mailing list for changes. The Debian CloudFront model suggests a path Canonical could adopt.

Sources used