Ubuntu 26.04 Container Worker: Four Hardening Gotchas

ansibleubuntusystemdcontainersjournald

When building a baseline Ansible role for Ubuntu 26.04 container workers, it's easy to copy-paste historical configurations. But operating systems evolve, and defaults that worked in 20.04 or 22.04 can become silent no-ops or hidden traps.

Here are four genuinely useful technical takeaways and gotchas discovered while hardening a production ubuntu_2604 container-host baseline.

1. Systemd's Unified Cgroup Hierarchy swallows DefaultCPUAccounting

The trap: You want to ensure your container agent processes and system services are tracked for CPU, memory, and task limits. You write a systemd manager drop-in (/etc/systemd/system.conf.d/density.conf) and set:

DefaultCPUAccounting=yes
DefaultMemoryAccounting=yes
DefaultTasksAccounting=yes

The file writes successfully. Ansible reports a clean run. But if you query the runtime state with systemctl show --property=DefaultCPUAccounting --value, you get an empty string.

Why it happens: Ubuntu 26.04 runs systemd with the unified cgroup hierarchy (cgroups v2) by default. On this hierarchy, CPU accounting is unconditional and always enabled. Because of this, systemd 258+ officially deprecated DefaultCPUAccounting. Setting it's parsed but ultimately ignored as a no-op.

The takeaway: If you're writing Infrastructure-as-Code tests (like Testinfra), don't test that DefaultCPUAccounting is set in the daemon's effective memory, because systemd drops it. Just assert on Memory and Tasks accounting, and leave CPU accounting out of your drop-ins entirely.

2. The vars/ vs defaults/ Silent Override

The trap: You define a role variable ubuntu_2604_timezone: UTC in vars/main.yml. A user comes along, includes your role, and sets ubuntu_2604_timezone: America/New_York in their inventory group_vars. The playbook runs, and the host is stubbornly set to UTC.

Why it happens: Ansible's variable precedence is notoriously complex, but the golden rule of roles is often forgotten: role variables (vars/main.yml) have extremely high precedence. They will silently override inventory variables, playbook variables, and group variables. The only things that beat vars/main.yml are set_fact, include_vars, and command-line --extra-vars.

The takeaway: If a variable is meant to be a user-facing knob, it must live in defaults/main.yml. defaults/main.yml has the lowest precedence in Ansible, allowing operators to smooth override it in their inventory without fighting the role.

3. Explicitly load overlay even if the kernel lazy-loads it

The trap: Modern Ubuntu kernels will automatically load the overlay kernel module the first time a process tries to mount an OverlayFS filesystem (which is how Docker, Podman, and containerd manage image layers). Because of this lazy-loading, many infrastructure scripts remove modprobe overlay from their provisioning steps to "clean up."

Why it happens: Relying on lazy-loading is fine for a general-purpose desktop. But for a dedicated container worker, the absence of the module in the active list (lsmod) means your monitoring, readiness probes, or IaC tests can't definitively prove the host is capable of running containers until a runtime actually attempts to start. Worse, if a minimal cloud image shipped with a stripped kernel missing the module, you won't find out during the provisioning phase. You'll find out when the container runtime crashes during deployment.

The takeaway: Force-load br_netfilter, nf_conntrack, and overlay during the Ansible run and mark them persistent. It shifts the failure domain left: if the kernel lacks container routing or filesystem capabilities, the provisioning fails immediately, rather than leaving a poisoned node in the cluster.

4. Bounding Journald disk pressure before the storm

The trap: A rogue container spins in a crash-loop, dumping Java stack traces to stdout at 50,000 lines a second. The container runtime forwards this to journald. The /var/log partition fills up to 100%. Package managers (apt) fail to run, SSH logins become erratic, and you can't even install diagnostic tools to fix the issue.

Why it happens: By default, systemd-journald will cap its persistent storage at 10% of the filesystem size. On a 50GB worker node, that's 5GB of logs. But if the node runs hot, or shares that partition with other operational data, a log storm can still cause a denial of service.

The takeaway: Explicit disk pressure controls are non-negotiable for container hosts. Add a drop-in to /etc/systemd/journald.conf.d/:

[Journal]
Storage=persistent
Compress=yes
# Cap total size on small worker disks
SystemMaxUse=1G
# Rotate before files become too expensive to vacuum
SystemMaxFileSize=128M
# Fail by dropping logs, not by filling the root partition
SystemKeepFree=2G

By enforcing SystemKeepFree=2G, you guarantee that even under the most severe application log storm, the OS will aggressively delete old journals to preserve 2GB of free space on the partition, keeping the host responsive and manageable.