I'm old enough to remember a time when it was a point of pride when a computer system can stay online for long periods of time (sometimes years) without crashing. It was regarded as one of the differentiations between desktop and server-class hardware to justify their significant price gap. Nowadays, a computer with years-long uptime is considered a liability: it certainly has not been updated with the latest security patches. Microsoft has a regular Patch Tuesday to roll out fixes, Apple rolls out their fixes on a less regular schedule, and Linux distributions are constantly releasing updates. For my computers running Ubuntu, running "sudo apt update" followed by "sudo apt upgrade" then "sudo reboot" is a regular maintenance task.

Recently (within the past few months) I started noticing a new behavior in my Ubuntu 22.04 installations: "sudo apt upgrade" no longer automatically installs all available updates, with a subset listed as "The following packages have been kept back". I first saw this message before, and at that time it meant there were version conflicts somewhere in the system. This was a recurring headache with Nvidia drivers in past years, but that has been (mostly) resolved. Also, if this were caused by conflicts, explicitly upgrading the package would list its dependencies. But when I explicitly upgrade a kept-back package, it installed without further complaint. What's going on?

$ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
Try Ubuntu Pro beta with a free personal subscription on up to 5 machines.
Learn more at https://ubuntu.com/pro
The following packages have been kept back:
  distro-info-data gnome-shell gnome-shell-common tzdata
The following packages will be upgraded:
  gir1.2-mutter-10 libmutter-10-0 libntfs-3g89 libpython3.10 libpython3.10-minimal libpython3.10-stdlib mutter-common ntfs-3g python3.10 python3.10-minimal
10 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
7 standard LTS security updates
Need to get 1,519 kB/9,444 kB of archives.
After this operation, 5,120 B disk space will be freed.
Do you want to continue? [Y/n]

A web search on "The following packages have been kept back" found lots of ways this message might come up. Some old problems going way back. But since this symptom may be caused by a large number of different causes, we can't just blindly try every possible fix. We also need some way to validate the cause so we can apply the right fix. I found several different potential causes, and none of the validations applied, so I kept looking until I found this AskUbuntu thread suggesting I am seeing the effect of a phased rollout. In other words: this is not a bug, it is a feature!

When an update is rolled out, sometimes the developers find out too late a problem has escaped their testing. Rolling an update out to everyone at once also means such problems hit everyone at once. Phased update rollout tries to mitigate the damage of such problems: when an update is released, it is only rolled out to a subset of applicable systems. If those rollouts go well, the following phase will distribute the update to more systems, repeating until it is available to everyone. But sometimes somebody wants to skip the wait and install the new thing before their turn in a phased rollout, so they are allowed to "sudo apt upgrade" a package explicitly without error.

So back to the problem validation step: how would we know if a package is kept back due to phased rollout? We can pull up the "apt-cache policy" associated with a package and look for a "phased" percentage associated with the latest version. If so, that means the update is in the middle of a phased rollout. If the updated package is important to us, we can explicitly upgrade now. But if it is not, we can just wait for the phases to include us and be installed in a future "sudo apt upgrade" run.

$ apt-cache policy tzdata
tzdata:
  Installed: 2022e-0ubuntu0.22.04.0
  Candidate: 2022f-0ubuntu0.22.04.0
  Version table:
     2022f-0ubuntu0.22.04.0 500 (phased 10%)
        500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages
 *** 2022e-0ubuntu0.22.04.0 100
        100 /var/lib/dpkg/status
     2022a-0ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu jammy/main i386 Packages