CloudLinux Gradual Rollout System

Issue

What is the Gradual Rollout System in CloudLinux?

Environment

CloudLinux OS 6(h)/7(h)/8/9

CentOS ELS

Solution

Here in CloudLinux, we do our best to make Linux servers stable, secure, and profitable. While releasing packages, we use many synthetic tests covering several cases and various configurations but still want to minimize all possible risks. 

For these purposes, we have implemented a gradual rollout system. It is available since the release of these versions of rhn-client-tools packages:

• rhn-client-tools 1.1.15-2 for CL6
• rhn-client-tools 2.8.16-14 for CL7 and CL8

You can find a full announcement here.

The main goal of the gradual rollout is to spread each update to only a limited subset of servers initially and then increase that number smoothly. It helps us to detect and fix issues much earlier than before.

How the Gradual Rollout System Works

We use the Rollout system for our main CloudLinux repositories and CloudLinux EasyApache4 one (repository with all ea-apache and ea-php packages required on cPanel systems).

For those repositories, we enable the Gradual Rollout by default. The system consists of six different repositories which are called “slots”. You can find the slots for cloudlinux-rollout and cloudlinux-ea4-rollout repositories in the output of # yum repolist command or in /etc/yum.repos.d/ directory. Each slot will be shown as a separate repository like:

cloudlinux-rollout-1/7/x86_64 CloudLinux-7 - Gradual Rollout Slot 1
cloudlinux-ea4-rollout-1/7/x86_64 CloudLinux-EA4-7 - Gradual Rollout Slot 1

Same as with stable repositories, an SSL certificate controls access to the gradual rollout repositories. It is generated automatically on the CLN side and saved on a server with the help of the rhn-client-tools package during the registration or check-in process. A server without a valid certificate will always see an empty dummy repository just to make yum/dnf work properly. 

We put new CloudLinux releases in one of the 6 rollout slots. Based on the information from the-client-tools, we randomly select up to ~1% of the servers to receive the update. If we don’t get any reports about problems with it, we reroll this server list, slowly increasing the percentage of the servers involved in the update. 

Usually, we make the update available to 100% of users within two weeks. Then we copy the package to the stable repository (so that everyone has access to it) while holding it in the rollout repo for one more day to let all servers safely complete updates.

How to force-install a new package from the rollout repository

Sometimes, you may want to force the update from the rollout repository because you urgently need a fix published in a package that undergoes rollout. For this purpose, we have added special bypass repositories. They can also be found in the output of # yum repolist command or in /etc/yum.repos.d/ directory:

!cloudlinux-rollout-1-bypass/7/x86_64 
 CloudLinux-7 - Gradual Rollout Slot 1 Bypass

These repositories always have the rollout packages available for installation but are disabled by default on your CloudLinux servers. You can enable the bypass repositories, but we recommend doing it only temporarily.

Note. We strongly recommend you not to enable those bypass repositories by default. Proceed with caution, and don’t forget to disable the Bypass repo once you’re done installing.

How to disable the rollout system

You may disable the gradual rollout system on your server entirely and receive updates only from the stable repositories. To perform this, please run the following command:

yum-config-manager --disable cloudlinux-rollout*

For CloudLinux 8:

dnf config-manager --disable cloudlinux-rollout*

Troubleshooting

Most issues with the rollout system are related to the licensing problems or incorrect work of the RHN tools or yum itself. Check our Yum & RPM Problems category to fix those issues.