LVE load fail: modprobe: ERROR: could not insert 'kmodlve': Cannot allocate memory

Issue

• LVE Module cannot be loaded with an error:

  modprobe: ERROR: could not insert 'kmodlve': Cannot allocate memory

• dmesg contains an error

  LVE: [53551] mount_cgroup_root_fs: failed to mount cgroup(memory), rc=-16

• The command below returns cgroup2fs

  # stat -fc %T /sys/fs/cgroup/
  cgroup2fs

• No systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller in /proc/cmdline

Environment

• CloudLinux 
• tuned-profiles-cloudlinux >= 0.2-5
• LVE

Solution

1. Edit the file /etc/default/grub:
   Change systemd.unified_cgroup_hierarchy=1 to systemd.unified_cgroup_hierarchy=0 in the file. If the variable is missing, add it to the end of the "GRUB_CMDLINE_LINUX" line. 
2. Do not forget to check if the correct EFI partition was mounted following this article:
   https://cloudlinux.zendesk.com/hc/en-us/articles/360021620699-CloudLinux-7-8-kernel-unable-to-boot-on-UEFI-enabled-soft-raid-1-server 

3. Check if the systemd.unified_cgroup_hierarchy=0 was applied:

   grep systemd.unified_cgroup_hierarchy /proc/cmdline

4. Regenerate the grub configuration

   grub2-mkconfig -o /etc/grub2.cfg 
   grub2-mkconfig -o /etc/grub2-efi.cfg 
   grub2-mkconfig -o /boot/grub2/grub.cfg 
   grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg 
   grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg

5. Reboot the server.

---

Also, the problem can be related to a specific tuned profile set before the cldeploy.
Please set it to the correct one:

tuned-adm profile cloudlinux-default

We are going to release the next version of tuned-profiles-cloudlinux 0.2-7 to automate the switching of the profile to ours.

---

Get current running kernel information:

grubby --info /boot/vmlinuz-$(uname -r) | grep systemd.unified_cgroup_hierarchy

Or, check all the installed kernels using:

grubby --info=ALL | grep systemd.unified_cgroup_hierarchy

In case the output is empty, add the required argument to rthe unning kernel:

grubby --args="systemd.unified_cgroup_hierarchy=0" --update-kernel=/boot/vmlinuz-$(uname -r)

or to a specific kernel by changing /boot/vmlinuz-$(uname -r) to the required kernel version. For example, /boot/vmlinuz-5.14.0-570.19.1.el9_6.x86_64

If there are multiple kernels, you can add the required argument to all with:

grubby --args="systemd.unified_cgroup_hierarchy=0" --update-kernel=ALL

Also, you may try changing GRUB_ENABLE_BLSCFG to false in /etc/default/grub

GRUB_ENABLE_BLSCFG=false

---

On CL 9, please check the file /etc/default/grub that may be updated after rebooting the server and restarting tuned service, as it may have strange kernel boot parameters:

ds=nocloud;[seedfrom=/var/lib/cloud/seed/nocloud];

Keep this ds parameter at the end of the line after all other kernel command line options. 

As an option, blscfg can be disabled through /etc/default/grub, and check if it is working or not.

---

Please also check if the dependency file is broken:

# depmod -a
depmod: WARNING: could not open modules.builtin at /lib/modules/5.14.0-427.42.1.el9_4.x86_64: No such file or directory

If yes, please reinstall the kernel packages using:

# yum reinstall kernel*-5.14.0-427.42*

---

If none of the above workarounds work, check /etc/default/grub for persistence of such strings:

GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:+$GRUB_CMDLINE_LINUX_DEFAULT }\$tuned_params"
GRUB_INITRD_OVERLAY="${GRUB_INITRD_OVERLAY:+$GRUB_INITRD_OVERLAY }\$tuned_initrd"

If they exist, comment them out and reboot.

Cause

Cgroups v2 is not supported by kmodlve.