LVE load fail: modprobe: ERROR: could not insert 'kmodlve': Cannot allocate memory

Issue

• LVE Module cannot be loaded with an error:

  modprobe: ERROR: could not insert 'kmodlve': Cannot allocate memory

• dmesg contains error

  LVE: [53551] mount_cgroup_root_fs: failed to mount cgroup(memory), rc=-16

• The command below returns cgroup2fs

  # stat -fc %T /sys/fs/cgroup/
  cgroup2fs

• No systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller in /proc/cmdline

Environment

• CloudLinux 
• tuned-profiles-cloudlinux >= 0.2-5
• LVE

Solution

1. Edit the file /etc/default/grub:
   Change systemd.unified_cgroup_hierarchy=1 to systemd.unified_cgroup_hierarchy=0 in the file. If the variable is missing, add it to the end of the "GRUB_CMDLINE_LINUX" line. 
2. Do not forget to check if the correct EFI partition was mounted following this article:
   https://cloudlinux.zendesk.com/hc/en-us/articles/360021620699-CloudLinux-7-8-kernel-unable-to-boot-on-UEFI-enabled-soft-raid-1-server 
3. Check if the systemd.unified_cgroup_hierarchy=0 was applied:

   grep systemd.unified_cgroup_hierarchy /proc/cmdline

4. Regenerate the grub configuration
   

   grub2-mkconfig -o /etc/grub2.cfg 
   grub2-mkconfig -o /etc/grub2-efi.cfg 
   grub2-mkconfig -o /boot/grub2/grub.cfg 
   grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg 
   grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg

5. Reboot the server.

---

Also, the problem can related to a specific tuned profile set before the cldeploy.
Please set it to the correct one:

tuned-adm profile cloudlinux-default

We are going to release the next version of tuned-profiles-cloudlinux 0.2-7 to automate the switching of the profile to ours.

---

In case the previous solutions didn't work, please make sure the cfg filename has the same output from the /etc/machine-id file.

Example:

cat /etc/machine-id
abcd1234567890

The file name should be:

/boot/loader/entries/abcd1234567890-x.x.x-x.x.x.x_x.x86_64.conf 

Where x.x.x-x.x.x.x_x = kernel version

Also, you may try changing GRUB_ENABLE_BLSCFG to true in /etc/default/grub

GRUB_ENABLE_BLSCFG=true

---

On CL 9, Please check the file /etc/default/grub that may be updated after rebooting the server and restarting tuned service, as it may have strange kernel boot parameters:

ds=nocloud;[seedfrom=/var/lib/cloud/seed/nocloud];

Keep this ds parameter at the end of the line after all others kernel command line options. 

Please disable blscfg through /etc/default/grub, and check if it is working or not.

---

Please also check if the dependency file is broken:

# depmod -a
depmod: WARNING: could not open modules.builtin at /lib/modules/5.14.0-427.42.1.el9_4.x86_64: No such file or directory

If yes, please reinstall the kernel packages using:

# yum reinstall kernel*-5.14.0-427.42*

---

If none of the above workarounds work, check /etc/default/grub for persistence of such strings:

GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT:+$GRUB_CMDLINE_LINUX_DEFAULT }\$tuned_params"
GRUB_INITRD_OVERLAY="${GRUB_INITRD_OVERLAY:+$GRUB_INITRD_OVERLAY }\$tuned_initrd"

If they exist, comment them out and reboot.

Cause

Cgroups v2 is not supported by kmodlve.