Page tree

Welcome to FreeSoftwareServers Confluence Wiki

Skip to end of metadata
Go to start of metadata
dnf install -y libhugetlbfs-utils

This is the 'Magic file'

 /etc/sysctl.conf

The goal is to make the output of this program "happy"

sudo hugeadm --set-recommended-shmmax

This sets changes in motion immediately, though a reboot is suggested once everything is configured.

 sysctl -p

We will make changes by echoing options to sysctl.conf

echo "vm.hugetlb_shm_group = 0" >> /etc/sysctl.conf
echo "vm.min_free_kbytes = 112640" >> /etc/sysctl.conf
echo "vm.nr_hugepages = 8600" >> /etc/sysctl.conf
echo "kernel.shmmax = 18035507200" >> /etc/sysctl.conf
echo "vm.swappiness = 0" >> /etc/sysctl.conf 

For example, if I would want to reserve 20GB for my guests, I would need 20*1024*1024/2048 = 10240 pages. (I used 8600 for 16GB + a small extra cushion)

 

[root@kvmatrix sysctl.d]# hugeadm --explain
Total System Memory: 64315 MB
Mount Point Options
/dev/hugepages rw,seclabel,relatime,pagesize=2M
Huge page pools:
Size Minimum Current Maximum Default
2097152 0 0 0 *
1073741824 0 0 0
Huge page sizes with configured pools:
The /proc/sys/vm/min_free_kbytes of 67584 is too small. To maximiuse efficiency
of fragmentation avoidance, there should be at least one huge page free per zone
in the system which minimally requires a min_free_kbytes value of 112640
A /proc/sys/kernel/shmmax value of 9223372036854775807 bytes may be sub-optimal. To maximise
shared memory usage, this should be set to the size of the largest shared memory
segment size you want to be able to use. Alternatively, set it to a size matching
the maximum possible allocation size of all huge pages. This can be done
automatically, using the --set-recommended-shmmax option.
hugeadm:WARNING: We can't make a shmmax recommendation until huge pages are configured!
To make your hugetlb_shm_group settings persistent, add the following line to /etc/sysctl.conf:
vm.hugetlb_shm_group = 0
Note: Permanent swap space should be preferred when dynamic huge page pools are used.
[root@kvmatrix sysctl.d]# cat /etc/fstab
hugetlbfs /dev/hugepages hugetlbfs defaults 
[root@kvmatrix sysctl.d]# echo "vm.nr_hugepages = 8600" >> /etc/sysctl.conf
[root@kvmatrix sysctl.d]# echo "vm.hugetlb_shm_group = 0" >> /etc/sysctl.conf
[root@kvmatrix sysctl.d]# echo "vm.min_free_kbytes = 67584" >> /etc/sysctl.conf
[root@kvmatrix sysctl.d]# sysctl -p
vm.nr_hugepages = 8600
vm.hugetlb_shm_group = 0
vm.min_free_kbytes = 67584
[root@kvmatrix sysctl.d]# sudo hugeadm --set-recommended-shmmax 
[root@kvmatrix sysctl.d]# hugeadm --explain
Total System Memory: 64315 MB
Mount Point Options
/dev/hugepages rw,seclabel,relatime,pagesize=2M
Huge page pools:
Size Minimum Current Maximum Default
2097152 8600 8600 8600 *
1073741824 0 0 0
Huge page sizes with configured pools:
2097152
The /proc/sys/vm/min_free_kbytes of 67584 is too small. To maximiuse efficiency
of fragmentation avoidance, there should be at least one huge page free per zone
in the system which minimally requires a min_free_kbytes value of 112640
The recommended shmmax for your currently allocated huge pages is 18035507200 bytes.
To make shmmax settings persistent, add the following line to /etc/sysctl.conf:
kernel.shmmax = 18035507200
To make your hugetlb_shm_group settings persistent, add the following line to /etc/sysctl.conf:
vm.hugetlb_shm_group = 0
Note: Permanent swap space should be preferred when dynamic huge page pools are used. 
[root@kvmatrix sysctl.d]# echo "vm.min_free_kbytes = 112640" >> /etc/sysctl.conf
[root@kvmatrix sysctl.d]# echo "kernel.shmmax = 18035507200" >> /etc/sysctl.conf
[root@kvmatrix sysctl.d]# echo "vm.swappiness = 0" >> /etc/sysctl.conf
[root@kvmatrix sysctl.d]# sysctl -p
vm.nr_hugepages = 8600
vm.hugetlb_shm_group = 0
vm.min_free_kbytes = 67584
vm.min_free_kbytes = 112640
kernel.shmmax = 18035507200
vm.swappiness = 0 
[root@kvmatrix sysctl.d]# hugeadm --explain
Total System Memory: 64315 MB
Mount Point Options
/dev/hugepages rw,seclabel,relatime,pagesize=2M
Huge page pools:
Size Minimum Current Maximum Default
2097152 8600 8600 8600 *
1073741824 0 0 0
Huge page sizes with configured pools:
2097152
The recommended shmmax for your currently allocated huge pages is 18035507200 bytes.
To make shmmax settings persistent, add the following line to /etc/sysctl.conf:
kernel.shmmax = 18035507200
To make your hugetlb_shm_group settings persistent, add the following line to /etc/sysctl.conf:
vm.hugetlb_shm_group = 0
Note: Permanent swap space should be preferred when dynamic huge page pools are used.
[root@kvmatrix sysctl.d]# 
[root@kvmatrix sysctl.d]# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
vm.nr_hugepages = 8600
vm.hugetlb_shm_group = 0
vm.min_free_kbytes = 112640
kernel.shmmax = 18035507200
vm.swappiness = 0
[root@kvmatrix sysctl.d]# cat /proc/meminfo | grep Huge
AnonHugePages:  30136320 kB
ShmemHugePages:        0 kB
HugePages_Total:    8600
HugePages_Free:     8600
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Now add this to guest and shutdown and turn on (not restart)

<memoryBacking>
<hugepages/>
</memoryBacking>

This shows its working, noticed the "Free"

[root@kvmatrix ~]# cat /proc/meminfo | grep Huge
AnonHugePages:  13359104 kB
ShmemHugePages:        0 kB
HugePages_Total:    8600
HugePages_Free:      408 << This is the extra cushion
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
sysctl -a | grep zone_reclaim_mode 
vm.zone_reclaim_mode = 0 

if it says 0, keep it that way!

https://discuss.aerospike.com/t/tuning-kernel-memory-for-performance/4195

 

  • No labels