What is sysctl vm.min_free_kbytes configurable for Linux kernel and what value should be set? In this article, we will explore this setting and its effect on a running Linux system. We will check its effect on the OS page cache and on mallocks, as well as on what the system free command shows when this parameter is set. We’ll make some educated guesses about the ideal values for this setting and show you how to permanently set vm.min_free_kbytes to survive reboots. So, let’s begin.
How vm.min_free_kbytes works
Allocating memory may be required by the system to ensure the proper functioning of the system itself. If the kernel allows all memory to be allocated, it may have difficulty in requiring memory for regular operations to keep the OS running smoothly. This is why the kernel provides a configurable vm.min_free_kbytes. The configurable parameter will cause the kernel memory manager to save at least X amount of free memory. Here is the official definition from the Linux kernel documentation: “This is used to force the Linux virtual machine to keep the minimum number of kilobytes free. The VM uses this number to calculate the watermark value [WMARK_MIN] for each lowmem zone in the system. Each lowmem zone gets the number of free pages reserved in proportion to its size. A certain minimum amount of memory is required to allocate PF_MEMALLOC; if you set this value below 1024KB, your system will become invisibly broken and prone to deadlock under high loads. Setting it too high will turn off your machine instantly. “
Checking vm.min_free_kbytes Works
To test that the min_free_kbytes setting works as intended, I created a virtual Linux instance with only 3.75 GB of RAM. Use the free command below to analyze your system:
# free -m
Consider the above utility for free memory using the -m flag to output the values in MB. The total storage capacity ranges from 3.5 to 3.75 GB. 121 MB of memory is used, 3.3 GB is free, the buffer cache is 251 MB. And 3.3GB of storage is available.
We are now going to change the value of vm.min_free_kbytes and see how this affects system memory. We will output the new value to the proc virtual filesystem to change the value of the kernel parameter as shown below:
# echo 1500000 > /proc/sys/vm/min_free_kbytes # sysctl vm.min_free_kbytes
It can be seen that the setting has been changed to approximately 1.5 GB and is now in effect. Now let’s use the free command again to see any changes recognized by the system.
# free -m
Free memory and buffer cache are not changed by the command, but the amount of memory shown as available has been reduced from 3327 MB to 1222 MB. Which roughly corresponds to reducing the parameter change to 1.5 GB min of free memory.
Now let’s create a 2 GB data file and then see what reading that file into the buffer cache does with the values. Here’s how to create a 2GB data file in two lines of bash script below. The script will generate a random 35MB file using the dd command and then copy it 70 times to a new data_file output:
# dd if=/dev/random of=/root/d1.txt count=1000000 # for i in `seq 1 70`; do echo $i; cat /root/d1.txt >> /root/data_file; doneprediv>
Let’s read the file and ignore its contents by reading and redirecting the file to / dev / null as shown below:
# cat data_file > /dev/null
Ok what happened to our system memory with this set of maneuvers, let’s check it out now:
# free -m
Analyzing the results above. We still have 1.8 GB of free memory, so the kernel has protected a large chunk of memory as reserved due to our min_free_kbytes setting. The buffer cache used 1691 MB, which is less than the total size of our data file, which is 2.3 GB. Obviously the entire data_file cannot be cached due to the lack of available memory to use for the buffer cache. We can check that the entire file is not stored in the cache, but we calculate the retry times to read the file. If it was cached, it would take a fraction of a second to read the file. Let’s try.
# time cat data_file > /dev/null # time cat data_file > /dev/null
It took almost 20 seconds to read the file, which means it almost certainly isn’t cached.
As a final check, let’s reduce vm.min_free_kbytes so that the page cache has more room to work with, and we can expect the cache to work and read the file much faster.
# echo 67584 > /proc/sys/vm/min_free_kbytes # time cat data_file > /dev/null # time cat data_file > /dev/null
With the extra memory available for caching, the file read time dropped from 20 seconds to 0.364 seconds when it was all in the cache.
I’m curious to do another experiment. What happens to malloc calls to allocate memory from a C program when vm.min_free_kbytes is very high. Will it handle malloc? Will the system die? First, reset the vm.min_free_kbytes parameter to a really high value to resume our experiments:
# echo 1500000 > /proc/sys/vm/min_free_kbytes
In theory, we have 1.9 GB of free space and 515 MB available. Let’s use a stress test program called stress-ng to use some memory and see where we fail. We will use vm tester and try to allocate 1 GB of memory. Since we only reserved 1.5 GB on the 3.75 GB system, we think this should work.
# stress-ng --vm 1 --vm-bytes 1G --timeout 60s stress-ng: info:  dispatching hogs: 1 vm stress-ng: info:  cache allocate: default cache size: 46080K stress-ng: info:  successful run completed in 60.09s (1 min, 0.09 secs) # stress-ng --vm 2 --vm-bytes 1G --timeout 60s # stress-ng --vm 3 --vm-bytes 1G --timeout 60s
Let’s try again with a lot of workers, we can try 1, 2, 3, 4 workers and it should fail at some point. In our test, it passed with 1 and 2 workers, but failed with 3 workers.
Let’s reset vm.min_free_kbytes to a lower value and see if that helps us to run 3 memory stressors of 1 GB each on a 3.75 GB system.
# echo 67584 > /proc/sys/vm/min_free_kbytes # stress-ng --vm 3 --vm-bytes 1G --timeout 60s
This time everything went well, no errors, we tried twice without any problems. Thus, we can conclude that there is a difference in behavior that more memory is available for malloc when vm.min_free_kbytes is set to a lower value.
Default setting for vm.min_free_kbytes
The default value for the parameter on my system is 67584, which is about 1.8% of the RAM on the system, or 64MB. For security reasons, on a heavily overloaded system, I would increase it slightly, perhaps up to 128MB to account for the more free memory reserved, however, for average use, the default seems reasonable enough. Official documentation warns about overstating the value. Setting 5 or 10% of system RAM is probably not the intended use of the setting and is too high.
Configuring vm.min_free_kbytes to survive reboots
To ensure that the parameter can withstand a reboot and does not reset to its defaults on reboot, be sure to make the sysctl parameter permanent by placing the desired new value in the /etc/sysctl.conf file.
We have seen that the Linux kernel tunable parameter vm.min_free_kbytes can be tweaked and can reserve memory on the system to provide more stable system performance, especially under heavy memory usage and memory allocation. The default settings may be too low, especially on systems with a lot of memory, and should be increased carefully. We have seen that the memory reserved by this parameter prevents the OS cache from using all memory, and also prevents some malloc operations from using all memory.