Ttgjyuk

I have a cloud server with ~14G of RAM and no swap. However, I occasionally see kswapd0 taking up some CPU when I run top. Why would kswapd0 be running at all if there's no swap space for it to manage?

It still has a process to check if there's any swap. To reduce it, you'll need to set your swappiness -

edit "/etc/sysctl.conf" as root, then change (or add)

vm.swappiness = 0

Swap space is only used for data that is not backed by any other file. Data that is mapped from other files on disk ( such as executable programs ) is still swapped to their respective files even if you don't have a swap device.

If you have no swap and kswapd0 is running, your system is actually using nearly all of the RAM at that moment. It's time to get better tools to monitor memory usage (or free/available memory in the system).

The real fix is to reduce memory usage (run processes with less memory leaks, run less processes, skip running some processes at all, limit number of children/worker processes of some server software) or to get more RAM. If the need for RAM is caused by memory leaks, you may opt to use swap instead. Linux should be pretty smart getting the leaked parts to swap given enough time. Having swap is better than nothing but that is not a real substitute for having adequate amount of RAM.

It is a well known problem that when Linux run out of memory it can enter swap loops instead of doing what it should be doing, killing processes to free up ram. There are an OOM (Out of Memory) killer that does this but only if Swap and RAM are full.

However this should not really be a problem. The problem is that if there are a bunch of offending processes like for example Firefox and Chrome tabs that are both using and grabbing memory then these processes will cause swap read back. Linux then enters a loop where the same memory are being moved back and fort between memory and hard drive. This in turn cause priority inversion where swapping a few processes back and fort makes the system unresponsive.

If you disable swap you make this problem worse as kswapd0 now have no option than to swap out mapped memory such as executables. If you swap out executables it is even more likely that they will be swapped back in again rather quickly.

I tried triggering this behavior in NetBSD for testing and what happened there is that the the offending process became incredible slow while the OS itself was very responsive. Meaning that the swapping problem do occur but there are no priority inversion. However NetBSD doesn't have AMDGPU drivers so I am sticking to Linux for time being. Perhaps NetBSD doesn't memory map executables and that is why it doesn't enter swap loops but I don't really know enough about it's implementation to say why it doesn't become unresponsive.

Facebook had this problem as well and created the OOMD which is the Out Of Memory Daemon. This is daemon that detects kswapd0 activity and starts killing processes. And according to Facebook this almost entirely removed the problem of Linux servers becoming unresponsive. However I have not tested it and I don't know how well it will work on other servers or desktop/laptops. Appealingly OOMD has some logic deciding what processes to kill first in order to preserve system processes and the part of their server system that are responsible for relaunching whatever was killed.

However this is not how it should be solved. OOMD is an UGLY HACK. The real solution is to fix the priority inversion that a swap loop causes as well as making the kernel OOM Killer more aggressive at killing processes to free memory. The fix belong in the kernel because that's the only place where it we can be sure that the problem is detected in time and processes are being properly killed.

Setting swappiness=0 is no solution because when the system is out of free RAM it starts swapping no matter what. There are no option to guarantee that the system doesn't start swapping.

And also fixing the offending applications is not a fix. Expecially not if a user wants to exploit this bug in order to intentionally make the OS unresponsive. To be responsive is the responsibility of the kernel. If Firefox makes itself unresponsive then the fix is to the application. However it is not only making itself unresponsive but causing the entire OS to become very slow and unresponsive. To the level that it can take half an hour to log in to SSH. The SSH has nothing to do with and if it doesn't get to run that is a bug in the kernel, not in any other part of the system. And it is not a bug it is two bugs. One bug is priority inversion where a off the rails swapping cycle is allowed to interfere with other processes than the offending process(es) and that in itself is bad. The other bug is that it doesn't detect that it is in a swap loop and that causes insane wear on the HDD/SSD or whatever storage that is backing the swap. When swapping executable this is less of a problem as they are read only memory maps which aren't written back to disks but kswapd0 still gets locked reading back what it at the same time are deleting from memory.

Oh and there is a third bug. The fact that there are no way to protect disk CACHE from being eaten when memory hungry applications swallow all available memory. This is one of the reasons that kswapd0 makes the system unresponsive. The most hot memory mapped data are usually stored in the disk cache but when firefox has eaten that cache, well it obviously means that disk reads will have to occur.

It's not necessarily Firefox that are causing your problem but it is the default browser, not Chrome. And both are widly known to trigger this problem as they treat available memory as something that is wasted, including cache and swap memory which in Linux counts as "available memory". So in order not to get "available memory" get's wasted it use it for caching and other stuff. Obviously using SWAP for DISK CACHE is a VERY BAD IDEA but the fellows at both Firefox and Chrome respond to that with "free memory is wasted memory".

So what we have here is three kernel bugs that the kernel team do not seem to consider bugs. And a bug in Firefox, Chrome and all derivatives that they do not consider a bug. I tried building Firefox on my Fedora laptop in order to look into this problem and perhaps patch it. Guess what. Building Firefox with GCC on a 4 core CPU with 4GB ram triggers a SWAP LOOP with PRIORITY INVERSION. So one of the applications that has to be rewritten is GCC. On NetBSD what happens is just the 4 running instances of GCC gets slower than running one instance but it doesn't freeze the system.

Yeah this is a bit of a rant but I hope that it clarifies the current problem with the Linux memory subsystems as well as the applications that cause it.