Caution
This document is hopelessly outdated and it asks for a completerewrite. It still contains a useful information so we are keeping ithere but make sure to check the current code if you need a deeperunderstanding.
Note
The Memory Resource Controller has generically been referred to as thememory controller in this document. Do not confuse memory controllerused here with the memory controller that is used in hardware.
Hint
When we mention a cgroup (cgroupfs’s directory) with memory controller,we call it “memory cgroup”. When you see git-log and source code, you’llsee patch’s title and function names tend to use “memcg”.In this document, we avoid using it.
Benefits and Purpose of the memory controller¶
The memory controller isolates the memory behaviour of a group of tasksfrom the rest of the system. The article on LWN 12 mentions some probableuses of the memory controller. The memory controller can be used to
Isolate an application or a group of applicationsMemory-hungry applications can be isolated and limited to a smalleramount of memory.
Create a cgroup with a limited amount of memory; this can be usedas a good alternative to booting with mem=XXXX.
Virtualization solutions can control the amount of memory they wantto assign to a virtual machine instance.
A CD/DVD burner could control the amount of memory used by therest of the system to ensure that burning does not fail due to lackof available memory.
There are several other use cases; find one or use the controller justfor fun (to learn and hack on the VM subsystem).
Current Status: linux-2.6.34-mmotm(development version of 2010/April)
Features:
accounting anonymous pages, file caches, swap caches usage and limiting them.
pages are linked to per-memcg LRU exclusively, and there is no global LRU.
optionally, memory+swap usage can be accounted and limited.
hierarchical accounting
soft limit
moving (recharging) account at moving a task is selectable.
usage threshold notifier
memory pressure notifier
oom-killer disable knob and oom-notifier
Root cgroup has no limit controls.
Kernel memory support is a work in progress, and the current version providesbasically functionality. (See section 2.7)
Brief summary of control files.
tasks | attach a task(thread) and show list ofthreads |
cgroup.procs | show list of processes |
cgroup.event_control | an interface for event_fd()This knob is not available on CONFIG_PREEMPT_RT systems. |
memory.usage_in_bytes | show current usage for memory(See 5.5 for details) |
memory.memsw.usage_in_bytes | show current usage for memory+Swap(See 5.5 for details) |
memory.limit_in_bytes | set/show limit of memory usage |
memory.memsw.limit_in_bytes | set/show limit of memory+Swap usage |
memory.failcnt | show the number of memory usage hits limits |
memory.memsw.failcnt | show the number of memory+Swap hits limits |
memory.max_usage_in_bytes | show max memory usage recorded |
memory.memsw.max_usage_in_bytes | show max memory+Swap usage recorded |
memory.soft_limit_in_bytes | set/show soft limit of memory usageThis knob is not available on CONFIG_PREEMPT_RT systems. |
memory.stat | show various statistics |
memory.use_hierarchy | set/show hierarchical account enabledThis knob is deprecated and shouldn’t beused. |
memory.force_empty | trigger forced page reclaim |
memory.pressure_level | set memory pressure notifications |
memory.swappiness | set/show swappiness parameter of vmscan(See sysctl’s vm.swappiness) |
memory.move_charge_at_immigrate | set/show controls of moving chargesThis knob is deprecated and shouldn’t beused. |
memory.oom_control | set/show oom controls. |
memory.numa_stat | show the number of memory usage per numanode |
memory.kmem.limit_in_bytes | Deprecated knob to set and read the kernelmemory hard limit. Kernel hard limit is notsupported since 5.16. Writing any value todo file will not have any effect same as ifnokmem kernel parameter was specified.Kernel memory is still charged and reportedby memory.kmem.usage_in_bytes. |
memory.kmem.usage_in_bytes | show current kernel memory allocation |
memory.kmem.failcnt | show the number of kernel memory usagehits limits |
memory.kmem.max_usage_in_bytes | show max kernel memory usage recorded |
memory.kmem.tcp.limit_in_bytes | set/show hard limit for tcp buf memory |
memory.kmem.tcp.usage_in_bytes | show current tcp buf memory allocation |
memory.kmem.tcp.failcnt | show the number of tcp buf memory usagehits limits |
memory.kmem.tcp.max_usage_in_bytes | show max tcp buf memory usage recorded |
1. History¶
The memory controller has a long history. A request for comments for the memorycontroller was posted by Balbir Singh 1. At the time the RFC was postedthere were several implementations for memory control. The goal of theRFC was to build consensus and agreement for the minimal features requiredfor memory control. The first RSS controller was posted by Balbir Singh 2in Feb 2007. Pavel Emelianov 3 4 5 has since posted three versionsof the RSS controller. At OLS, at the resource management BoF, everyonesuggested that we handle both page cache and RSS together. Another request wasraised to allow user space handling of OOM. The current memory controller isat version 6; it combines both mapped (RSS) and unmapped PageCache Control 11.
2. Memory Control¶
Memory is a unique resource in the sense that it is present in a limitedamount. If a task requires a lot of CPU processing, the task can spreadits processing over a period of hours, days, months or years, but withmemory, the same physical memory needs to be reused to accomplish the task.
The memory controller implementation has been divided into phases. Theseare:
Memory controller
mlock(2) controller
Kernel user memory accounting and slab control
user mappings length controller
The memory controller is the first controller developed.
2.1. Design¶
The core of the design is a counter called the page_counter. Thepage_counter tracks the current memory usage and limit of the group ofprocesses associated with the controller. Each cgroup has a memory controllerspecific data structure (mem_cgroup) associated with it.
2.2. Accounting¶
Figure 1: Hierarchy of Accounting¶
+--------------------+ | mem_cgroup | | (page_counter) | +--------------------+ / ^ \ / | \ +---------------+ | +---------------+ | mm_struct | |.... | mm_struct | | | | | | +---------------+ | +---------------+ | + --------------+ | +---------------+ +------+--------+ | page +----------> page_cgroup| | | | | +---------------+ +---------------+
Figure 1 shows the important aspects of the controller
Accounting happens per cgroup
Each mm_struct knows about which cgroup it belongs to
Each page has a pointer to the page_cgroup, which in turn knows thecgroup it belongs to
The accounting is done as follows: mem_cgroup_charge_common() is invoked toset up the necessary data structures and check if the cgroup that is beingcharged is over its limit. If it is, then reclaim is invoked on the cgroup.More details can be found in the reclaim section of this document.If everything goes well, a page meta-data-structure called page_cgroup isupdated. page_cgroup has its own LRU on cgroup.(*) page_cgroup structure is allocated at boot/memory-hotplug time.
2.2.1 Accounting details¶
All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.Some pages which are never reclaimable and will not be on the LRUare not accounted. We just account pages under usual VM management.
RSS pages are accounted at page_fault unless they’ve already been accountedfor earlier. A file page will be accounted for as Page Cache when it’sinserted into inode (xarray). While it’s mapped into the page tables ofprocesses, duplicate accounting is carefully avoided.
An RSS page is unaccounted when it’s fully unmapped. A PageCache page isunaccounted when it’s removed from xarray. Even if RSS pages are fullyunmapped (by kswapd), they may exist as SwapCache in the system until theyare really freed. Such SwapCaches are also accounted.A swapped-in page is accounted after adding into swapcache.
Note: The kernel does swapin-readahead and reads multiple swaps at once.Since page’s memcg recorded into swap whatever memsw enabled, the page willbe accounted after swapin.
At page migration, accounting information is kept.
Note: we just account pages-on-LRU because our purpose is to control amountof used pages; not-on-LRU pages tend to be out-of-control from VM view.
2.4 Swap Extension¶
Swap usage is always recorded for each of cgroup. Swap Extension allows you toread and limit it.
When CONFIG_SWAP is enabled, following files are added.
memory.memsw.usage_in_bytes.
memory.memsw.limit_in_bytes.
memsw means memory+swap. Usage of memory+swap is limited bymemsw.limit_in_bytes.
Example: Assume a system with 4G of swap. A task which allocates 6G of memory(by mistake) under 2G memory limitation will use all swap.In this case, setting memsw.limit_in_bytes=3G will prevent bad use of swap.By using the memsw limit, you can avoid system OOM which can be caused by swapshortage.
2.4.1 why ‘memory+swap’ rather than swap¶
The global LRU(kswapd) can swap out arbitrary pages. Swap-out meansto move account from memory to swap...there is no change in usage ofmemory+swap. In other words, when we want to limit the usage of swap withoutaffecting global LRU, memory+swap limit is better than just limiting swap froman OS point of view.
2.4.2. What happens when a cgroup hits memory.memsw.limit_in_bytes¶
When a cgroup hits memory.memsw.limit_in_bytes, it’s useless to do swap-outin this cgroup. Then, swap-out will not be done by cgroup routine and filecaches are dropped. But as mentioned above, global LRU can do swapout memoryfrom it for sanity of the system’s memory management state. You can’t forbidit by cgroup.
2.5 Reclaim¶
Each cgroup maintains a per cgroup LRU which has the same structure asglobal VM. When a cgroup goes over its limit, we first tryto reclaim memory from the cgroup so as to make space for the newpages that the cgroup has touched. If the reclaim is unsuccessful,an OOM routine is invoked to select and kill the bulkiest task in thecgroup. (See 10. OOM Control below.)
The reclaim algorithm has not been modified for cgroups, except thatpages that are selected for reclaiming come from the per-cgroup LRUlist.
Note
Reclaim does not work for the root cgroup, since we cannot set anylimits on the root cgroup.
Note
When panic_on_oom is set to “2”, the whole system will panic.
When oom event notifier is registered, event will be delivered.(See oom_control section)
2.6 Locking¶
Lock order is as follows:
folio_lock mm->page_table_lock or split pte_lock folio_memcg_lock (memcg->move_lock) mapping->i_pages lock lruvec->lru_lock.
Per-node-per-memcgroup LRU (cgroup’s private LRU) is guarded bylruvec->lru_lock; the folio LRU flag is cleared beforeisolating a page from its LRU under lruvec->lru_lock.
2.7 Kernel Memory Extension¶
With the Kernel memory extension, the Memory Controller is able to limitthe amount of kernel memory used by the system. Kernel memory is fundamentallydifferent than user memory, since it can’t be swapped out, which makes itpossible to DoS the system by consuming too much of this precious resource.
Kernel memory accounting is enabled for all memory cgroups by default. Butit can be disabled system-wide by passing cgroup.memory=nokmem to the kernelat boot time. In this case, kernel memory will not be accounted at all.
Kernel memory limits are not imposed for the root cgroup. Usage for the rootcgroup may or may not be accounted. The memory used is accumulated intomemory.kmem.usage_in_bytes, or in a separate counter when it makes sense.(currently only for tcp).
The main “kmem” counter is fed into the main counter, so kmem charges willalso be visible from the user counter.
Currently no soft limit is implemented for kernel memory. It is future workto trigger slab reclaim when those limits are reached.
2.7.1 Current Kernel Memory resources accounted¶
- stack pages:
every process consumes some stack pages. By accounting intokernel memory, we prevent new processes from being created when the kernelmemory usage is too high.
- slab pages:
pages allocated by the SLAB or SLUB allocator are tracked. A copyof each kmem_cache is created every time the cache is touched by the first timefrom inside the memcg. The creation is done lazily, so some objects can still beskipped while the cache is being created. All objects in a slab page shouldbelong to the same memcg. This only fails to hold when a task is migrated to adifferent memcg during the page allocation by the cache.
- sockets memory pressure:
some sockets protocols have memory pressurethresholds. The Memory Controller allows them to be controlled individuallyper cgroup, instead of globally.
- tcp memory pressure:
sockets memory pressure for the tcp protocol.
2.7.2 Common use cases¶
Because the “kmem” counter is fed to the main user counter, kernel memory cannever be limited completely independently of user memory. Say “U” is the userlimit, and “K” the kernel limit. There are three possible ways limits can beset:
- U != 0, K = unlimited:
This is the standard memcg limitation mechanism already present before kmemaccounting. Kernel memory is completely ignored.
- U != 0, K < U:
Kernel memory is a subset of the user memory. This setup is useful indeployments where the total amount of memory per-cgroup is overcommitted.Overcommitting kernel memory limits is definitely not recommended, since thebox can still run out of non-reclaimable memory.In this case, the admin could set up K so that the sum of all groups isnever greater than the total memory, and freely set U at the cost of hisQoS.
Warning
In the current implementation, memory reclaim will NOT be triggered fora cgroup when it hits K while staying below U, which makes this setupimpractical.
- U != 0, K >= U:
Since kmem charges will also be fed to the user counter and reclaim will betriggered for the cgroup for both kinds of memory. This setup gives theadmin a unified view of memory, and it is also useful for people who justwant to track kernel memory usage.
3. User Interface¶
To use the user interface:
Enable CONFIG_CGROUPS and CONFIG_MEMCG options
Prepare the cgroups (see Why are cgroups needed? for the background information):
# mount -t tmpfs none /sys/fs/cgroup# mkdir /sys/fs/cgroup/memory# mount -t cgroup none /sys/fs/cgroup/memory -o memory
Make the new group and move bash into it:
# mkdir /sys/fs/cgroup/memory/0# echo $$ > /sys/fs/cgroup/memory/0/tasks
Since now we’re in the 0 cgroup, we can alter the memory limit:
# echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytes
The limit can now be queried:
# cat /sys/fs/cgroup/memory/0/memory.limit_in_bytes4194304
Note
We can use a suffix (k, K, m, M, g or G) to indicate values in kilo,mega or gigabytes. (Here, Kilo, Mega, Giga are Kibibytes, Mebibytes,Gibibytes.)
Note
We can write “-1” to reset the *.limit_in_bytes(unlimited).
Note
We cannot set limits on the root cgroup any more.
We can check the usage:
# cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes1216512
A successful write to this file does not guarantee a successful setting ofthis limit to the value written into the file. This can be due to anumber of factors, such as rounding up to page boundaries or the totalavailability of memory on the system. The user is required to re-readthis file after a write to guarantee the value committed by the kernel:
# echo 1 > memory.limit_in_bytes# cat memory.limit_in_bytes4096
The memory.failcnt field gives the number of times that the cgroup limit wasexceeded.
The memory.stat file gives accounting information. Now, the number ofcaches, RSS and Active pages/Inactive pages are shown.
4. Testing¶
For testing features and implementation, see Memory Resource Controller(Memcg) Implementation Memo.
Performance test is also important. To see pure memory controller’s overhead,testing on tmpfs will give you good numbers of small overheads.Example: do kernel make on tmpfs.
Page-fault scalability is also important. At measuring parallelpage fault test, multi-process test may be better than multi-threadtest because it has noise of shared objects/status.
But the above two are testing extreme situations.Trying usual test under memory controller is always helpful.
4.1 Troubleshooting¶
Sometimes a user might find that the application under a cgroup isterminated by the OOM killer. There are several causes for this:
The cgroup limit is too low (just too low to do anything useful)
The user is using anonymous memory and swap is turned off or too low
A sync followed by echo 1 > /proc/sys/vm/drop_caches will help get rid ofsome of the pages cached in the cgroup (page cache pages).
To know what happens, disabling OOM_Kill as per “10. OOM Control” (below) and seeing what happens will behelpful.
4.2 Task migration¶
When a task migrates from one cgroup to another, its charge is notcarried forward by default. The pages allocated from the original cgroup stillremain charged to it, the charge is dropped when the page is freed orreclaimed.
You can move charges of a task along with task migration.See 8. “Move charges at task migration”
4.3 Removing a cgroup¶
A cgroup can be removed by rmdir, but as discussed in sections 4.1 and 4.2, a cgroup might have some chargeassociated with it, even though all tasks have migrated away from it. (becausewe charge against pages, not against tasks.)
We move the stats to parent, and no change on the charge except unchargingfrom the child.
Charges recorded in swap information is not updated at removal of cgroup.Recorded information is discarded and a cgroup which uses swap (swapcache)will be charged as a new owner of it.
5. Misc. interfaces¶
5.1 force_empty¶
memory.force_empty interface is provided to make cgroup’s memory usage empty.When writing anything to this:
# echo 0 > memory.force_emptythe cgroup will be reclaimed and as many pages reclaimed as possible.
The typical use case for this interface is before calling rmdir().Though rmdir() offlines memcg, but the memcg may still stay there due tocharged file caches. Some out-of-use page caches may keep charged untilmemory pressure happens. If you want to avoid that, force_empty will be useful.
5.2 stat file¶
memory.stat file includes following statistics:
per-memory cgroup local status
cache
# of bytes of page cache memory.
rss
# of bytes of anonymous and swap cache memory (includestransparent hugepages).
rss_huge
# of bytes of anonymous transparent hugepages.
mapped_file
# of bytes of mapped file (includes tmpfs/shmem)
pgpgin
# of charging events to the memory cgroup. The chargingevent happens each time a page is accounted as either mappedanon page(RSS) or cache page(Page Cache) to the cgroup.
pgpgout
# of uncharging events to the memory cgroup. The unchargingevent happens each time a page is unaccounted from thecgroup.
swap
# of bytes of swap usage
swapcached
# of bytes of swap cached in memory
dirty
# of bytes that are waiting to get written back to the disk.
writeback
# of bytes of file/anon cache that are queued for syncing todisk.
inactive_anon
# of bytes of anonymous and swap cache memory on inactiveLRU list.
active_anon
# of bytes of anonymous and swap cache memory on activeLRU list.
inactive_file
# of bytes of file-backed memory and MADV_FREE anonymousmemory (LazyFree pages) on inactive LRU list.
active_file
# of bytes of file-backed memory on active LRU list.
unevictable
# of bytes of memory that cannot be reclaimed (mlocked etc).
status considering hierarchy (see memory.use_hierarchy settings):
hierarchical_memory_limit
# of bytes of memory limit with regard tohierarchyunder which the memory cgroup is
hierarchical_memsw_limit
# of bytes of memory+swap limit with regard tohierarchy under which memory cgroup is.
total_<counter>
# hierarchical version of <counter>, which inaddition to the cgroup’s own value includes thesum of all hierarchical children’s values of<counter>, i.e. total_cache
additional vm parameters (depends on CONFIG_DEBUG_VM):
recent_rotated_anon
VM internal parameter. (see mm/vmscan.c)
recent_rotated_file
VM internal parameter. (see mm/vmscan.c)
recent_scanned_anon
VM internal parameter. (see mm/vmscan.c)
recent_scanned_file
VM internal parameter. (see mm/vmscan.c)
Hint
recent_rotated means recent frequency of LRU rotation.recent_scanned means recent # of scans to LRU.showing for better debug please see the code for meanings.
Note
Only anonymous and swap cache memory is listed as part of ‘rss’ stat.This should not be confused with the true ‘resident set size’ or theamount of physical memory used by the cgroup.
‘rss + mapped_file” will give you resident set size of cgroup.
(Note: file and shmem may be shared among other cgroups. In that case,mapped_file is accounted only when the memory cgroup is owner of pagecache.)
5.3 swappiness¶
Overrides /proc/sys/vm/swappiness for the particular group. The tunablein the root cgroup corresponds to the global swappiness setting.
Please note that unlike during the global reclaim, limit reclaimenforces that 0 swappiness really prevents from any swapping even ifthere is a swap storage available. This might lead to memcg OOM killerif there are no file pages to reclaim.
5.4 failcnt¶
A memory cgroup provides memory.failcnt and memory.memsw.failcnt files.This failcnt(== failure count) shows the number of times that a usage counterhit its limit. When a memory cgroup hits a limit, failcnt increases andmemory under it will be reclaimed.
You can reset failcnt by writing 0 to failcnt file:
# echo 0 > .../memory.failcnt
5.5 usage_in_bytes¶
For efficiency, as other kernel components, memory cgroup uses some optimizationto avoid unnecessary cacheline false sharing. usage_in_bytes is affected by themethod and doesn’t show ‘exact’ value of memory (and swap) usage, it’s a fuzzvalue for efficient access. (Of course, when necessary, it’s synchronized.)If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)value in memory.stat(see 5.2).
5.6 numa_stat¶
This is similar to numa_maps but operates on a per-memcg basis. This isuseful for providing visibility into the numa locality information withinan memcg since the pages are allowed to be allocated from any physicalnode. One of the use cases is evaluating application performance bycombining this information with the application’s CPU allocation.
Each memcg’s numa_stat file includes “total”, “file”, “anon” and “unevictable”per-node page counts including “hierarchical_<counter>” which sums up allhierarchical children’s values in addition to the memcg’s own value.
The output format of memory.numa_stat is:
total=<total pages> N0=<node 0 pages> N1=<node 1 pages> ...file=<total file pages> N0=<node 0 pages> N1=<node 1 pages> ...anon=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...unevictable=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...hierarchical_<counter>=<counter pages> N0=<node 0 pages> N1=<node 1 pages> ...
The “total” count is sum of file + anon + unevictable.
6. Hierarchy support¶
The memory controller supports a deep hierarchy and hierarchical accounting.The hierarchy is created by creating the appropriate cgroups in thecgroup filesystem. Consider for example, the following cgroup filesystemhierarchy:
root / | \ / | \a b c | \ | \ d e
In the diagram above, with hierarchical accounting enabled, all memoryusage of e, is accounted to its ancestors up until the root (i.e, c and root).If one of the ancestors goes over its limit, the reclaim algorithm reclaimsfrom the tasks in the ancestor and the children of the ancestor.
6.1 Hierarchical accounting and reclaim¶
Hierarchical accounting is enabled by default. Disabling the hierarchicalaccounting is deprecated. An attempt to do it will result in a failureand a warning printed to dmesg.
For compatibility reasons writing 1 to memory.use_hierarchy will always pass:
# echo 1 > memory.use_hierarchy
7. Soft limits¶
Soft limits allow for greater sharing of memory. The idea behind soft limitsis to allow control groups to use as much of the memory as needed, provided
There is no memory contention
They do not exceed their hard limit
When the system detects memory contention or low memory, control groupsare pushed back to their soft limits. If the soft limit of each controlgroup is very high, they are pushed back as much as possible to makesure that one control group does not starve the others of memory.
Please note that soft limits is a best-effort feature; it comes withno guarantees, but it does its best to make sure that when memory isheavily contended for, memory is allocated based on the soft limithints/setup. Currently soft limit based reclaim is set up such thatit gets invoked from balance_pgdat (kswapd).
7.1 Interface¶
Soft limits can be setup by using the following commands (in this example weassume a soft limit of 256 MiB):
# echo 256M > memory.soft_limit_in_bytes
If we want to change this to 1G, we can at any time use:
# echo 1G > memory.soft_limit_in_bytes
Note
Soft limits take effect over a long period of time, since they involvereclaiming memory for balancing between memory cgroups
Note
It is recommended to set the soft limit always below the hard limit,otherwise the hard limit will take precedence.
8. Move charges at task migration (DEPRECATED!)¶
THIS IS DEPRECATED!
It’s expensive and unreliable! It’s better practice to launch workloadtasks directly from inside their target cgroup. Use dedicated workloadcgroups to allow fine-grained policy adjustments without having tomove physical pages between control domains.
Users can move charges associated with a task along with task migration, thatis, uncharge task’s pages from the old cgroup and charge them to the new cgroup.This feature is not supported in !CONFIG_MMU environments because of lack ofpage tables.
8.1 Interface¶
This feature is disabled by default. It can be enabled (and disabled again) bywriting to memory.move_charge_at_immigrate of the destination cgroup.
If you want to enable it:
# echo (some positive value) > memory.move_charge_at_immigrate
Note
Each bits of move_charge_at_immigrate has its own meaning about what typeof charges should be moved. See section 8.2 for details.
Note
Charges are moved only when you move mm->owner, in other words,a leader of a thread group.
Note
If we cannot find enough space for the task in the destination cgroup, wetry to make space by reclaiming memory. Task migration may fail if wecannot make enough space.
Note
It can take several seconds if you move charges much.
And if you want disable it again:
# echo 0 > memory.move_charge_at_immigrate
8.2 Type of charges which can be moved¶
Each bit in move_charge_at_immigrate has its own meaning about what type ofcharges should be moved. But in any case, it must be noted that an account ofa page or a swap can be moved only when it is charged to the task’s current(old) memory cgroup.
bit | what type of charges would be moved ? |
|---|---|
A charge of an anonymous page (or swap of it) used by the target task.You must enable Swap Extension (see 2.4) to enable move of swap charges. | |
1 | A charge of file pages (normal file, tmpfs file (e.g. ipc shared memory)and swaps of tmpfs file) mmapped by the target task. Unlike the case ofanonymous pages, file pages (and swaps) in the range mmapped by the taskwill be moved even if the task hasn’t done page fault, i.e. they mightnot be the task’s “RSS”, but other task’s “RSS” that maps the same file.The mapcount of the page is ignored (the page can be moved independentof the mapcount). You must enable Swap Extension (see 2.4) toenable move of swap charges. |
8.3 TODO¶
All of moving charge operations are done under cgroup_mutex. It’s not goodbehavior to hold the mutex too long, so we may need some trick.
9. Memory thresholds¶
Memory cgroup implements memory thresholds using the cgroups notificationAPI (see Control Groups). It allows to register multiple memory and memswthresholds and gets notifications when it crosses.
To register a threshold, an application must:
create an eventfd using eventfd(2);
open memory.usage_in_bytes or memory.memsw.usage_in_bytes;
write string like “<event_fd> <fd of memory.usage_in_bytes> <threshold>” tocgroup.event_control.
Application will be notified through eventfd when memory usage crossesthreshold in any direction.
It’s applicable for root and non-root cgroup.
10. OOM Control¶
memory.oom_control file is for OOM notification and other controls.
Memory cgroup implements OOM notifier using the cgroup notificationAPI (See Control Groups). It allows to register multiple OOM notificationdelivery and gets notification when OOM happens.
To register a notifier, an application must:
create an eventfd using eventfd(2)
open memory.oom_control file
write string like “<event_fd> <fd of memory.oom_control>” tocgroup.event_control
The application will be notified through eventfd when OOM happens.OOM notification doesn’t work for the root cgroup.
You can disable the OOM-killer by writing “1” to memory.oom_control file, as:
#echo 1 > memory.oom_control
If OOM-killer is disabled, tasks under cgroup will hang/sleepin memory cgroup’s OOM-waitqueue when they request accountable memory.
For running them, you have to relax the memory cgroup’s OOM status by
enlarge limit or reduce usage.
To reduce usage,
kill some tasks.
move some tasks to other group with account migration.
remove some files (on tmpfs?)
Then, stopped tasks will work again.
At reading, current status of OOM is shown.
oom_kill_disable 0 or 1(if 1, oom-killer is disabled)
under_oom 0 or 1(if 1, the memory cgroup is under OOM, tasks may be stopped.)
oom_kill integer counterThe number of processes belonging to this cgroup killed by anykind of OOM killer.
11. Memory Pressure¶
The pressure level notifications can be used to monitor the memoryallocation cost; based on the pressure, applications can implementdifferent strategies of managing their memory resources. The pressurelevels are defined as following:
The “low” level means that the system is reclaiming memory for newallocations. Monitoring this reclaiming activity might be useful formaintaining cache level. Upon notification, the program (typically“Activity Manager”) might analyze vmstat and act in advance (i.e.prematurely shutdown unimportant services).
The “medium” level means that the system is experiencing medium memorypressure, the system might be making swap, paging out active file caches,etc. Upon this event applications may decide to further analyzevmstat/zoneinfo/memcg or internal memory usage statistics and free anyresources that can be easily reconstructed or re-read from a disk.
The “critical” level means that the system is actively thrashing, it isabout to out of memory (OOM) or even the in-kernel OOM killer is on itsway to trigger. Applications should do whatever they can to help thesystem. It might be too late to consult with vmstat or any otherstatistics, so it’s advisable to take an immediate action.
By default, events are propagated upward until the event is handled, i.e. theevents are not pass-through. For example, you have three cgroups: A->B->C. Nowyou set up an event listener on cgroups A, B and C, and suppose group Cexperiences some pressure. In this situation, only group C will receive thenotification, i.e. groups A and B will not receive it. This is done to avoidexcessive “broadcasting” of messages, which disturbs the system and which isespecially bad if we are low on memory or thrashing. Group B, will receivenotification only if there are no event listeners for group C.
There are three optional modes that specify different propagation behavior:
“default”: this is the default behavior specified above. This mode is thesame as omitting the optional mode parameter, preserved by backwardscompatibility.
“hierarchy”: events always propagate up to the root, similar to the defaultbehavior, except that propagation continues regardless of whether there areevent listeners at each level, with the “hierarchy” mode. In the aboveexample, groups A, B, and C will receive notification of memory pressure.
“local”: events are pass-through, i.e. they only receive notifications whenmemory pressure is experienced in the memcg for which the notification isregistered. In the above example, group C will receive notification ifregistered for “local” notification and the group experiences memorypressure. However, group B will never receive notification, regardless ifthere is an event listener for group C or not, if group B is registered forlocal notification.
The level and event notification mode (“hierarchy” or “local”, if necessary) arespecified by a comma-delimited string, i.e. “low,hierarchy” specifieshierarchical, pass-through, notification for all ancestor memcgs. Notificationthat is the default, non pass-through behavior, does not specify a mode.“medium,local” specifies pass-through notification for the medium level.
The file memory.pressure_level is only used to setup an eventfd. Toregister a notification, an application must:
create an eventfd using eventfd(2);
open memory.pressure_level;
write string as “<event_fd> <fd of memory.pressure_level> <level[,mode]>”to cgroup.event_control.
Application will be notified through eventfd when memory pressure is atthe specific level (or higher). Read/write operations tomemory.pressure_level are no implemented.
Test:
Here is a small script example that makes a new cgroup, sets up amemory limit, sets up a notification in the cgroup and then makes childcgroup experience a critical pressure:
# cd /sys/fs/cgroup/memory/# mkdir foo# cd foo# cgroup_event_listener memory.pressure_level low,hierarchy &# echo 8000000 > memory.limit_in_bytes# echo 8000000 > memory.memsw.limit_in_bytes# echo $$ > tasks# dd if=/dev/zero | read x(Expect a bunch of notifications, and eventually, the oom-killer willtrigger.)
12. TODO¶
Make per-cgroup scanner reclaim not-shared pages first
Teach controller to account for shared-pages
Start reclamation in the background when the limit isnot yet hit but the usage is getting closer
Summary¶
Overall, the memory controller has been a stable controller and has beencommented and discussed quite extensively in the community.
References¶
- 1
Singh, Balbir. RFC: Memory Controller, http://lwn.net/Articles/206697/
- 2
Singh, Balbir. Memory Controller (RSS Control),http://lwn.net/Articles/222762/
- 3
Emelianov, Pavel. Resource controllers based on process cgroupshttps://lore.kernel.org/r/45ED7DEC.7010403@sw.ru
- 4
Emelianov, Pavel. RSS controller based on process cgroups (v2)https://lore.kernel.org/r/461A3010.90403@sw.ru
- 5
Emelianov, Pavel. RSS controller based on process cgroups (v3)https://lore.kernel.org/r/465D9739.8070209@openvz.org
Menage, Paul. Control Groups v10, http://lwn.net/Articles/236032/
Vaidyanathan, Srinivasan, Control Groups: Pagecache accounting and controlsubsystem (v3), http://lwn.net/Articles/235534/
Singh, Balbir. RSS controller v2 test results (lmbench),https://lore.kernel.org/r/464C95D4.7070806@linux.vnet.ibm.com
Singh, Balbir. RSS controller v2 AIM9 resultshttps://lore.kernel.org/r/464D267A.50107@linux.vnet.ibm.com
Singh, Balbir. Memory controller v6 test results,https://lore.kernel.org/r/20070819094658.654.84837.sendpatchset@balbir-laptop
- 11
Singh, Balbir. Memory controller introduction (v6),https://lore.kernel.org/r/20070817084228.26003.12568.sendpatchset@balbir-laptop
- 12
Corbet, Jonathan, Controlling memory use in cgroups,http://lwn.net/Articles/243795/
