Link Search Menu Expand Document

Using userspace tools to manage cgroups and persist changes

  • Working with the cgroups subsystems by manipulating directories and files directly is a fast and convenient way to prototype and test changes, however, this comes with few drawbacks, namely the changes made will not persist a server restart and there’s not much error reporting or handling.

  • To address this, there are packages that provide userspace tools and daemons that are quite easy to use. Let’s see a few examples.

  • To install the tools on Debian/Ubuntu, run the following:

root@server:~# apt-get install -y cgroup-bin cgroup-lite libcgroup1
root@server:~# service cgroup-lite start

On RHEL/CentOS, execute the following:

root@server:~# yum install libcgroup
root@server:~# service cgconfig start

To mount all subsystems, run the following:

root@server:~# cgroups-mount
root@server:~# cat /proc/mounts | grep cgroup
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuset,clone_children 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu,release_agent=/run/cgmanager/agents/cgm-release-agent.cpu 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct 0 0


Notice from the preceding output the location of the cgroups - /sys/fs/cgroup. This is the default location on many Linux distributions and in most cases the various subsystems have already been mounted.

To verify what cgroup subsystems are in use, we can check with the following commands:

root@server:~# cat /proc/cgroups
#subsys_name  hierarchy  num_cgroups  enabled
cpuset  7  1  1
cpu  8  2  1
cpuacct  9  1  1
memory  10  2  1
devices  11  1  1
freezer  12  1  1
blkio  6  3  1
perf_event  13  1  1
hugetlb  14  1  1


Next, let’s create a blkio hierarchy and add an already running process to it with cgclassify. This is similar to what we did earlier, by creating the directories and the files by hand:

root@server:~# cgcreate -g blkio:high_io
root@server:~# cgcreate -g blkio:low_io
root@server:~# cgclassify -g blkio:low_io $(pidof app1)
root@server:~# cat /sys/fs/cgroup/blkio/low_io/tasks
8052
root@server:~# cgset -r blkio.weight=1000 high_io
root@server:~# cgset -r blkio.weight=100 low_io
root@server:~# cat /sys/fs/cgroup/blkio/high_io/blkio.weight
1000
root@server:~#


Now that we have defined the high_io and low_io cgroups and added a process to them, let’s generate a configuration file that can be used later to reapply the setup:

root@server:~# cgsnapshot -s -f /tmp/cgconfig_io.conf
cpuset = /sys/fs/cgroup/cpuset;
cpu = /sys/fs/cgroup/cpu;
cpuacct = /sys/fs/cgroup/cpuacct;
memory = /sys/fs/cgroup/memory;
devices = /sys/fs/cgroup/devices;
freezer = /sys/fs/cgroup/freezer;
blkio = /sys/fs/cgroup/blkio;
perf_event = /sys/fs/cgroup/perf_event;
hugetlb = /sys/fs/cgroup/hugetlb;
root@server:~# cat /tmp/cgconfig_io.conf
# Configuration file generated by cgsnapshot
mount {
    blkio = /sys/fs/cgroup/blkio;
}
group low_io {
    blkio {
        blkio.leaf_weight="500";
    blkio.leaf_weight_device="";
    blkio.weight="100";
    blkio.weight_device="";
    blkio.throttle.write_iops_device="";
    blkio.throttle.read_iops_device="";
    blkio.throttle.write_bps_device="";
    blkio.throttle.read_bps_device="";
    blkio.reset_stats="";
  }
}
group high_io {
blkio {
    blkio.leaf_weight="500";
    blkio.leaf_weight_device="";
    blkio.weight="1000";
    blkio.weight_device="";
    blkio.throttle.write_iops_device="";
    blkio.throttle.read_iops_device="";
    blkio.throttle.write_bps_device="";
    blkio.throttle.read_bps_device="";
    blkio.reset_stats="";
  }
}
root@server:~#


To start a new process in the high_io group, we can use the cgexec command:

root@server:~# cgexec -g blkio:high_io bash
root@server:~# echo $$
19654
root@server:~# cat /sys/fs/cgroup/blkio/high_io/tasks
19654
root@server:~#

In the preceding example, we started a new bash process in the high_io cgroup, as confirmed by looking at the tasks file.

To move an already running process to the memory subsystem, first we create the high_prio and low_prio groups and move the task with cgclassify:


root@server:~# cgcreate -g cpu,memory:high_prio
root@server:~# cgcreate -g cpu,memory:low_prio
root@server:~# cgclassify -g cpu,memory:high_prio 8052
root@server:~# cat /sys/fs/cgroup/memory/high_prio/tasks
8052
root@server:~# cat /sys/fs/cgroup/cpu/high_prio/tasks
8052
root@server:~#


To set the memory and CPU limits, we can use the cgset command. In contrast, remember that we used the echo command to manually move the PIDs and memory limits to the tasks and the memory.limit_in_bytes files:

root@server:~# cgset -r memory.limit_in_bytes=1G low_prio
root@server:~# cat /sys/fs/cgroup/memory/low_prio/memory.limit_in_bytes
1073741824
root@server:~# cgset -r cpu.shares=1000 high_prio
root@server:~# cat /sys/fs/cgroup/cpu/high_prio/cpu.shares
1000
root@server:~#

To see how the cgroup hierarchies look, we can use the lscgroup utility:

root@server:~# lscgroup
cpuset:/
cpu:/
cpu:/low_prio
cpu:/high_prio
cpuacct:/
memory:/
memory:/low_prio
memory:/high_prio
devices:/
freezer:/
blkio:/
blkio:/low_io
blkio:/high_io
perf_event:/
hugetlb:/
root@server:~#

The preceding output confirms the existence of the blkio, memory, and cpu hierarchies and their children.

Once finished, you can delete the hierarchies with cgdelete, which deletes the respective directories on the VFS:

root@server:~# cgdelete -g cpu,memory:high_prio
root@server:~# cgdelete -g cpu,memory:low_prio
root@server:~# lscgroup
cpuset:/
cpu:/
cpuacct:/
memory:/
devices:/
freezer:/
blkio:/
blkio:/low_io
blkio:/high_io
perf_event:/
hugetlb:/
root@server:~#

To completely clear the cgroups, we can use the cgclear utility, which will unmount the cgroup directories:

root@server:~# cgclear
root@server:~# lscgroup
cgroups can't be listed: Cgroup is not mounted
root@server:~#