Getting systemd to not use cpuacct cgroup on RHEL 7.2
/* No systemd religious wars please - thank you */ Hi folks, I'm trying to get Slurm working on a RHEL7.2 system and I've hit an issue where systemd is already using the cpuacct cgroup hierarchy and that prevents Slurm from using it as it seems to be the one case where it can only be in use once. I.e. having this mount create by systemd: cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0 causes: 12725 mount("cgroup", "/cgroup/cpuacct", "cgroup", MS_NOSUID|MS_NODEV|MS_NOEXEC, "cpuacct") = -1 EBUSY (Device or resource busy) Anyone got any ideas on how to stop systemd from using it? All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
Chris Samuel via luv-main <luv-main@luv.asn.au> wrote:
I'm trying to get Slurm working on a RHEL7.2 system and I've hit an issue where systemd is already using the cpuacct cgroup hierarchy and that prevents Slurm from using it as it seems to be the one case where it can only be in use once.
While this doesn't directly answer your question, does anything on the following page provide a clue? https://wiki.archlinux.org/index.php/Slurm It's clear from this page that Slurm and Systemd are compatible. Upon checking an Arch Linux system: cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) shows that the control group is mounted. This doesn't tell us how they worked around the issue. Do you have the Slurm unit file for systemd? This might contain the necessary ingredient, but that's just a guess on my part.
On Mon, 21 Dec 2015 02:52:04 PM Jason White via luv-main wrote:
While this doesn't directly answer your question, does anything on the following page provide a clue? https://wiki.archlinux.org/index.php/Slurm
Hmm, don't trust any page that links to the LLNL docs, they're really old & no longer apply to current versions, that page needs to be fixed to point to http://slurm.schedmd.com/ as that's where the developers are now (and yes, it's still GPL).
This doesn't tell us how they worked around the issue. Do you have the Slurm unit file for systemd? This might contain the necessary ingredient, but that's just a guess on my part.
Yeah, I've got those (they're in the source distribution) but it is a bit more involved than that. I've just got this email on the slurm-devel list that essentially confirms and expands on what Clinton mentioned: http://permalink.gmane.org/gmane.comp.distributed.slurm.devel/9100 All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
On 21 December 2015 at 22:10, Chris Samuel via luv-main <luv-main@luv.asn.au
wrote:
Anyone got any ideas on how to stop systemd from using it?
I don't know slurm at all, but it appears to have a cgroups.conf file, and something like: CgroupMountpoint=/sys CgroupAutomount=no Might do what you need? This doesn't stop systmed from mounting the cgroups, it just tells slurm where they're already mounted. -- Clinton Roy Software Developer Netboxblue.com Total Internet Management Control Internet usage within and outside your organisation.
On Tue, 22 Dec 2015 06:59:24 PM Clinton Roy via luv-main wrote:
Might do what you need?
An interesting idea.
This doesn't stop systmed from mounting the cgroups, it just tells slurm where they're already mounted.
To be honest on an HPC node it's the batch system that should be controlling the cgroups rather than systemd. What would be ideal (if you can sacrifice the cores) would be to tell systemd to just use a single core for all its stuff and leave everything else available for Slurm to manage. That way there's no OS jitter from system daemons running on the same cores as the applications themselves. I know, I can dream.. ;-) -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
Hi Chris, On Mon, Dec 21, 2015 at 11:10:47PM +1100, Chris Samuel via luv-main wrote:
I'm trying to get Slurm working on a RHEL7.2 system and I've hit an issue where systemd is already using the cpuacct cgroup hierarchy and that prevents Slurm from using it as it seems to be the one case where it can only be in use once.
I.e. having this mount create by systemd:
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
causes:
12725 mount("cgroup", "/cgroup/cpuacct", "cgroup", MS_NOSUID|MS_NODEV|MS_NOEXEC, "cpuacct") = -1 EBUSY (Device or resource busy)
I suspect if slurm mounted the cgroup as cpu,cpuacct then it would work. but it's because it's trying just cpuacct that it's failing. that's the sort of behaviour I've seen before. the OS appears to bind mount them if they're the same. you have the slurm source - care to hack it and give that a go? :) torque 6's cgroup code uses cpu,cpuacct which I presume is in order to play nicely with rhel7. cheers, robin
On Tue, 22 Dec 2015 07:38:15 AM Robin Humble via luv-main wrote:
I suspect if slurm mounted the cgroup as cpu,cpuacct then it would work. but it's because it's trying just cpuacct that it's failing. that's the sort of behaviour I've seen before.
the OS appears to bind mount them if they're the same.
Bingo! You are a star sir. :-) [root@spartan001 ~]# mount -v -t cgroup -o cpuacct none /cgroup/cpuacct mount: none is already mounted or /cgroup/cpuacct busy [root@spartan001 ~]# mount -v -t cgroup -o cpu,cpuacct none /cgroup/cpuacct mount: none mounted on /cgroup/cpuacct.
you have the slurm source - care to hack it and give that a go?
I suspect I'll just open a bug with SchedMD tomorrow and let them fix that, I've got my hands with my other work full until next year now.
torque 6's cgroup code uses cpu,cpuacct which I presume is in order to play nicely with rhel7.
Yeah, sounds like it.. Thanks Robin! :-) -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
participants (4)
-
Chris Samuel -
Clinton Roy -
Jason White -
Robin Humble