21 Dec
2015
21 Dec
'15
12:10 p.m.
/* No systemd religious wars please - thank you */
Hi folks,
I'm trying to get Slurm working on a RHEL7.2 system and I've hit an issue where
systemd is already using the cpuacct cgroup hierarchy and that prevents Slurm
from using it as it seems to be the one case where it can only be in use once.
I.e. having this mount create by systemd:
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
causes:
12725 mount("cgroup", "/cgroup/cpuacct", "cgroup",
MS_NOSUID|MS_NODEV|MS_NOEXEC, "cpuacct") = -1 EBUSY (Device or resource busy)
Anyone got any ideas on how to stop systemd from using it?
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
21 Dec
21 Dec
7:52 p.m.
Chris Samuel via luv-main <luv-main(a)luv.asn.au> wrote:
I'm trying to get Slurm working on a RHEL7.2
system and I've hit an issue where
systemd is already using the cpuacct cgroup hierarchy and that prevents Slurm
from using it as it seems to be the one case where it can only be in use once.
While this doesn't directly answer your question, does anything on the
following page provide a clue?
https://wiki.archlinux.org/index.php/Slurm
It's clear from this page that Slurm and Systemd are compatible.
Upon checking an Arch Linux system:
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
shows that the control group is mounted.
This doesn't tell us how they worked around the issue. Do you have the Slurm
unit file for systemd? This might contain the necessary ingredient, but that's
just a guess on my part.
22 Dec
22 Dec
12:35 p.m.
On Mon, 21 Dec 2015 02:52:04 PM Jason White via luv-main wrote:
While this doesn't directly answer your question,
does anything on the
following page provide a clue?
https://wiki.archlinux.org/index.php/Slurm
Hmm, don't trust any page that links to the LLNL docs, they're really old & no
longer apply to current versions, that page needs to be fixed to point to
http://slurm.schedmd.com/ as that's where the developers are now (and yes,
it's still GPL).
This doesn't tell us how they worked around the
issue. Do you have the Slurm
unit file for systemd? This might contain the necessary ingredient, but
that's just a guess on my part.
Yeah, I've got those (they're in the source distribution) but it is a bit more
involved than that. I've just got this email on the slurm-devel list that
essentially confirms and expands on what Clinton mentioned:
http://permalink.gmane.org/gmane.comp.distributed.slurm.devel/9100
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
8:59 a.m.
On 21 December 2015 at 22:10, Chris Samuel via luv-main <luv-main(a)luv.asn.au
wrote:
Anyone got any ideas on how to stop systemd from using it?
I don't know slurm at all, but it appears to have a cgroups.conf file, and
something like:
CgroupMountpoint=/sys
CgroupAutomount=no
Might do what you need?
This doesn't stop systmed from mounting the cgroups, it just tells slurm
where they're
already mounted.
--
Clinton Roy
Software Developer
Netboxblue.com
Total Internet Management
Control Internet usage within and outside your organisation.
12:22 p.m.
On Tue, 22 Dec 2015 06:59:24 PM Clinton Roy via luv-main wrote:
Might do what you need?
An interesting idea.
This doesn't stop systmed from mounting the
cgroups, it just tells slurm
where they're already mounted.
To be honest on an HPC node it's the batch system that should be controlling
the cgroups rather than systemd.
What would be ideal (if you can sacrifice the cores) would be to tell systemd
to just use a single core for all its stuff and leave everything else
available for Slurm to manage.
That way there's no OS jitter from system daemons running on the same cores as
the applications themselves.
I know, I can dream.. ;-)
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
12:38 p.m.
Hi Chris,
On Mon, Dec 21, 2015 at 11:10:47PM +1100, Chris Samuel via luv-main wrote:
I'm trying to get Slurm working on a RHEL7.2 system
and I've hit an issue where
systemd is already using the cpuacct cgroup hierarchy and that prevents Slurm
from using it as it seems to be the one case where it can only be in use once.
I.e. having this mount create by systemd:
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
causes:
12725 mount("cgroup", "/cgroup/cpuacct", "cgroup",
MS_NOSUID|MS_NODEV|MS_NOEXEC, "cpuacct") = -1 EBUSY (Device or resource busy)
I suspect if slurm mounted the cgroup as cpu,cpuacct then it would
work. but it's because it's trying just cpuacct that it's failing.
that's the sort of behaviour I've seen before.
the OS appears to bind mount them if they're the same.
you have the slurm source - care to hack it and give that a go? :)
torque 6's cgroup code uses cpu,cpuacct which I presume is in order to
play nicely with rhel7.
cheers,
robin
12:46 p.m.
On Tue, 22 Dec 2015 07:38:15 AM Robin Humble via luv-main wrote:
I suspect if slurm mounted the cgroup as cpu,cpuacct
then it would
work. but it's because it's trying just cpuacct that it's failing.
that's the sort of behaviour I've seen before.
the OS appears to bind mount them if they're the same.
Bingo! You are a star sir. :-)
[root@spartan001 ~]# mount -v -t cgroup -o cpuacct none /cgroup/cpuacct
mount: none is already mounted or /cgroup/cpuacct busy
[root@spartan001 ~]# mount -v -t cgroup -o cpu,cpuacct none /cgroup/cpuacct
mount: none mounted on /cgroup/cpuacct.
you have the slurm source - care to hack it and give
that a go?
I suspect I'll just open a bug with SchedMD tomorrow and let them fix that,
I've got my hands with my other work full until next year now.
torque 6's cgroup code uses cpu,cpuacct which I
presume is in order to
play nicely with rhel7.
Yeah, sounds like it..
Thanks Robin! :-)
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
2539
days inactive
2540
days old
6 comments
4 participants
participants (4)
-
Chris Samuel -
Clinton Roy -
Jason White -
Robin Humble