oom-killer being invoked instead of swapping

21 Apr 2017

      Hi,

I have a raspberry pi 2 on which I recently installed Ubuntu 16.04.2. 
After the install oom-killer has been killing processes, even when there 
is 1GB of swap free. I would have expected the kernel to swap something 
out rather than killing processes.

How can I use swap rather than having oom-killer killing processes?

Here is a typical oom-kill as recorded by syslog:

Apr 21 15:45:01 pi2 kernel: [72182.070173] sh invoked oom-killer: 
gfp_mask=0x24004c0, order=2, oom_score_adj=0
Apr 21 15:45:01 pi2 kernel: [72182.070196] sh cpuset=/ mems_allowed=0
Apr 21 15:45:01 pi2 kernel: [72182.070221] CPU: 2 PID: 10341 Comm: sh 
Not tainted 4.4.0-1052-raspi2 #59-Ubuntu
Apr 21 15:45:01 pi2 kernel: [72182.070228] Hardware name: BCM2709
Apr 21 15:45:01 pi2 kernel: [72182.070265] [<80111e94>] 
(unwind_backtrace) from [<8010d100>] (show_stack+0x20/0x24)
Apr 21 15:45:01 pi2 kernel: [72182.070286] [<8010d100>] (show_stack) 
from [<8056a52c>] (dump_stack+0xc8/0x10c)
Apr 21 15:45:01 pi2 kernel: [72182.070308] [<8056a52c>] (dump_stack) 
from [<802a888c>] (dump_header+0x68/0x1c8)
Apr 21 15:45:01 pi2 kernel: [72182.070327] [<802a888c>] (dump_header) 
from [<80247800>] (oom_kill_process+0x36c/0x4b4)
Apr 21 15:45:01 pi2 kernel: [72182.070344] [<80247800>] 
(oom_kill_process) from [<80247bb8>] (out_of_memory+0x1f8/0x35c)
Apr 21 15:45:01 pi2 kernel: [72182.070361] [<80247bb8>] (out_of_memory) 
from [<8024d93c>] (__alloc_pages_nodemask+0xd38/0xd78)
Apr 21 15:45:01 pi2 kernel: [72182.070378] [<8024d93c>] 
(__alloc_pages_nodemask) from [<8024d99c>] (__get_free_pages+0x20/0x64)
Apr 21 15:45:01 pi2 kernel: [72182.070395] [<8024d99c>] 
(__get_free_pages) from [<801190d4>] (pgd_alloc+0x28/0x11c)
Apr 21 15:45:01 pi2 kernel: [72182.070411] [<801190d4>] (pgd_alloc) from 
[<80121520>] (mm_init+0xd8/0x170)
Apr 21 15:45:01 pi2 kernel: [72182.070427] [<80121520>] (mm_init) from 
[<80121bbc>] (mm_alloc+0x5c/0x68)
Apr 21 15:45:01 pi2 kernel: [72182.070445] [<80121bbc>] (mm_alloc) from 
[<802b3ecc>] (do_execveat_common+0x280/0x744)
Apr 21 15:45:01 pi2 kernel: [72182.070463] [<802b3ecc>] 
(do_execveat_common) from [<802b4624>] (SyS_execve+0x40/0x48)
Apr 21 15:45:01 pi2 kernel: [72182.070481] [<802b4624>] (SyS_execve) 
from [<80108c60>] (ret_fast_syscall+0x0/0x1c)
Apr 21 15:45:01 pi2 kernel: [72182.070489] Mem-Info:
Apr 21 15:45:01 pi2 kernel: [72182.070510] active_anon:7968 
inactive_anon:36348 isolated_anon:0
Apr 21 15:45:01 pi2 kernel: [72182.070510]  active_file:35828 
inactive_file:127471 isolated_file:0
Apr 21 15:45:01 pi2 kernel: [72182.070510]  unevictable:0 dirty:5 
writeback:0 unstable:0
Apr 21 15:45:01 pi2 kernel: [72182.070510]  slab_reclaimable:15804 
slab_unreclaimable:4482
Apr 21 15:45:01 pi2 kernel: [72182.070510]  mapped:4644 shmem:1239 
pagetables:434 bounce:0
Apr 21 15:45:01 pi2 kernel: [72182.070510]  free:2363 free_pcp:0 free_cma:15
Apr 21 15:45:01 pi2 kernel: [72182.070554] Normal free:9452kB min:3836kB 
low:4792kB high:5752kB active_anon:31872kB inactive_anon:145392kB 
active_file:143312kB inactive_fi
le:509884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:966656kB managed:942128kB mlocked:0kB dirty:20kB writeback:0kB 
mapped:18576kB shmem:4956kB slab_r
eclaimable:63216kB slab_unreclaimable:17928kB kernel_stack:1656kB 
pagetables:1736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB 
free_cma:60kB writeback_tmp:0kB page
s_scanned:16 all_unreclaimable? no
Apr 21 15:45:01 pi2 kernel: [72182.070563] lowmem_reserve[]: 0 0
Apr 21 15:45:01 pi2 kernel: [72182.070577] Normal: 1251*4kB (UME) 
579*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 
0*2048kB 0*4096kB = 9636kB
Apr 21 15:45:01 pi2 kernel: [72182.070625] 165625 total pagecache pages
Apr 21 15:45:01 pi2 kernel: [72182.070635] 1119 pages in swap cache
Apr 21 15:45:01 pi2 kernel: [72182.070643] Swap cache stats: add 23103, 
delete 21984, find 25546/29364
Apr 21 15:45:01 pi2 kernel: [72182.070650] Free swap  = 1004812kB
Apr 21 15:45:01 pi2 kernel: [72182.070656] Total swap = 1048572kB
Apr 21 15:45:01 pi2 kernel: [72182.070663] 241664 pages RAM
Apr 21 15:45:01 pi2 kernel: [72182.070669] 0 pages HighMem/MovableOnly
Apr 21 15:45:01 pi2 kernel: [72182.070675] 6132 pages reserved
Apr 21 15:45:01 pi2 kernel: [72182.070681] 2048 pages cma reserved
Apr 21 15:45:01 pi2 kernel: [72182.070688] [ pid ]   uid  tgid 
total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Apr 21 15:45:01 pi2 kernel: [72182.070717] [  345]     0   345 1225      
228       6       0       40             0 systemd-journal
Apr 21 15:45:01 pi2 kernel: [72182.070730] [  383]     0   383 
3124       36       7       0        3             0 lvmetad
Apr 21 15:45:01 pi2 kernel: [72182.070744] [  431]     0   431 2997      
145       8       0      128         -1000 systemd-udevd
Apr 21 15:45:01 pi2 kernel: [72182.070758] [  684]   100   684 3064      
123       7       0       28             0 systemd-timesyn
Apr 21 15:45:01 pi2 kernel: [72182.070771] [  701]     0   701 592       
38       6       0        4             0 rpc.idmapd
Apr 21 15:45:01 pi2 kernel: [72182.070783] [  805]     0   805 834      
287       6       0       33             0 systemd-logind
Apr 21 15:45:01 pi2 kernel: [72182.070796] [  808]     0   808 346       
63       5       0        3             0 acpid
Apr 21 15:45:01 pi2 kernel: [72182.070808] [  815]     0   815 1130      
129       7       0       19             0 cron
Apr 21 15:45:01 pi2 kernel: [72182.070821] [  819]     0   819 7684      
292      10       0        1             0 lxcfs
Apr 21 15:45:01 pi2 kernel: [72182.070834] [  822]   107   822 1226      
296       6       0       48          -900 dbus-daemon
Apr 21 15:45:01 pi2 kernel: [72182.070846] [  831]   104   831 7794      
168       9       0      122             0 rsyslogd
Apr 21 15:45:01 pi2 kernel: [72182.070859] [  833]     0   833 9084      
635      12       0       63             0 accounts-daemon
Apr 21 15:45:01 pi2 kernel: [72182.070871] [  836]     0   836 712       
55       6       0       33             0 atd
Apr 21 15:45:01 pi2 kernel: [72182.070883] [  854]     0   854 8711      
305      13       0       57             0 polkitd
Apr 21 15:45:01 pi2 kernel: [72182.070896] [ 1002]   113  1002 29479     
1911      30       0     4528             0 named
Apr 21 15:45:01 pi2 kernel: [72182.070909] [ 1012]     0  1012 1997      
225       7       0      104         -1000 sshd
Apr 21 15:45:01 pi2 kernel: [72182.070922] [ 1015]   114  1015 2560      
865      10       0      945             0 dhcpd
Apr 21 15:45:01 pi2 kernel: [72182.070935] [ 1045]     0  1045 
1359       97       6       0       80             0 rpcbind
Apr 21 15:45:01 pi2 kernel: [72182.070948] [ 1049]     0  1049 1420      
516       6       0      119             0 rpc.mountd
Apr 21 15:45:01 pi2 kernel: [72182.070962] [ 1143]     0  1143 832      
118       6       0       17             0 irqbalance
Apr 21 15:45:01 pi2 kernel: [72182.070974] [ 1170]     0  1170 862       
61       5       0       25             0 agetty
Apr 21 15:45:01 pi2 kernel: [72182.070987] [ 1172]     0  1172 908       
10       6       0       19             0 agetty
Apr 21 15:45:01 pi2 kernel: [72182.071000] [ 1254]     0  1254 10969     
1791      16       0      710             0 fail2ban-server
Apr 21 15:45:01 pi2 kernel: [72182.071012] [ 1263]     0  1263 2225     
1240       8       0       45             0 apache2
Apr 21 15:45:01 pi2 kernel: [72182.071025] [ 1358]     0  1358 7968      
143       9       0       65             0 master
Apr 21 15:45:01 pi2 kernel: [72182.071037] [ 1360]   112  1360 8020      
335       9       0       45             0 qmgr
Apr 21 15:45:01 pi2 kernel: [72182.071050] [ 1716]  1000  1716 
1581       32       8       0      409             0 systemd
Apr 21 15:45:01 pi2 kernel: [72182.071062] [ 1720]  1000  1720 
1813       33       7       0      397             0 (sd-pam)
Apr 21 15:45:01 pi2 kernel: [72182.071075] [ 1912]     0  1912 2945      
119      11       0      165             0 sshd
Apr 21 15:45:01 pi2 kernel: [72182.071088] [ 1946]  1000  1946 4041     
1235      14       0      146             0 sshd
Apr 21 15:45:01 pi2 kernel: [72182.071100] [ 1947]  1000  1947 36295    
35191      75       0       10             0 unison
Apr 21 15:45:01 pi2 kernel: [72182.071114] [ 6694]    33  6694 
58348      900      36       0       45             0 apache2
Apr 21 15:45:01 pi2 kernel: [72182.071127] [ 6695]    33  6695 
58092      899      36       0       45             0 apache2
Apr 21 15:45:01 pi2 kernel: [72182.071141] [10335]   112 10335 7986      
721       8       0        0             0 pickup
Apr 21 15:45:01 pi2 kernel: [72182.071153] [10337]     0 10337 1470      
504       7       0        5             0 cron
Apr 21 15:45:01 pi2 kernel: [72182.071166] [10338]  1000 10338 365      
101       4       0        0             0 sh
Apr 21 15:45:01 pi2 kernel: [72182.071179] [10339]  1000 10339 3084     
2179      10       0        0             0 optus-status
Apr 21 15:45:01 pi2 kernel: [72182.071191] [10340]  1000 10340 365       
93       5       0        0             0 sh
Apr 21 15:45:01 pi2 kernel: [72182.071203] [10341]  1000 10341 365       
16       4       0        0             0 sh
Apr 21 15:45:01 pi2 kernel: [72182.071212] Out of memory: Kill process 
1947 (unison) score 70 or sacrifice child
Apr 21 15:45:01 pi2 kernel: [72182.078994] Killed process 1947 (unison) 
total-vm:145180kB, anon-rss:139988kB, file-rss:776kB
Apr 21 15:45:02 pi2 systemd[1]: Stopping User Manager for UID 1000...
Apr 21 15:45:02 pi2 systemd[1716]: Stopped target Default.
Apr 21 15:45:02 pi2 systemd[1716]: Reached target Shutdown.
Apr 21 15:45:02 pi2 systemd[1716]: Stopped target Basic System.
Apr 21 15:45:02 pi2 systemd[1716]: Stopped target Paths.
Apr 21 15:45:02 pi2 systemd[1716]: Stopped target Sockets.
Apr 21 15:45:02 pi2 systemd[1716]: Stopped target Timers.
Apr 21 15:45:02 pi2 systemd[1716]: Starting Exit the Session...
Apr 21 15:45:02 pi2 systemd[1716]: Received SIGRTMIN+24 from PID 10345 
(kill).
Apr 21 15:45:02 pi2 systemd[1]: Stopped User Manager for UID 1000.
Apr 21 15:45:02 pi2 systemd[1]: Removed slice User Slice of kim.

Kim Oldfield

tags

participants (1)