25 Apr
2018
25 Apr
'18
12:46 p.m.
# free -m
total used free shared buff/cache available
Mem: 7962 2212 498 533 5251 4942
Swap: 10719 1732 8986
The above is from my workstation. It's running KDE, Chrome, KTorrent, and not
much else. My understanding of the above is that most RAM is being used for
cache and it's quite likely that this achieves the goal of reducing the number
of storage accesses.
The problem is that I don't want to reduce the number of storage accesses, I
want to improve the performance of interactive tasks. Ktorrent is configured
to only upload 60KB/s so a lack of caching of the torrents shouldn't prevent
it from uploading at the maximum speed I permit. When large interactive
programs like Chrome and Kmail get paged out it causes annoying delays when I
want to perform what should be quick tasks like replying to a single message
or viewing a single web page.
Any suggestions as to how to optimise for this use case? I already have swap
on one of the fastest SSDs I own and don't feel like buying NVMe for this
purpose or buying a system with more RAM, so software changes are required.
When replying please feel free to diverge from the topic. I think this is an
area where most Linux users know less than they would like so randomly
educational replies will be appreciated even if they don't help me with this
problem.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
25 Apr
25 Apr
1:42 p.m.
On Wed, Apr 25, 2018 at 10:46:15PM +1000, luv-main wrote:
# free -m
total used free shared buff/cache available
Mem: 7962 2212 498 533 5251 4942
Swap: 10719 1732 8986
The above is from my workstation. It's running KDE, Chrome, KTorrent, and not
much else. My understanding of the above is that most RAM is being used for
cache and it's quite likely that this achieves the goal of reducing the number
of storage accesses.
The problem is that I don't want to reduce the number of storage accesses, I
want to improve the performance of interactive tasks. Ktorrent is configured
to only upload 60KB/s so a lack of caching of the torrents shouldn't prevent
it from uploading at the maximum speed I permit. When large interactive
programs like Chrome and Kmail get paged out it causes annoying delays when I
want to perform what should be quick tasks like replying to a single message
or viewing a single web page.
Any suggestions as to how to optimise for this use case? I already have swap
on one of the fastest SSDs I own and don't feel like buying NVMe for this
purpose or buying a system with more RAM, so software changes are required.
When replying please feel free to diverge from the topic. I think this is an
area where most Linux users know less than they would like so randomly
educational replies will be appreciated even if they don't help me with this
problem.
Wow. I've never seen output from free like that before.
My understanding has always been that you only get paging after cache is
exhausted, but maybe I have been mistaken all this time.
If you run without swap, does the OOM killer come in?
Does top show plausibly large working sets for your apps e.g. such that they
wouldn't all fit in RAM together?
I used to have probems with Firefox using all of 1 cpu and (sometimes) all of
RAM as well, but not since I unpinned dilbert.com.
HTH
Cheers ... Duncan.
9:42 p.m.
# free -m
total used free shared buff/cache available
Mem: 7962 2212 498 533 5251 4942
Swap: 10719 1732 8986
The above is from my workstation. It's running KDE, Chrome, KTorrent, and
not much else. My understanding of the above is that most RAM is being
used for cache and it's quite likely that this achieves the goal of reducing the
number of storage accesses.
The problem is that I don't want to reduce the number of storage accesses, I
want to improve the performance of interactive tasks. Ktorrent is configured
to only upload 60KB/s so a lack of caching of the torrents shouldn't prevent it
from uploading at the maximum speed I permit. When large interactive
programs like Chrome and Kmail get paged out it causes annoying delays
when I want to perform what should be quick tasks like replying to a single
message or viewing a single web page.
Any suggestions as to how to optimise for this use case? I already have swap
on one of the fastest SSDs I own and don't feel like buying NVMe for this
purpose or buying a system with more RAM, so software changes are
required.
Have you done the analysis to confirm your suspicions? Iostat while switching to Chrome
etc?
I believe Linux will prematurely swap stuff out without invalidating the actual memory
pages, so that when it actually needs the memory it can just use it without first having
to wait for it to be written to disk, and conversely if the original owner of the page
needs the memory back it's still there. While a page is in this stage - ready to be
used for something else but still containing a valid copy of the original page - I
don't know how it shows in the output of free. It is simultaneously "paged
out", "free", and "in use", so I think maybe some further
analysis is required to see exactly what is going on.
Based on the above output of free, you could try leaving Chrome alone for a while, then
when you want to use it, turn swap off then on again, then see if Chrome is fast to switch
to. If not, then the delay might be coming from elsewhere.
If you just want to tinker, start with lowering the "swappiness" control.
James
26 Apr
26 Apr
12:47 a.m.
Is swappiness setting in Linux what you are looking for?
How to change the Swappiness of your Linux system
https://www.howtoforge.com/tutorial/linux-swappiness/
Daniel.
On Wed, 2018-04-25 at 22:46 +1000, Russell Coker via luv-main wrote:
# free -m
total used free shared buff/cache ava
ilable
Mem: 7962 2212 498 533 5251
4942
Swap: 10719 1732 8986
The above is from my workstation. It's running KDE, Chrome,
KTorrent, and not
much else. My understanding of the above is that most RAM is being
used for
cache and it's quite likely that this achieves the goal of reducing
the number
of storage accesses.
The problem is that I don't want to reduce the number of storage
accesses, I
want to improve the performance of interactive tasks. Ktorrent is
configured
to only upload 60KB/s so a lack of caching of the torrents shouldn't
prevent
it from uploading at the maximum speed I permit. When large
interactive
programs like Chrome and Kmail get paged out it causes annoying
delays when I
want to perform what should be quick tasks like replying to a single
message
or viewing a single web page.
Any suggestions as to how to optimise for this use case? I already
have swap
on one of the fastest SSDs I own and don't feel like buying NVMe for
this
purpose or buying a system with more RAM, so software changes are
required.
When replying please feel free to diverge from the topic. I think
this is an
area where most Linux users know less than they would like so
randomly
educational replies will be appreciated even if they don't help me
with this
problem.
9 Jul
9 Jul
11:19 a.m.
On Thursday, 26 April 2018 10:47:52 AM AEST Daniel Jitnah via luv-main wrote:
Is swappiness setting in Linux what you are looking
for?
How to change the Swappiness of your Linux system
https://www.howtoforge.com/tutorial/linux-swappiness/
Thanks Daniel and James.
I changed the swappiness of the system in question by putting vm.swappiness =
10 in /etc/sysctl.conf. Generally the performance seems better since doing
that. However there are some downsides, after running for a while with a low
value I've started seeing the following errors in the kernel message log when
using mplayer while running kernel 4.16.0-1-amd64. When I play a 720p video
(EG one downloaded from SBS on demand with youtube-dl) it works fine (although
usually having that kernel error), but a FullHD video downloaded from YouTube
causes extremely poor performance (hangs of a second every few seconds and
jittering of the video) and is unusable.
This isn't a total surprise, presumably the defaults were set for a reason and
mplayer causes the X server to do some things that it doesn't do in other
situations. Maybe I would have the same kernel issue if I tried 3D gaming,
but I haven't done that much in recent times.
I really need to get more RAM.
[3175207.254191] radeon 0000:01:00.0: swiotlb buffer is full (sz: 2097152
bytes)
[3175207.254196] radeon 0000:01:00.0: swiotlb: coherent allocation failed,
size=2097152
[3175207.254199] CPU: 5 PID: 1199 Comm: Xorg Not tainted 4.16.0-1-amd64 #1
Debian 4.16.5-1
[3175207.254201] Hardware name: System manufacturer System Product Name/P8H61-
M LE, BIOS 0703 06/22/2011
[3175207.254202] Call Trace:
[3175207.254211] dump_stack+0x5c/0x85
[3175207.254215] swiotlb_alloc_coherent+0x19b/0x1b0
[3175207.254223] ttm_dma_pool_get_pages+0x235/0x620 [ttm]
[3175207.254229] ttm_dma_populate+0x25e/0x350 [ttm]
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
10 Jul
10 Jul
3:24 a.m.
New subject: radeon video acceleration (was Re: swap vs cache)
NOTE: I am not an expert with video decoding, but hopefully some of the
following will be of use in improving your video playback...
On Mon, Jul 09, 2018 at 09:19:34PM +1000, Russell Coker wrote:
[...] I've started seeing the following errors in
the kernel message log
when using mplayer while running kernel 4.16.0-1-amd64. When I play a 720p
video (EG one downloaded from SBS on demand with youtube-dl) it works fine
(although usually having that kernel error), but a FullHD video downloaded
from YouTube causes extremely poor performance (hangs of a second every few
seconds and jittering of the video) and is unusable.
What kind of GPU do you have? an AMD Radeon, obviosly from the dmesg output
but what model ("lspci | grep VGA" if you're not sure)
Are you using VAAPI or VDPAU with mplayer? VDPAU was originally
nvidia-specific video acceleration API but for many years now there's been
open source libs to support intel and amd GPUs.
I use nvidia GPUs with the proprietary nvidia driver so vdpau stuff "Just
Works"...and I don't have any direct experience using vdpau with other
hardware or drivers, so consider the following a pointer in roughly the right
direction rather than detailed instructions:
You'll need to install at least the following packages if they're not already
installed:
apt-get install vdpauinfo libvdpau1 libdrm-amdgpu1 libvdpau-va-gl1
libvdpau-va-gl1 allows some programs using the VDPAU API to use OpenGL and/or
VA-API acceleration. To see what kind of video acceleration is supported
by your hardware & radeon driver, run "vdpauinfo". You may need to run it
as "VDPAU_DRIVER=va_gl vdpauinfo" to set the env var before running it (see
/usr/share/doc/libvdpau-va-gl1/README.Debian)
This should show at least mpeg-2, mpeg-4, and probably h.264 acceleration. If
it does, edit your ~/.mplayer/config to add
vo=vdpau,
vc=ffh264vdpau,ffmpeg12vdpau,ffodivxvdpau,ffwmv3vdpau,ffvc1vdpau,ffhevcvdpau
The 'vc=' line will probably need tweaking to match your hardware & drivers.
Playing with mplayer's cache settings might help too - e.g. 'cache = 8192'.
BTW, what kind of video formats are used by the 720p and Full HD videos? If
the 720p is mpeg4 or h.264 while the Full HD is h.265 that may explain the
problem - AFAIK, HW-accelerated h.265 decoding isn't supported on anything
except the proprietary nvidia driver with a geforce 900 series GPU or newer
(e.g. GTX-9xx or GTX-10xx).
Without HW acceleration, it falls back to software decoding, much slower
than using the GPU HW. You may be able to configure mplayer to use
more CPU cores/threads to speed it up (e.g. with 'lavdopts=threads=2'
in ~/.mplayer/config). or use mpv instead of mplayer, which IIRC does
multi-threading by default.
Try running 'mediainfo' on the video files. 'apt-get install mediainfo'
if
it's not already installed. This will tell you what kind of container it is,
and what kind of video & audio codecs were used (and also if it contains any
text streams for subtitles)
Also worth considering: if your machine struggles to play Full HD video, but
has no problem with 720p then don't make it play Full HD - stick to 720p or
less. lower resolution that plays without stuttering or pausing is infinitely
better than high-res that doesn't play smoothly.
Anyway, when you run mplayer from the command line, check its output to see
what it's using. if vdpau is enabled and working on your system, You should
see it mentioned in mplayer's terminal output. e.g. on my system:
[...irrelevant junk deleted...]
libavformat version 57.71.100 (internal)
libavformat file format detected.
[lavf] stream 0: video (h264), -vid 0
[lavf] stream 1: audio (aac), -aid 0
VIDEO: [H264] 720x404 0bpp 23.976 fps 0.0 kbps ( 0.0 kbyte/s)
==========================================================================
Forced video codec: ffh264vdpau
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 57.89.100 (internal)
Selected video codec: [ffh264vdpau] vfm: ffmpeg (FFmpeg H.264 (VDPAU))
==========================================================================
Clip info:
encoder: libebml v1.3.3 + libmatroska v1.4.4
creation_time: 2018-06-28T04:53:20.000000Z
Load subtitles in /home/cas/video/
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 2 ch, floatle, 0.0 kbit/0.00% (ratio: 0->384000)
Selected audio codec: [ffaac] afm: ffmpeg (FFmpeg AAC (MPEG-2/MPEG-4 Audio))
==========================================================================
AO: [pulse] 48000Hz 2ch floatle (4 bytes per sample)
Starting playback...
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [vdpau] 720x404 => 720x404 H.264 VDPAU acceleration
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [vdpau] 720x404 => 720x404 H.264 VDPAU acceleration
A: 1.8 V: 1.8 A-V: 0.001 ct: 0.031 0/ 0 16% 2% 0.5% 8 0 98%
You may also want to try using smplayer. It's no more or less capable than
mplayer, but may be easier to configure. smplayer certainly - it's a GUI
wrapper around mplayer (or mpv) with configuration dialogs...which is a lot
easier to make sense of than mplayer's extraordinarily dense man page (the
info's all there in man but doesn't make much sense unless you already know a
lot about video playback and codecs and acceleration, which I don't...my video
knowledge is cherry-picked googling and cargo-culting until it kind of works.
Worse, this stuff changes all the time so it's very hard to keep current with
what works and what doesn't).
This isn't a total surprise, presumably the
defaults were set for a reason and
mplayer causes the X server to do some things that it doesn't do in other
situations. Maybe I would have the same kernel issue if I tried 3D gaming,
but I haven't done that much in recent times.
I really need to get more RAM.
what you're describing really doesn't look like a memory shortage. Your
original post back in April showed around 5GB of RAM free.
Chromium can be a bloated pig (on my system, it's currently using 9.5GB of
RAM and often uses 13GB or more (At the moment, I have about a dozen chromium
windows with a total of 112 tabs open), but if that were your problem you
would see very little free RAM, very little RAM used by buffers/cache, and
LOTS of swap in use. None of that seems to be the case from your original
message.
[3175207.254191] radeon 0000:01:00.0: swiotlb buffer
is full (sz: 2097152 bytes)
[3175207.254196] radeon 0000:01:00.0: swiotlb: coherent allocation failed, size=2097152
[3175207.254199] CPU: 5 PID: 1199 Comm: Xorg Not tainted 4.16.0-1-amd64 #1 Debian
4.16.5-1
[3175207.254201] Hardware name: System manufacturer System Product Name/P8H61-M LE, BIOS
0703 06/22/2011
This "swiotlb buffer is full" message seems to be a kernel bug introduced
around Jan 2018.
https://bugs.freedesktop.org/show_bug.cgi?id=104082
i've only quickly scanned that page but it seems like it's just a warning
rather than a serious problem - something about the kernel not being able to
allocate a single 2MB chunk of memory so having to fall back to multiple 4K
blocks instead.
maybe try upgrading to linux-image-4.16.0-2-amd64 (upstream kernel version
4.16.16 rather than 4.16.5)
craig
--
craig sanders <cas(a)taz.net.au>
1611
days inactive
1687
days old
5 comments
5 participants
participants (5)
-
Craig Sanders -
Daniel Jitnah -
Duncan Roe -
James Harper -
Russell Coker