# free -m total used free shared buff/cache available Mem: 7962 2212 498 533 5251 4942 Swap: 10719 1732 8986 The above is from my workstation. It's running KDE, Chrome, KTorrent, and not much else. My understanding of the above is that most RAM is being used for cache and it's quite likely that this achieves the goal of reducing the number of storage accesses. The problem is that I don't want to reduce the number of storage accesses, I want to improve the performance of interactive tasks. Ktorrent is configured to only upload 60KB/s so a lack of caching of the torrents shouldn't prevent it from uploading at the maximum speed I permit. When large interactive programs like Chrome and Kmail get paged out it causes annoying delays when I want to perform what should be quick tasks like replying to a single message or viewing a single web page. Any suggestions as to how to optimise for this use case? I already have swap on one of the fastest SSDs I own and don't feel like buying NVMe for this purpose or buying a system with more RAM, so software changes are required. When replying please feel free to diverge from the topic. I think this is an area where most Linux users know less than they would like so randomly educational replies will be appreciated even if they don't help me with this problem. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Wed, Apr 25, 2018 at 10:46:15PM +1000, luv-main wrote:
# free -m total used free shared buff/cache available Mem: 7962 2212 498 533 5251 4942 Swap: 10719 1732 8986
The above is from my workstation. It's running KDE, Chrome, KTorrent, and not much else. My understanding of the above is that most RAM is being used for cache and it's quite likely that this achieves the goal of reducing the number of storage accesses.
The problem is that I don't want to reduce the number of storage accesses, I want to improve the performance of interactive tasks. Ktorrent is configured to only upload 60KB/s so a lack of caching of the torrents shouldn't prevent it from uploading at the maximum speed I permit. When large interactive programs like Chrome and Kmail get paged out it causes annoying delays when I want to perform what should be quick tasks like replying to a single message or viewing a single web page.
Any suggestions as to how to optimise for this use case? I already have swap on one of the fastest SSDs I own and don't feel like buying NVMe for this purpose or buying a system with more RAM, so software changes are required.
When replying please feel free to diverge from the topic. I think this is an area where most Linux users know less than they would like so randomly educational replies will be appreciated even if they don't help me with this problem.
Wow. I've never seen output from free like that before. My understanding has always been that you only get paging after cache is exhausted, but maybe I have been mistaken all this time. If you run without swap, does the OOM killer come in? Does top show plausibly large working sets for your apps e.g. such that they wouldn't all fit in RAM together? I used to have probems with Firefox using all of 1 cpu and (sometimes) all of RAM as well, but not since I unpinned dilbert.com. HTH Cheers ... Duncan.
# free -m total used free shared buff/cache available Mem: 7962 2212 498 533 5251 4942 Swap: 10719 1732 8986
The above is from my workstation. It's running KDE, Chrome, KTorrent, and not much else. My understanding of the above is that most RAM is being used for cache and it's quite likely that this achieves the goal of reducing the number of storage accesses.
The problem is that I don't want to reduce the number of storage accesses, I want to improve the performance of interactive tasks. Ktorrent is configured to only upload 60KB/s so a lack of caching of the torrents shouldn't prevent it from uploading at the maximum speed I permit. When large interactive programs like Chrome and Kmail get paged out it causes annoying delays when I want to perform what should be quick tasks like replying to a single message or viewing a single web page.
Any suggestions as to how to optimise for this use case? I already have swap on one of the fastest SSDs I own and don't feel like buying NVMe for this purpose or buying a system with more RAM, so software changes are required.
Have you done the analysis to confirm your suspicions? Iostat while switching to Chrome etc? I believe Linux will prematurely swap stuff out without invalidating the actual memory pages, so that when it actually needs the memory it can just use it without first having to wait for it to be written to disk, and conversely if the original owner of the page needs the memory back it's still there. While a page is in this stage - ready to be used for something else but still containing a valid copy of the original page - I don't know how it shows in the output of free. It is simultaneously "paged out", "free", and "in use", so I think maybe some further analysis is required to see exactly what is going on. Based on the above output of free, you could try leaving Chrome alone for a while, then when you want to use it, turn swap off then on again, then see if Chrome is fast to switch to. If not, then the delay might be coming from elsewhere. If you just want to tinker, start with lowering the "swappiness" control. James
Is swappiness setting in Linux what you are looking for? How to change the Swappiness of your Linux system https://www.howtoforge.com/tutorial/linux-swappiness/ Daniel. On Wed, 2018-04-25 at 22:46 +1000, Russell Coker via luv-main wrote:
# free -m total used free shared buff/cache ava ilable Mem: 7962 2212 498 533 5251 4942 Swap: 10719 1732 8986
The above is from my workstation. It's running KDE, Chrome, KTorrent, and not much else. My understanding of the above is that most RAM is being used for cache and it's quite likely that this achieves the goal of reducing the number of storage accesses.
The problem is that I don't want to reduce the number of storage accesses, I want to improve the performance of interactive tasks. Ktorrent is configured to only upload 60KB/s so a lack of caching of the torrents shouldn't prevent it from uploading at the maximum speed I permit. When large interactive programs like Chrome and Kmail get paged out it causes annoying delays when I want to perform what should be quick tasks like replying to a single message or viewing a single web page.
Any suggestions as to how to optimise for this use case? I already have swap on one of the fastest SSDs I own and don't feel like buying NVMe for this purpose or buying a system with more RAM, so software changes are required.
When replying please feel free to diverge from the topic. I think this is an area where most Linux users know less than they would like so randomly educational replies will be appreciated even if they don't help me with this problem.
On Thursday, 26 April 2018 10:47:52 AM AEST Daniel Jitnah via luv-main wrote:
Is swappiness setting in Linux what you are looking for?
How to change the Swappiness of your Linux system https://www.howtoforge.com/tutorial/linux-swappiness/
Thanks Daniel and James. I changed the swappiness of the system in question by putting vm.swappiness = 10 in /etc/sysctl.conf. Generally the performance seems better since doing that. However there are some downsides, after running for a while with a low value I've started seeing the following errors in the kernel message log when using mplayer while running kernel 4.16.0-1-amd64. When I play a 720p video (EG one downloaded from SBS on demand with youtube-dl) it works fine (although usually having that kernel error), but a FullHD video downloaded from YouTube causes extremely poor performance (hangs of a second every few seconds and jittering of the video) and is unusable. This isn't a total surprise, presumably the defaults were set for a reason and mplayer causes the X server to do some things that it doesn't do in other situations. Maybe I would have the same kernel issue if I tried 3D gaming, but I haven't done that much in recent times. I really need to get more RAM. [3175207.254191] radeon 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes) [3175207.254196] radeon 0000:01:00.0: swiotlb: coherent allocation failed, size=2097152 [3175207.254199] CPU: 5 PID: 1199 Comm: Xorg Not tainted 4.16.0-1-amd64 #1 Debian 4.16.5-1 [3175207.254201] Hardware name: System manufacturer System Product Name/P8H61- M LE, BIOS 0703 06/22/2011 [3175207.254202] Call Trace: [3175207.254211] dump_stack+0x5c/0x85 [3175207.254215] swiotlb_alloc_coherent+0x19b/0x1b0 [3175207.254223] ttm_dma_pool_get_pages+0x235/0x620 [ttm] [3175207.254229] ttm_dma_populate+0x25e/0x350 [ttm] -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
NOTE: I am not an expert with video decoding, but hopefully some of the following will be of use in improving your video playback... On Mon, Jul 09, 2018 at 09:19:34PM +1000, Russell Coker wrote:
[...] I've started seeing the following errors in the kernel message log when using mplayer while running kernel 4.16.0-1-amd64. When I play a 720p video (EG one downloaded from SBS on demand with youtube-dl) it works fine (although usually having that kernel error), but a FullHD video downloaded from YouTube causes extremely poor performance (hangs of a second every few seconds and jittering of the video) and is unusable.
What kind of GPU do you have? an AMD Radeon, obviosly from the dmesg output but what model ("lspci | grep VGA" if you're not sure) Are you using VAAPI or VDPAU with mplayer? VDPAU was originally nvidia-specific video acceleration API but for many years now there's been open source libs to support intel and amd GPUs. I use nvidia GPUs with the proprietary nvidia driver so vdpau stuff "Just Works"...and I don't have any direct experience using vdpau with other hardware or drivers, so consider the following a pointer in roughly the right direction rather than detailed instructions: You'll need to install at least the following packages if they're not already installed: apt-get install vdpauinfo libvdpau1 libdrm-amdgpu1 libvdpau-va-gl1 libvdpau-va-gl1 allows some programs using the VDPAU API to use OpenGL and/or VA-API acceleration. To see what kind of video acceleration is supported by your hardware & radeon driver, run "vdpauinfo". You may need to run it as "VDPAU_DRIVER=va_gl vdpauinfo" to set the env var before running it (see /usr/share/doc/libvdpau-va-gl1/README.Debian) This should show at least mpeg-2, mpeg-4, and probably h.264 acceleration. If it does, edit your ~/.mplayer/config to add vo=vdpau, vc=ffh264vdpau,ffmpeg12vdpau,ffodivxvdpau,ffwmv3vdpau,ffvc1vdpau,ffhevcvdpau The 'vc=' line will probably need tweaking to match your hardware & drivers. Playing with mplayer's cache settings might help too - e.g. 'cache = 8192'. BTW, what kind of video formats are used by the 720p and Full HD videos? If the 720p is mpeg4 or h.264 while the Full HD is h.265 that may explain the problem - AFAIK, HW-accelerated h.265 decoding isn't supported on anything except the proprietary nvidia driver with a geforce 900 series GPU or newer (e.g. GTX-9xx or GTX-10xx). Without HW acceleration, it falls back to software decoding, much slower than using the GPU HW. You may be able to configure mplayer to use more CPU cores/threads to speed it up (e.g. with 'lavdopts=threads=2' in ~/.mplayer/config). or use mpv instead of mplayer, which IIRC does multi-threading by default. Try running 'mediainfo' on the video files. 'apt-get install mediainfo' if it's not already installed. This will tell you what kind of container it is, and what kind of video & audio codecs were used (and also if it contains any text streams for subtitles) Also worth considering: if your machine struggles to play Full HD video, but has no problem with 720p then don't make it play Full HD - stick to 720p or less. lower resolution that plays without stuttering or pausing is infinitely better than high-res that doesn't play smoothly. Anyway, when you run mplayer from the command line, check its output to see what it's using. if vdpau is enabled and working on your system, You should see it mentioned in mplayer's terminal output. e.g. on my system: [...irrelevant junk deleted...] libavformat version 57.71.100 (internal) libavformat file format detected. [lavf] stream 0: video (h264), -vid 0 [lavf] stream 1: audio (aac), -aid 0 VIDEO: [H264] 720x404 0bpp 23.976 fps 0.0 kbps ( 0.0 kbyte/s) ========================================================================== Forced video codec: ffh264vdpau Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family libavcodec version 57.89.100 (internal) Selected video codec: [ffh264vdpau] vfm: ffmpeg (FFmpeg H.264 (VDPAU)) ========================================================================== Clip info: encoder: libebml v1.3.3 + libmatroska v1.4.4 creation_time: 2018-06-28T04:53:20.000000Z Load subtitles in /home/cas/video/ ========================================================================== Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders AUDIO: 48000 Hz, 2 ch, floatle, 0.0 kbit/0.00% (ratio: 0->384000) Selected audio codec: [ffaac] afm: ffmpeg (FFmpeg AAC (MPEG-2/MPEG-4 Audio)) ========================================================================== AO: [pulse] 48000Hz 2ch floatle (4 bytes per sample) Starting playback... Movie-Aspect is 1.78:1 - prescaling to correct movie aspect. VO: [vdpau] 720x404 => 720x404 H.264 VDPAU acceleration Movie-Aspect is 1.78:1 - prescaling to correct movie aspect. VO: [vdpau] 720x404 => 720x404 H.264 VDPAU acceleration A: 1.8 V: 1.8 A-V: 0.001 ct: 0.031 0/ 0 16% 2% 0.5% 8 0 98% You may also want to try using smplayer. It's no more or less capable than mplayer, but may be easier to configure. smplayer certainly - it's a GUI wrapper around mplayer (or mpv) with configuration dialogs...which is a lot easier to make sense of than mplayer's extraordinarily dense man page (the info's all there in man but doesn't make much sense unless you already know a lot about video playback and codecs and acceleration, which I don't...my video knowledge is cherry-picked googling and cargo-culting until it kind of works. Worse, this stuff changes all the time so it's very hard to keep current with what works and what doesn't).
This isn't a total surprise, presumably the defaults were set for a reason and mplayer causes the X server to do some things that it doesn't do in other situations. Maybe I would have the same kernel issue if I tried 3D gaming, but I haven't done that much in recent times.
I really need to get more RAM.
what you're describing really doesn't look like a memory shortage. Your original post back in April showed around 5GB of RAM free. Chromium can be a bloated pig (on my system, it's currently using 9.5GB of RAM and often uses 13GB or more (At the moment, I have about a dozen chromium windows with a total of 112 tabs open), but if that were your problem you would see very little free RAM, very little RAM used by buffers/cache, and LOTS of swap in use. None of that seems to be the case from your original message.
[3175207.254191] radeon 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes) [3175207.254196] radeon 0000:01:00.0: swiotlb: coherent allocation failed, size=2097152 [3175207.254199] CPU: 5 PID: 1199 Comm: Xorg Not tainted 4.16.0-1-amd64 #1 Debian 4.16.5-1 [3175207.254201] Hardware name: System manufacturer System Product Name/P8H61-M LE, BIOS 0703 06/22/2011
This "swiotlb buffer is full" message seems to be a kernel bug introduced around Jan 2018. https://bugs.freedesktop.org/show_bug.cgi?id=104082 i've only quickly scanned that page but it seems like it's just a warning rather than a serious problem - something about the kernel not being able to allocate a single 2MB chunk of memory so having to fall back to multiple 4K blocks instead. maybe try upgrading to linux-image-4.16.0-2-amd64 (upstream kernel version 4.16.16 rather than 4.16.5) craig -- craig sanders <cas@taz.net.au>
participants (5)
-
Craig Sanders -
Daniel Jitnah -
Duncan Roe -
James Harper -
Russell Coker