15 Jul
2018
15 Jul
'18
4:33 p.m.
What's a good real-time log watching program?
This is something I've wanted for a while but not had the inclination to get
it going. The thing that finally made me want to do it is when my workstation
gave the below log messages and decided to stop supporting USB 3.0 ports. As
the ports that are most convenient for connecting my keyboard and mouse are
USB 3.0 that was inconvenient. Removing the xhci modules and loading them
again fixed the problem. So what I want in this instance is a program that
will detect one of the below messages and then run a script that will run some
rmmod and modprobe commands to fix it. I want it to know which log entries it
has dealt with to avoid performing the operation twice (which would cause
problems) or maybe have a list of log messages in the problem solved category.
Any suggestions?
[18408.111698] xhci_hcd 0000:06:00.0: xHCI host not responding to stop
endpoint command.
[18408.111716] xhci_hcd 0000:06:00.0: xHCI host controller not responding,
assume dead
[18408.111738] xhci_hcd 0000:06:00.0: HC died; cleaning up
[18408.111747] xhci_hcd 0000:06:00.0: Timeout while waiting for configure
endpoint command
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
15 Jul
15 Jul
5:35 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi,
On 15/07/18 16:33, Russell Coker via luv-main wrote:
What's a good real-time log watching program?
This is something I've wanted for a while but not had the
inclination to get it going. The thing that finally made me want
to do it is when my workstation gave the below log messages and
decided to stop supporting USB 3.0 ports. As the ports that are
most convenient for connecting my keyboard and mouse are USB 3.0
that was inconvenient. Removing the xhci modules and loading them
again fixed the problem. So what I want in this instance is a
program that will detect one of the below messages and then run a
script that will run some rmmod and modprobe commands to fix it. I
want it to know which log entries it has dealt with to avoid
performing the operation twice (which would cause problems) or
maybe have a list of log messages in the problem solved category.
Any suggestions?
[18408.111698] xhci_hcd 0000:06:00.0: xHCI host not responding to
stop endpoint command. [18408.111716] xhci_hcd 0000:06:00.0: xHCI
host controller not responding, assume dead [18408.111738] xhci_hcd
0000:06:00.0: HC died; cleaning up [18408.111747] xhci_hcd
0000:06:00.0: Timeout while waiting for configure endpoint command
Simple bash?
$ l --full-time coker*
- -rw-r--r-- 1 andrewm andrewm 60 2018-07-15 17:23:06.450452422 +1000
coker-test.log
- -rw-r--r-- 1 andrewm andrewm 12 2018-07-15 17:23:06.450452422 +1000
coker-test.action-time
- -rw-r--r-- 1 andrewm andrewm 0 2018-07-15 17:23:07.278459755 +1000
coker-test.lastdone
In another terminal:
$ (tail -n0 -f coker-test.log |egrep --line-buffered '(aaa|aab)') >>
coker-test.action-time & (while :;do [[ coker-test.action-time -nt
coker-test.lastdone ]] && { echo do stuff;touch
coker-test.lastdone;break; };sleep 10;echo -n .;done)
Then echo some strings to coker-test.log from first terminal.
( Check status in second terminal each time)
$ echo zzz >> coker-test.log
$ echo aab >> coker-test.log
The echo of aab will cause the second terminal to show the following
and quit the loop:
...do stuff
The testing each 10 seconds (with sleep), so about 30 seconds in it
sees the aab string.
$ ls -lart --full-time coker*
- -rw-r--r-- 1 andrewm andrewm 68 2018-07-15 17:24:09.579011308 +1000
coker-test.log
- -rw-r--r-- 1 andrewm andrewm 16 2018-07-15 17:24:09.579011308 +1000
coker-test.action-time
- -rw-r--r-- 1 andrewm andrewm 0 2018-07-15 17:24:13.259043876 +1000
coker-test.lastdone
When you are happy that is all working fine, wrap the above to keep it
active:
while :;do (tail -n0 -f coker-test.log |egrep --line-buffered
'(aaa|aab)') >> coker-test.action-time & (while :;do [[
coker-test.action-time -nt coker-test.lastdone ]] && { echo do
stuff;touch coker-test.lastdone;break; };sleep 10;echo -n .;done);done
Cheers
A.
-----BEGIN PGP SIGNATURE-----
iF4EAREIAAYFAltK+UIACgkQqBZry7fv4vtQbwEAo5cVG3CPAxg22bPcEO4DPeXo
onMvipitnY0b9AoNevcA/RFx1eZKvqgx24okoRVw7RKSIaioTCNjeoML+9i9Xwjs
=YJA3
-----END PGP SIGNATURE-----
11:18 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi,
On 15/07/18 17:35, Andrew McGlashan via luv-main wrote:
Hi,
On 15/07/18 16:33, Russell Coker via luv-main wrote:
The multitail option runs the "command" for every single match, the
original bash method seems best so far.
Cheers
A.
What's a good real-time log watching program?
This is something I've wanted for a while but
not had the
inclination to get it going. The thing that finally made me
want to do it is when my workstation gave the below log messages
and decided to stop supporting USB 3.0 ports. As the ports that
are most convenient for connecting my keyboard and mouse are USB
3.0 that was inconvenient. Removing the xhci modules and loading
them again fixed the problem. So what I want in this instance is
a program that will detect one of the below messages and then run
a script that will run some rmmod and modprobe commands to fix
it. I want it to know which log entries it has dealt with to
avoid performing the operation twice (which would cause problems)
or maybe have a list of log messages in the problem solved
category.
Any suggestions?
[18408.111698] xhci_hcd 0000:06:00.0: xHCI host
not responding
to stop endpoint command. [18408.111716] xhci_hcd 0000:06:00.0:
xHCI host controller not responding, assume dead [18408.111738]
xhci_hcd 0000:06:00.0: HC died; cleaning up [18408.111747]
xhci_hcd 0000:06:00.0: Timeout while waiting for configure
endpoint command
Simple bash?
$ l --full-time coker* -rw-r--r-- 1 andrewm andrewm 60 2018-07-15
17:23:06.450452422 +1000 coker-test.log -rw-r--r-- 1 andrewm
andrewm 12 2018-07-15 17:23:06.450452422 +1000
coker-test.action-time -rw-r--r-- 1 andrewm andrewm 0 2018-07-15
17:23:07.278459755 +1000 coker-test.lastdone
In another terminal:
$ (tail -n0 -f coker-test.log |egrep --line-buffered '(aaa|aab)')
> coker-test.action-time & (while :;do [[
coker-test.action-time
-nt coker-test.lastdone ]] && { echo do stuff;touch
coker-test.lastdone;break; };sleep 10;echo -n .;done)
Then echo some strings to coker-test.log from first terminal.
( Check status in second terminal each time)
$ echo zzz >> coker-test.log $ echo aab >> coker-test.log
The echo of aab will cause the second terminal to show the
following and quit the loop:
...do stuff
The testing each 10 seconds (with sleep), so about 30 seconds in
it sees the aab string.
$ ls -lart --full-time coker* -rw-r--r-- 1 andrewm andrewm 68
2018-07-15 17:24:09.579011308 +1000 coker-test.log -rw-r--r-- 1
andrewm andrewm 16 2018-07-15 17:24:09.579011308 +1000
coker-test.action-time -rw-r--r-- 1 andrewm andrewm 0 2018-07-15
17:24:13.259043876 +1000 coker-test.lastdone
When you are happy that is all working fine, wrap the above to keep
it active:
while :;do (tail -n0 -f coker-test.log |egrep --line-buffered
'(aaa|aab)') >> coker-test.action-time & (while :;do [[
coker-test.action-time -nt coker-test.lastdone ]] && { echo do
stuff;touch coker-test.lastdone;break; };sleep 10;echo -n
.;done);done -----BEGIN PGP SIGNATURE-----
iF4EAREIAAYFAltLSZIACgkQqBZry7fv4vurOAEA1GQYhjM/MyidTnv1u1GRVh/T
pCT8slYx1QzoPsyY1jsA/jeBTvwfdfqDz5svId473SrTKLTcw5/uMtdydxQm2xRv
=8bJ0
-----END PGP SIGNATURE-----
16 Jul
16 Jul
10:31 a.m.
Reading this thread inspired the question of how to solve the problem if the
systemd journal is used.
The best that I could find via a quick web search is
https://jjacky.com/2013-10-06-run-triggers-on-systemd-journal-messages/
Are there better solutions?
(Just curiosity - I have no immediate need.)
15 Jul
15 Jul
6:30 p.m.
Hi Russell,
You could look at multitail.
https://www.vanheusden.com/multitail/
On 15/07/18 16:33, Russell Coker via luv-main wrote:
What's a good real-time log watching program?
This is something I've wanted for a while but not had the inclination to get
it going. The thing that finally made me want to do it is when my workstation
gave the below log messages and decided to stop supporting USB 3.0 ports. As
the ports that are most convenient for connecting my keyboard and mouse are
USB 3.0 that was inconvenient. Removing the xhci modules and loading them
again fixed the problem. So what I want in this instance is a program that
will detect one of the below messages and then run a script that will run some
rmmod and modprobe commands to fix it. I want it to know which log entries it
has dealt with to avoid performing the operation twice (which would cause
problems) or maybe have a list of log messages in the problem solved category.
Any suggestions?
[18408.111698] xhci_hcd 0000:06:00.0: xHCI host not responding to stop
endpoint command.
[18408.111716] xhci_hcd 0000:06:00.0: xHCI host controller not responding,
assume dead
[18408.111738] xhci_hcd 0000:06:00.0: HC died; cleaning up
[18408.111747] xhci_hcd 0000:06:00.0: Timeout while waiting for configure
endpoint command
8:10 p.m.
Hi,
On 15/07/18 18:30, Tony White via luv-main wrote:
You could look at multitail.
https://www.vanheusden.com/multitail/
This example works with multitail:
$ multitail -ex 'aaa|aab|aac' "echo do work $(date) >>
coker-test.action-time" -l "tail -n0 -f coker-test.log|egrep
--line-buffered '(aaa|aab|aac)'"
Cheers
A.
8:16 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi,
On 15/07/18 18:30, Tony White via luv-main wrote:
You could look at multitail.
https://www.vanheusden.com/multitail/
This is a better example with multitail:
$ multitail -ex '(aaa|aab|aac)' "echo do work \$(date) >>
coker-test.action-time" -l "tail -n0 -f coker-test.log|egrep
'(aaa|aab|aac)'"
(don't need --line-buffered with this one)
NB: The \$ so that the command runs at that time of the detection,
rather than when multitail was initiated.
Cheers
A.
-----BEGIN PGP SIGNATURE-----
iF4EAREIAAYFAltLHuIACgkQqBZry7fv4vu31gD8DNtRgsrBK9TTjAZAtIRblqRa
Sxg9mZwoLCSEqpwEg5kBAIWkfU4JBspe6Xl1xHZ0NzDkwvFKRwODWgwwS6UzHPF2
=N5BJ
-----END PGP SIGNATURE-----
16 Jul
16 Jul
11:42 a.m.
On Sun, Jul 15, 2018 at 04:33:46PM +1000, russell(a)coker.com.au wrote:
What's a good real-time log watching program?
This is something I've wanted for a while but not had the inclination to get
it going. [...]
Any suggestions?
Perl's File::Tail module is made for this. It shouldn't take more than 10 or
15 lines of mostly boilerplate code to monitor the kernel log and reload the
modules when it sees the xhci errors.
Alternatively, if you're already running fail2ban, you can use it to monitor
the kernel log, and create a custom action to unload and reload xhci.
BTW, long before fail2ban existed, I implemented a very primitive version of
what it does using perl + File::Tail.
craig
PS: xhci shouldn't be dying like that. Restarting it is just a work-around
hack - you really ought to find out what the real problem is and fix it.
--
craig sanders <cas(a)taz.net.au>
1719
days inactive
1720
days old
7 comments
5 participants
participants (5)
-
Andrew McGlashan -
Craig Sanders -
Jason White -
Russell Coker -
Tony White