Error1: ssh_exchange_identification: Connection closed by remote host Error2: ssh_exchange_identification: Connection closed by remote host rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9] I'm getting lots of errors like the above from cron jobs in situtions where running the same script from an interactive session works. The first error is from a cron job that rsync's files to a server and the second error is from rsyncing files from the server via ssh. On one of my systems I've seen a pair of cron jobs fail with that error every half hour for the past 4 hours but when I run both jobs from an interactive session they work every time. What can be different about the operation of ssh in this regard? It's a passwordless login with RSA key so there's no reason for it to need user input and in any case the error message doesn't indicate that as being a problem. I've compared the cron environment with the environment from the interactive session and apart from /bin/sh (DASH on Debian) vs /bin/bash I can't see anything of note. Any ideas? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 29 June 2015 at 22:00, Russell Coker <russell@coker.com.au> wrote:
Error1: ssh_exchange_identification: Connection closed by remote host
Error2: ssh_exchange_identification: Connection closed by remote host rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9]
I normally see this error when there's some authentication problem. In the past, it has either been due to hosts.deny/hosts.allow rules, or if the server has no daemons available for authentication (the MaxStartups value in sshd_config). I also think I've seen it when the nofile ulimit has been reached for the user trying to authenticate.
I'm getting lots of errors like the above from cron jobs in situtions where running the same script from an interactive session works. The first error is from a cron job that rsync's files to a server and the second error is from rsyncing files from the server via ssh. On one of my systems I've seen a pair of cron jobs fail with that error every half hour for the past 4 hours but when I run both jobs from an interactive session they work every time.
What can be different about the operation of ssh in this regard? It's a passwordless login with RSA key so there's no reason for it to need user input and in any case the error message doesn't indicate that as being a problem.
I've compared the cron environment with the environment from the interactive session and apart from /bin/sh (DASH on Debian) vs /bin/bash I can't see anything of note.
Any ideas?
Instead of trying an rsync, can you get the cron script to run a command via ssh, with max verbosity? e.g. ssh -vvv user@server /usr/bin/id just to see where the ssh process is failing? Sean
-- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/ _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
On Mon, 29 Jun 2015 10:34:07 PM Sean Crosby wrote:
On 29 June 2015 at 22:00, Russell Coker <russell@coker.com.au> wrote:
Error1: ssh_exchange_identification: Connection closed by remote host
Error2: ssh_exchange_identification: Connection closed by remote host rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9]
I normally see this error when there's some authentication problem. In the past, it has either been due to hosts.deny/hosts.allow rules,
I think that the fact that it works from an interactive session rules out authentication problems.
or if the server has no daemons available for authentication (the MaxStartups value in sshd_config). I also think I've seen it when the nofile ulimit has been reached for the user trying to authenticate.
The default for MaxStartups is to rate limit connections when there are 10 unauthenticated connections and to stop them entirely when there are 60 unauthenticated connections. A quick inspection by ps indicates no unauthenticated connections so I don't think this is the issue.
Instead of trying an rsync, can you get the cron script to run a command via ssh, with max verbosity? e.g.
ssh -vvv user@server /usr/bin/id
just to see where the ssh process is failing?
That's a good idea. However the system in question has inconveniently started working again. I'll leave a cron job of that running until it breaks again. Thanks for the suggestions! -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Hi, On 30 June 2015 at 00:19, Russell Coker <russell@coker.com.au> wrote:
On Mon, 29 Jun 2015 10:34:07 PM Sean Crosby wrote:
On 29 June 2015 at 22:00, Russell Coker <russell@coker.com.au> wrote:
Error1: ssh_exchange_identification: Connection closed by remote host
Error2: ssh_exchange_identification: Connection closed by remote host rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9]
I normally see this error when there's some authentication problem. In the past, it has either been due to hosts.deny/hosts.allow rules,
I think that the fact that it works from an interactive session rules out authentication problems.
or if the server has no daemons available for authentication (the MaxStartups value in sshd_config). I also think I've seen it when the nofile ulimit has been reached for the user trying to authenticate.
The default for MaxStartups is to rate limit connections when there are 10 unauthenticated connections and to stop them entirely when there are 60 unauthenticated connections. A quick inspection by ps indicates no unauthenticated connections so I don't think this is the issue.
MaxStartups still could be a problem. For example, if there are a group of client machines all trying to fetch/upload stats to one central server simultaneously every */5 minutes. A machine here has === # grep MaxS /etc/ssh/sshd_config MaxStartups 5:30:10 === Then if there are some attackers trying to guess passwords at the same time as the cron jobs, the regular cron jobs can start failing intermittently. John
Instead of trying an rsync, can you get the cron script to run a command
via ssh, with max verbosity? e.g.
ssh -vvv user@server /usr/bin/id
just to see where the ssh process is failing?
That's a good idea. However the system in question has inconveniently started working again. I'll leave a cron job of that running until it breaks again.
Thanks for the suggestions!
-- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/ _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
On Tue, 30 Jun 2015, John Mann <john.mann@monash.edu> wrote:
MaxStartups still could be a problem. For example, if there are a group of client machines all trying to fetch/upload stats to one central server simultaneously every */5 minutes.
A machine here has === # grep MaxS /etc/ssh/sshd_config MaxStartups 5:30:10 ===
Then if there are some attackers trying to guess passwords at the same time as the cron jobs, the regular cron jobs can start failing intermittently.
Great point. I've changed the configuration on the server to have significantly larger values for MaxStartups and I'll see what happens next. Also I'm going to change the cron jobs to interfere with each other less, EG by making hourly jobs start at 5 minutes past the hour so they don't match up with jobs that run every 10 minutes. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker <russell@coker.com.au> writes:
Error1: ssh_exchange_identification: Connection closed by remote host
Error2: ssh_exchange_identification: Connection closed by remote host rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9]
These appear to be client-side errors. By design, SSHv2 does not disclose much error information to the client. Check the SERVER-side log (e.g. /var/log/auth.log or /var/log/secure). I agree with the other posters that rate limiting is a likely candidate.
participants (4)
-
John Mann -
Russell Coker -
Sean Crosby -
trentbuck＠gmail.com