Hi there, I have a server that regularly runs jobs to import email via IMAP from remote servers. The cron runs a bash scrip that in turn iterates through the customer's install directories and then goes through each mail box that users have configured and parses the email. This is done by going back to get 100 email headers checking that the message ID hasn't been imported and if not getting the full body. The problem is that these jobs seem to run over each other and then eventually (occasionally) it will hang the server. I've tried a few things like getting less headers, spacing the cron jobs out and using lock files but neither combination has resolved this issue. I suspect that some of the IMAP services take too long to respond or rate limit connections (but since I don't control them I can't be sure). I am looking for suggestions as to: - How to fix the underlying problem - How to configure Nagios to monitor specific processes Many thanks Piers
Hi Piers, You could try creating a new file in the /tmp folder in process then remove it on closing of the script. Subsequent scripts might start then check for for the presence of this file. If it exists simply delay until it is gone or exit the script. Don't forget to log the exec and the exits if they are produced. At least you will then see the issues clearly. Example #ScriptRun # while fileexists /tmp/ScriptOn Sleep(5) # or if FileExists /tmp/ScriptOn Exit; CreateFile /tmp/ScriptOn #Do the rest of your work after the check. ... #On exit from current instance Ddeletefile /tmp/ScriptOn Hope this helps. best wishes Tony White On 13/2/19 12:45 pm, Piers via luv-main wrote:
Hi there,
I have a server that regularly runs jobs to import email via IMAP from remote servers.
The cron runs a bash scrip that in turn iterates through the customer's install directories and then goes through each mail box that users have configured and parses the email. This is done by going back to get 100 email headers checking that the message ID hasn't been imported and if not getting the full body.
The problem is that these jobs seem to run over each other and then eventually (occasionally) it will hang the server.
I've tried a few things like getting less headers, spacing the cron jobs out and using lock files but neither combination has resolved this issue. I suspect that some of the IMAP services take too long to respond or rate limit connections (but since I don't control them I can't be sure).
I am looking for suggestions as to:
- How to fix the underlying problem
- How to configure Nagios to monitor specific processes
Many thanks
Piers
_______________________________________________ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
Hi Piers, Alternatively, you could use flock to do the same. Something like this, http://www.elevatedcode.com/2013/05/07/flock-for-cron-jobs.html -Manoj.C On Wed, Feb 13, 2019 at 1:40 PM Tony White via luv-main <luv-main@luv.asn.au> wrote:
Hi Piers, You could try creating a new file in the /tmp folder in process then remove it on closing of the script. Subsequent scripts might start then check for for the presence of this file. If it exists simply delay until it is gone or exit the script. Don't forget to log the exec and the exits if they are produced. At least you will then see the issues clearly.
Example #ScriptRun # while fileexists /tmp/ScriptOn Sleep(5) # or if FileExists /tmp/ScriptOn Exit; CreateFile /tmp/ScriptOn #Do the rest of your work after the check. ... #On exit from current instance Ddeletefile /tmp/ScriptOn
Hope this helps.
best wishes Tony White
On 13/2/19 12:45 pm, Piers via luv-main wrote:
Hi there,
I have a server that regularly runs jobs to import email via IMAP from remote servers.
The cron runs a bash scrip that in turn iterates through the customer's install directories and then goes through each mail box that users have configured and parses the email. This is done by going back to get 100 email headers checking that the message ID hasn't been imported and if not getting the full body.
The problem is that these jobs seem to run over each other and then eventually (occasionally) it will hang the server.
I've tried a few things like getting less headers, spacing the cron jobs out and using lock files but neither combination has resolved this issue. I suspect that some of the IMAP services take too long to respond or rate limit connections (but since I don't control them I can't be sure).
I am looking for suggestions as to:
- How to fix the underlying problem
- How to configure Nagios to monitor specific processes
Many thanks
Piers
_______________________________________________ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
_______________________________________________ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
On 13/02/19 14:45, Manoj C Menon via luv-main wrote:
Hi Piers,
Alternatively, you could use flock to do the same.
Something like this, http://www.elevatedcode.com/2013/05/07/flock-for-cron-jobs.html
Hi Manoj, I was using the PHP equivalent however I probably did use it correctly (because there is always PHP jobs running so I guess it was locking itself out). I really appreciate your helpful answer. Thanks Piers
Hi Tony,
You could try creating a new file in the /tmp folder in process then remove it on closing of the script. Subsequent scripts might start then check for for the presence of this file. If it exists simply delay until it is gone or exit the script.
I tried this in PHP but must have done it badly because it didn't work. Actually doing this in the parent bash script makes a whole lot more sense
Example #ScriptRun # while fileexists /tmp/ScriptOn Sleep(5) # or if FileExists /tmp/ScriptOn Exit; CreateFile /tmp/ScriptOn #Do the rest of your work after the check. ... #On exit from current instance Ddeletefile /tmp/ScriptOn
Is that supposed to be *Deletefile* ?
Hope this helps.
It does & thank you Tony. Cheers Piers
On Thu, Feb 14, 2019 at 09:02:54AM +1000, Piers wrote:
Is that supposed to be *Deletefile* ?
Does it matter? It's clearly neither PHP code nor sh code. It's very obviously pseudocode intended to show the steps that need to performed to implement a simple semaphore/lock-file. craig -- craig sanders <cas@taz.net.au> BOFH excuse #63: not properly grounded, please bury computer
On 13.02.19 11:45, Piers via luv-main wrote:
I have a server that regularly runs jobs to import email via IMAP from remote servers. ... The problem is that these jobs seem to run over each other and then eventually (occasionally) it will hang the server. ... I am looking for suggestions as to:
- How to fix the underlying problem
For a couple of decades now, I've used fetchmail to bring in mail from IMAP or POP mailservers, either corporate or (now in retirement) at the ISP. I configure it to run each 10 minutes, and it never jams. Restarting after a power outage can result in a query: "Delete stale lockfile?" There's a fetchmailconf command to assist with initial setup, so you don't have to hack .fetchmailrc out of raw vacuum. I see mine has barely changed in nearly 20 years. (fetchmailconf datestamps its work) You can have it send mail directly to procmail (if that's what you use to distribute it to multiple inboxes): fetchmail -m procmail or just let it forward to port 25. That's what I tend to do, even though it's probably a complete waste to run postfix with only one mail recipient now. (Old habits die hard.) I'm all for a good excuse for hacking a script, but a long-solved problem isn't such a use case. Erik
I'm all for a good excuse for hacking a script, but a long-solved problem isn't such a use case.
Erik
Thanks Erik, The software is part of a LAMP stack and is used to import documents from attachments into a CRM and attach them to people's records, convert them to text, index them and them parse them for data. I am sure fetchmail does a great job for your purposes but the IMAP connection is only the tip of the ice berg so jumping out of the LAMP stack for something that is already part of PHP would increase the complexity unnecessarily tho. Thanks for your reply. Cheers P
On 14.02.19 09:00, Piers via luv-main wrote:
I'm all for a good excuse for hacking a script, but a long-solved problem isn't such a use case.
Erik
Thanks Erik,
The software is part of a LAMP stack and is used to import documents from attachments into a CRM and attach them to people's records, convert them to text, index them and them parse them for data.
I am sure fetchmail does a great job for your purposes but the IMAP connection is only the tip of the ice berg so jumping out of the LAMP stack for something that is already part of PHP would increase the complexity unnecessarily tho.
Your problems are not the first time that poor task partitioning has made life more difficult. If you were to let fetchmail efficiently and problem-free handle mail delivery, then you only have to either: 1) Pick the mail out of the inbox in your script(s), or 2) deliver each mail directly to a converting filter if your pipeline prefers that, and can buffer adequately. To attempt to mix up post-arrival grist milling with mail delivery makes no sense, I suggest. An inbox makes a fine buffer between the two temporally disconnected activities, each with its own fluctuating data rate, but any buffer will do. In the past I have scripted a similar system. Received emails had attachments stripped, passed through the grist mill, and reports forwarded to appropriate recipients. I let mail utilities handle mail, and kept the back end as a separate back end. Easy to test, and easy to keep working. Erik
On Thu, Feb 14, 2019 at 07:17:49PM +1100, luv-main@luv.asn.au wrote:
1) Pick the mail out of the inbox in your script(s), or 2) deliver each mail directly to a converting filter if your pipeline prefers that, and can buffer adequately.
Yes. Break larger tasks into multiple smaller tasks, each of which is either easily solvable or already has tried-and-tested solutions.
To attempt to mix up post-arrival grist milling with mail delivery makes no sense, I suggest. An inbox makes a fine buffer between the two temporally disconnected activities, each with its own fluctuating data rate, but any buffer will do.
In other words, the inbox is a queue for other script(s) to process in order, without having to complicate those scripts by adding a poor imitation of an already-existing, full-featured, and debugged imap mail fetcher. Alternatively if using an MDA like procmail or deliver (or some other programmable MDA), incoming mail can be automatically piped into the appropriate mail-processing script when it arrives, with pattern rules allowing mail to be delivered to one (or more) of several different scripts depending on which pattern(s) any given message matches. Matchable patterns include headers such as To:, From:, Subject:, X-Mailer:, and body text (these days, it's worthwhile getting procmail/deliver/whatever to mime-decode the text/plain or text/html attachments BEFORE trying to match patterns in the body) Using existing MDA like this allows your script to just do the one thing it has to do (i.e. process the mail) without also having to fetch it and/or decide whether that message is one that should be processed by the script. In short: don't re-invent the wheel, especially when there are already several good wheels to choose from, each with decaades of bug-fixes and refinement. Instead, use multiple small tools, each of which is exceptionally good at the one thing it does. craig -- craig sanders <cas@taz.net.au> BOFH excuse #249: Unfortunately we have run out of bits/bytes/whatever. Don't worry, the next supply will be coming next week.
participants (5)
-
Craig Sanders -
Erik Christiansen -
Manoj C Menon -
Piers -
Tony White