Hi, On 12/02/2012 2:48 PM, Chris Samuel wrote:

http://lists.luv.asn.au/options/luv-main

# If the list has member personalized messages enabled, and you elect # to receive copies, every copy will have a X-Mailman-Copy: yes header # added to it.

Thank you, I've made that adjustment -- much appreciated. On a side note, it is actually better NOT to cc or bcc those that are subscribed and only to reply to list, but not everyone practices that and it can be difficult to "oblige" people to do that. Cheers -- Kind Regards AndrewM

On 12.02.12 19:16, Jason White wrote:

I know there are people who will say that Maildrop is better and that I should be using it instead of Procmail, but at the moment I don't have a compelling reason to switch.

Procmail has reached such a state of user satisfaction that there's little or nothing to add or change. That just about makes it a delighfully stable product to move _to_, perhaps? ;-) Seriously, it just keeps on working, so the talk on the procmail ML in 2010, of doing some further development, was probably premature. Erik -- Habit is habit, and not to be flung out of the window by any man, but coaxed down-stairs a step at a time. - Mark Twain, "Pudd'nhead Wilson's Calendar

On Sun, Feb 12, 2012 at 07:16:52PM +1100, Jason White wrote:

You can, however, use the ^TO_ expression in a Procmail recipe to match any message destined for the list (whether in the "To" or "Cc" field), placing it into the appropriate folder.

i find that rules like the following catch every msg for a list, regardless of the list software being used: :0 E * (^TO|^From|^FROM_DAEMON|^Sender:|^X-Been-There:|^List-[^:]*:).*luv-main(-owner)?@(lists\.)?luv\.asn\.au $LISTDIR/luv/main/incoming ^TO and ^FROM_DAEMON are expanded to much larger regexps and, like ^TO_, are documented in procmailrc(5). craig -- craig sanders <cas@taz.net.au>

Hi, On 12/02/2012 7:30 PM, Erik Christiansen wrote:

OK, you're using Mozilla, on MSW, so unixy solutions for that aren't applicable, I guess. On linux, you could have used something like procmail to put the first copy to arrive into the list mailbox, and the second into "duplicates". (Or irrevocably discard it, if daring.)

I use .forward filters on my IMAP server -- List-Id is the best way to deal with list emails unless the list doesn't use a List-Id, then it becomes more painful.

Mind you, I agree with you. The habit of sending "courtesy copies" is one which only seems to have merit if the receiving parties are behind with their list mail, and the poster is impatient.

Some people demand that you DO NOT cc them as they are subscribed... you can't win and you can't please everyone. At least the mailman adjustment suits me.

In any event, the procmail rule keeps my main inbox free of these things. :-)

Exim with IMAP storage and .forward filters keeps things sorted for me. I actually deliver some emails to multiple mail boxes on purpose -- the main list one as an archive that I rarely look at and one to a folder that I am checking more regularly, that one shares all incoming mail for MLUG as well as LUV and other related lists... ;-) TB on MSW downloads IMAP folders for me.... -- Kind Regards AndrewM Andrew McGlashan Broadband Solutions now including VoIP Current Land Line No: 03 9012 2102 Mobile: 04 2574 1827 Fax: 03 9012 2178 National No: 1300 85 3804 Affinity Vision Australia Pty Ltd http://www.affinityvision.com.au http://adsl2choice.net.au In Case of Emergency -- http://www.affinityvision.com.au/ice.html

On 12.02.12 23:39, Craig Sanders wrote:

i probably should just set up fetchmail or something. also, i can tolerate TB because of the external editor plugin which lets me use gvim to edit messages.

Yes, much can be forgiven so long as composing is in your natural environment. Never having bought into the hassles of trying to synchronise mail here and there on the internet, I've always just used fetchmail. There's not much to set up after package install, apart from taking a quick look in /etc/init.d/fetchmail to read two line on auto-starting the thing. (Both ubuntu 7.10 and 10.04 had "START_DAEMON=no" in /etc/default/fetchmail, on installation. (Especially if re-using an old .fetchmailrc)

at home, i still use mutt & procmail. in a screen session, with a script to start up about 20 mutts backgrounded (one for each mailbox i switch to regularly - otherwise switching mailboxes(*) in mutt involves closing the current mailbox. doing this, switching mboxes is as simple as ^Z and then 'fg n' - the script starts the mutts in the same order every time, so i've memorised the job numbers for particular mboxes by now, e.g. 'fg 1' for ~mail/cas, 'fg 10' for luv-main, 'fg 11' for luv-talk

Now _that_ is an idea. When using just one mutt, it's so easy for mails to cross since you're blind while composing. Hit send, and I see 7 people have answered the guy's question while I was labouring over a long-winded reply.

(*) yes, i still use mbox files...haven't ever had any really compelling reason to switch to Maildir

Yes, well, there seem to be all sorts of things, these days, which are newer and therefore better. I probably have less than a million emails, and they're in 1041 mailboxes, so I can't think of a reason either. Erik -- There are two kinds of fool. One says, "This is old, and therefore good." And one says, "This is new, and therefore better" - John Brunner, "The Shockwave Rider"

On Mon, 13 Feb 2012, Erik Christiansen wrote:

...

(*) yes, i still use mbox files...haven't ever had any really compelling reason to switch to Maildir

Yes, well, there seem to be all sorts of things, these days, which are newer and therefore better. I probably have less than a million emails, and they're in 1041 mailboxes, so I can't think of a reason either.

I can think of a reason :) Backups of mboxes are a real pain. Append 4k of text, and have to rebackup a 1GB mbox file. Maildir allows me to just link that 4k message into the backup, and effectively keep the earlier snapshots without having to purge them out earlier because I've run out of space on my backup regime. Of course, the reason why I originally moved to Maildir was because I was doing a migration for work, which had been using the original mbox imapd from UW. Gee, that was a poorly written piece of crap (but it has an excuse - being the reference imap server). The server struggled a lot less once I replaced the IMAP daemon :) -- Tim Connors

On Mon, Feb 13, 2012 at 11:26:29AM +1100, Tim Connors wrote:

I can think of a reason :)

Backups of mboxes are a real pain. Append 4k of text, and have to rebackup a 1GB mbox file. Maildir allows me to just link that 4k message into the backup, and effectively keep the earlier snapshots without having to purge them out earlier because I've run out of space on my backup regime.

i can think of a counter-reason that, for me, outweighs your reason :) mbox files can easily be renamed to something like listname.YYYYMM or listname.YYYYMM-YYYYMM and then optionally compressed (mutt can open compressed mbox files). I've got almost two decade's worth of personal mail and mailing list archives organised like this. for some lists, i rename them monthly, for others every three or six months. depends on traffic. very few lists grow to anywhere near 1GB in a month. or even a year. i think the highest-volume list i subscribe to is mythtv-users, and that used about 14MB from 22 Dec 2011 to today. In fact, the entire archives of the mythtv-users list on my system, spanning from Feb 2003 to today, add up to just under 1GB uncompressed. I just ran gzip -9 on them, and that reduced them to 208MB. if i stored my lists in Maildir, then mutt would have to scan through hundreds of thousands of messages for some lists, every time it opened that list. also some filesystems suck when you have more than a few hundred or a few thousand files in a directory. craig -- craig sanders <cas@taz.net.au>

On 2012-02-13 12:56, Jason White wrote:

Craig Sanders <cas@taz.net.au> wrote:

...

mbox files can easily be renamed to something like listname.YYYYMM or listname.YYYYMM-YYYYMM and then optionally compressed (mutt can open compressed mbox files).

I just tested and found that Mutt 1.5.21 (the version currently installed here) can read Mbox files compressed with xz. That's impressive.

It's not really that impressive. It'll read mbox if you tell it how: mattcen@andy:tmp$ cat /etc/Muttrc.d/compressed-folders.rc # Use folders which match on \\.gz$ or \\.bz2$ as [gb]zipped folders: open-hook \\.gz$ "gzip -cd '%f' > '%t'" close-hook \\.gz$ "gzip -c '%t' > '%f'" append-hook \\.gz$ "gzip -c '%t' >> '%f'" open-hook \\.bz2$ "bzip2 -cd '%f' > '%t'" close-hook \\.bz2$ "bzip2 -c '%t' > '%f'" append-hook \\.bz2$ "bzip2 -c '%t' >> '%f'" open-hook \\.xz$ "xz -cd %f > %t" close-hook \\.xz$ "xz -c %t > %f" append-hook \\.xz$ "xz -c %t >> %f" -- Regards, Matthew Cengia

Craig Sanders <cas@taz.net.au> wrote:

I use TB ("icedove", actually - same thing, sans trademark) at work, because the university's exchange server seems to have really low timeouts for imap connections, so mutt gets disconnected in the time it takes me to reply to or even read a msg. i probably should just set up fetchmail or something.

Either Fetchmail or Offlineimap (I haven't used the latter but I've heard favourable comments about it) would be reasonable options. These days, I simply have an MX record for my primary host; if it's unavailable, mail is queued by a secondary and I can just wait for delivery when the primary host returns, or run fetchmail -p etrn --fetchdomains <my-domain-name> <secondary-host> to hasten the process.

Quoting Jason White (jason@jasonjgw.net):

...

These days, I simply have an MX record for my primary host; if it's unavailable, mail is queued by a secondary and I can just wait for delivery when the primary host returns, or run fetchmail -p etrn --fetchdomains <my-domain-name> <secondary-host> to hasten the process.

I stopped having backup MXes for two reasons:

1. People whom you rely upon to do backup MX for your domain proved to be flaky one time too often. (Back in Pleistocene days, it was common to serve as backup MX for each other as a courtesy.) Nothing quite as lovely as seeing a friend 55x-reject all of your domain's mail because he/she 'forgot' to preserve forwarding for your domain during his/her last rebuild.

And very hard to debug when you get occasional "hey I sent an email to you and it didn't go through". Years ago when I was just learning about this stuff we were having problems sending email to someone very occasionally and it turned out to be a rogue secondary.

2. Of course, that still leaves the options of backup MXing for yourself among multiple hosts of one's own, and of paying a commercial concern for the service.

Or having a mutual arrangement with a (trusted) 3rd party where you each run a virtual server at each others site.

In both cases, one hitch is spammers who preferentially dump spam onto high-numbered MXes (against RFC intent) on the plausible theory of MX secondaries tending to have looser antispam. The secondary MX then works hard to pump the relayed spam onto the primary, and various forms of badness can ensue including the primary working to teergrube the secondaries.

This "spam uses highest number MX" used to be a lot more common than it is now, to the point where you could exploit it by having a tertiary MX that always gave a "try again later" and the spambot would give up whilst having no impact on legitimate email. Additionally, I built some logic into my spam filter where mail could be rejected (or the spam score increased) by the secondary if the primary was known to be up. I don't see it happen nearly as often these days though, or maybe my filter is rejecting such things well enough that I don't see the recipient logged at the point of rejection. James

Quoting Jason White (jason@jasonjgw.net):

In general, though, I would advise anyone contemplating a similar arrangement to consider Rick's suggestion and the issues raised in this thread.

It's worth knowing what exactly happens when all of one's domain's MXes are unreachable for mail delivery. The situation's both more and less bad than you might expect. That's in contrast to losing all authoritative nameservice for a domain, which is immediately and severely bad. All the SMTP hosts that have outbound mail queued up for your domain will keep trying. The _customary_ timeout period for permanent delivery failure (55x reject), established by Sendmail's default, is five days -- and I vaguely recall that most Unixey MTAs such as Exim4 at least approximately follow suit (4 days in Exim4's case). This doesn't prevent local admins from overriding the MTA's retry timeout, of course. That de-facto customary expectation is, naturally enough, not universally respected. Courier-MTA defaults to seven days; MS-Exchange Server defaults to two days. And so on. My rule of thumb is that you'll have negligible problems and no mail loss whatsoever if you are able to deploy at least one replacement receiving MTA somewhere in the world, and point your DNS at it, within a day or two of downtime. But it turns out there's one annoyance I hadn't anticipated, that's a consequence of 99% of Internet users' refusal to even try to read automatic notices correctly: I refer to DSNs. Another customary expectation established by Sendmail is that Delivery Status Notifications should start being issued to sender after outbound mail has been undeliverable for _four hours_. Naturally, this MTA default can be locally altered and is nowhere near universally followed by other MTA software packages -- but, to a first approximation, people trying to reach your domain will start getting 4xx (temporary failure) DSN advisories saying something like: 451 Please try again later foo@bar.com... Deferred: 451 Please try again later Warning: message still undelivered after 4 hours Will keep trying until message is 5 days old And do they bloody well bother to _read+ what it says? No, of course they don't. You'll start getting telephone calls and alternate-delivery messages complaining about your domain 'bouncing' the sender's mail. To many such people, any notice of delivery problems is a 'bounce', and understanding what they say is Somebody Else's Problem. -- Cheers, Rick Moen Never ask a sysadmin "What's up?" rick@linuxmafia.com McQ! (4x80)

Quoting James Harper (james.harper@bendigoit.com.au):

This "spam uses highest number MX" used to be a lot more common than it is now, to the point where you could exploit it by having a tertiary MX that always gave a "try again later" and the spambot would give up whilst having no impact on legitimate email.

I remember reading about this ploy a few years ago, and hoisted a mug in honour of whatever Right Bastardly sysadmin invented it. Appreciate the news that the spam wars have moved on from spammer-using-highest-MX days. I've been not paying much attention to the latest skirmishes.

Additionally, I built some logic into my spam filter where mail could be rejected (or the spam score increased) by the secondary if the primary was known to be up.

Another fine idea. For my own use case, the previously cited woes, fixable though they might be, served to make me more seriously weigh whether backup MXes were worth the bother, and judge whether a single-MX scheme could be practical and reliable as alternative. The answers turned out to be no and yes, respectively.

Rick Moen wrote:

Quoting James Harper (james.harper@bendigoit.com.au):

...

This "spam uses highest number MX" used to be a lot more common than it is now, to the point where you could exploit it by having a tertiary MX that always gave a "try again later" and the spambot would give up whilst having no impact on legitimate email.

I remember reading about this ploy a few years ago, and hoisted a mug in honour of whatever Right Bastardly sysadmin invented it.

Appreciate the news that the spam wars have moved on from spammer-using-highest-MX days. I've been not paying much attention to the latest skirmishes.

FYI, $ dig mx cyber.com.au +short 10 null-mx.cyber.com.au. 20 mail.cyber.com.au. 30 exetel.cyber.com.au. 40 tarbaby.junkemailfilter.com. First one has --dport smtp -j REJECT. Second is the "real" MTA, third is the same thing via backup DSL line. Fourth is a tarpit. That, greylisting and DNS RBLs are enough that I don't bother to parse message payloads (except for a few "problem" users, who get crm114).

Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:

That's quite clever, but it looks to me that it will add too many points of potential failure for legitimate mail.

I don't think so - it requires 2 and 3 to fail for an extended period if mail is to be bounced.

Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:

Sure, but it might also "confuse" some mail servers.... I don't know. Guess some will sent to the lowest MX and not to any other mail server and will never get to you, but that would probably be rare.

My understanding is that only scripts used by spammers do that. I've never heard of such behaviour from a legitimate MTA - if there is one, it has such a small share of MTA installations that I think we can safely not care about it. I might be wrong, but I suspect that you have to try the secondaries in order to implement the RFCs properly.

On 15 February 2012 16:33, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:

Therein lies the problem.  If the RFCs are implemented properly then we wouldn't need special exceptions for the likes of Google, Yahoo, Hotmail (M$) or other larger mailers which re-send after 451 greylist messages via different IP addresses :(

Is there an RFC that says you have to retry from the same source IP address? -- Brian May <brian@microcomaustralia.com.au>

On 2012-02-15 19:03, Jason White wrote: [...]

based on a few minutes of reading, I couldn't find anything in RFC2821 that would require it.

FYI, this is obsoleted by RFC5321 now (and updated by 5336). -- Regards, Matthew Cengia

On Wed, 15 Feb 2012, Andrew McGlashan wrote:

On 15/02/2012 7:20 PM, Matthew Cengia wrote:

...

On 2012-02-15 19:03, Jason White wrote: [...]

...

based on a few minutes of reading, I couldn't find anything in RFC2821 that would require it.

And it's odd that Andrew would claim otherwise. It's not even a sane thing to put into the relevant standards.

...

FYI, this is obsoleted by RFC5321 now (and updated by 5336).

Okay, well it doesn't appear to be part of the standards, but it is painful for greylisting. Although I can understand that one IP may become blocked or have issues, whilst an alternate (sending) IP might be fine.

Perhaps the standard needs to be amended to ensure that if there is 451 or other TEMPORARY type error that the mail server should be able to expect further communication from the same sending SMTP host using the original IP if possible -- this would help significantly with greylisting.

And make it impossible to implement certain types of highly available MTA clusters. What happens when the MTA that sent out the email that you have greylisted, then crashes before it gets a chance to try again? Should no other MTA in the cluster be allowed to pull the message off the spool on shared storage and retry itself? The specific implementation of your greylisting is not of concern to people running big MTAs. There are many implementations of greylisting and spam filtering in general. Pick another one that works for the kinds of email from the kinds of people you expect to receive. -- Tim Connors

On 15/02/2012 8:19 PM, Tim Connors wrote:

On Wed, 15 Feb 2012, Andrew McGlashan wrote:

...

On 15/02/2012 7:20 PM, Matthew Cengia wrote:

...

On 2012-02-15 19:03, Jason White wrote: [...]

...

based on a few minutes of reading, I couldn't find anything in RFC2821 that would require it.

And it's odd that Andrew would claim otherwise. It's not even a sane thing to put into the relevant standards.

Sure it is ;-) -- at least I think so... or at minimum consider it.

...

Perhaps the standard needs to be amended to ensure that if there is 451 or other TEMPORARY type error that the mail server should be able to expect further communication from the same sending SMTP host using the original IP if possible -- this would help significantly with greylisting.

And make it impossible to implement certain types of highly available MTA clusters. What happens when the MTA that sent out the email that you have greylisted, then crashes before it gets a chance to try again? Should no other MTA in the cluster be allowed to pull the message off the spool on shared storage and retry itself?

Large MTA clusters seem to have enough issues of their own .... how many Big Pong emails have gone missing or been delayed for extraordinary amounts of time?

The specific implementation of your greylisting is not of concern to people running big MTAs. There are many implementations of greylisting and spam filtering in general. Pick another one that works for the kinds of email from the kinds of people you expect to receive.

Greylisting works on tuples, one part of the tuple is the originating IP address. If you have a bunch of servers with different public facing IP addresses retrying the same email, then that makes it a much greater workload to finally offload the email to the recipient server. If you want to deliver and finish with the email more quickly, then resend it from the same original IP address when you can. If that server dies, then offload it to another server, but don't try to send it from a zillion addresses and making the problem so much greater. -- Kind Regards AndrewM

Andrew McGlashan wrote:

On 15/02/2012 7:20 PM, Matthew Cengia wrote:

...

On 2012-02-15 19:03, Jason White wrote: [...]

...

based on a few minutes of reading, I couldn't find anything in RFC2821 that would require it.

FYI, this is obsoleted by RFC5321 now (and updated by 5336).

Okay, well it doesn't appear to be part of the standards, but it is painful for greylisting. Although I can understand that one IP may become blocked or have issues, whilst an alternate (sending) IP might be fine.

When you run into such a site, it is a good idea to (attempt to) notify the site's sysadmin that he is, indirectly, making life easier for spammers. After all, there's a slight chance that 1) they don't know there's a problem; 2) they're technically comptetent enough to fix it; and 3) enough sysadmins complain that they're actually motivated to do so. Ah, who am I kidding...

At 03:43 PM 2/15/2012, Andrew McGlashan wrote:

Sure, but it might also "confuse" some mail servers.... I don't know. Guess some will sent to the lowest MX and not to any other mail server and will never get to you, but that would probably be rare.

I have seen that problem in the real world. Several years ago, a particularly brain dead version of Sendmail on a SunOS (gives away its age!) box couldn't be convinced to send to the secondary MX in a similar configuration. Unfortunately, that particular brain dead MTA belonged to either a client or partner we needed to be able to receive email from. :) 73 de VK3JED / VK3IRL http://vkradio.com

Quoting Craig Sanders (cas@taz.net.au):

might have been me. i don't recall reading or hearing of it before I started doing it, or before i started mentioning it as a useful anti-spam technique.

of course, it's something so simple and obvious that it's likely to have been independently invented several times.

Which is true of most truly inspired ideas: They're obvious and elegant, after you hear and admire them. Sysadminly judo at its finest. Antispam is an area that exposes a common computerist flaw: Lack of perspective and a tendency to energetically focus on solving the wrong problem -- which in turn produces tactical mistakes like C-R software, huge / slow / buggy Perl regexes, and elaborate schemes to conceal e-mail addresses. It's rare to see a thoughtful balance.

On 12/02/2012, at 19:02, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:

On a side note, it is actually better NOT to cc or bcc those that are subscribed and only to reply to list, but not everyone practices that and it can be difficult to "oblige" people to do that.

except that not all posts to mailing lists are from subscribers of that list. I will continue to use reply-all and let the recipient deal with the single copy not from the list, it duplicate copies. Then, both the poster and the list benefit.

On Sunday 12 February 2012 20:54:47 hannah commodore wrote:

except that not all posts to mailing lists are from subscribers of that list.

Most lists I am on these days only allow subscribers to post to them, the notable exception being the kernel.org lists, but they're a bit of a special case. Coincidentally they're the only lists I'm on which attempt to deliver spam to me. cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP

Chris Samuel <chris@csamuel.org> wrote:

Most lists I am on these days only allow subscribers to post to them, the notable exception being the kernel.org lists, but they're a bit of a special case. Coincidentally they're the only lists I'm on which attempt to deliver spam to me.

I've occasionally encountered spam on a Debian list to which I subscribe, but I suspect it allows posts from non-subscribers too. I've only seen a few instances of actual subscriber addresses being harvested by spammers and then used (with forge headers) to post spam to the mailing lists.