On Wed, Nov 11, 2015 at 09:11:37AM +1100, Craig Sanders via luv-talk wrote: Much worse, because it looks like you're not only munging the From: header, you're also munging the Reply-To: header (probably as a Q&D hack to "repair" the damage you've done by munging From:) how much spam, if any, have you ever seen that forged a luv-* list address in the From: header? You're breaking standards and useful functionality (private replies) in order to solve a non-existent problem. craig -- craig sanders <cas(a)taz.net.au>

On Wed, Nov 11, 2015 at 03:10:06PM +1100, Russell Coker wrote: well, then, it's a good thing that absolutely everybody in the world uses either Kmail or K9. And no-one ever needs to specify their own preferred (or only viable/working) address in the Reply-To: header. Reply-To munging is broken, it destroys the sender's intent (and makes it *impossible* to contact them) when you override their Reply-To with yours. Mailman already uses the correct header Mail-Followup-To: (and Sender: for more primitive MUAs). Munging the From: header is similarly broken. there has to be another solution because the one you have chosen is broken. craig ps: surrendering to the arbitrary decrees and whims of giant corporations seems like a bad idea to me. -- craig sanders <cas(a)taz.net.au>

Quoting Peter Ross via luv-talk (luv-talk(a)luv.asn.au): What you said. Craig, I am in sympathy with what you said, but, as far as I can see, Russell is _very_ arguably choosing the best of among various unpalatable options. You already saw my heads-up that Joe-jobs have just (IMO) taken a significant jump in intelligence, and I am predicting that successful delivery of forged-including-envelope mails to mailing lists, believably impersonating legitimate senders, is poised to start becoming a big problem. What stands in the way of that? Only two things: SPF and DMARC. (I notice that yahoo.com still uses DMARC's deprecated 2004 predecessor, DomainKeys.) All forms of mail-redirection have been caught in the crossfire, and are potential victims of Joe-job detection measures: /etc/alias redirections to off-system, ~/.forward files that redirect off-system, CGIs that send mail on behalf of users, mailing list manager, and many other casual methods of reflecting / redireting / impersonating other domains' mail that we became accustomed to, during the Internet's long period of pretending that we're all friends here. The SPF folks (Meng Wong of pobox.com, possibly others) proposed the Sender Rewriting Scheme (SRS) as a workaround for some of these redirectors -- but that scheme is quite awkward and there has been extremely little uptake, and mostly what's been happening is gradual deprecation of those redirection methods (the likes of /etc/alias and ~/.forward) at all. I'm really not surprised that mailing lists are finally at risk from DMARC implementation. Unlike the case with /etc/alias and ~/.forward, we don't want to just stop using them to send mail between systems. So mailing lists -- and Russ ss LUV listadmin -- are increasingly caught between a rock and a hard place. Personally, I think DMARC is overscoped. It _could_ have been limited to just validating the SMTP IP aka validating the 'From ' envelope sender -- but instead its designers went overboard and decided to extend the initial DomainKeys functionality and crypto-sign internal SMTP headers, too. This is one of several reasons I've leaned in the past towards SPF: It does only the essential, hence creates less collateral damage.[1] But both SPF and DMARC are increasingly relevant, and mailing lists that continue to flunk DMARC testing are going to have greater and greater delivery problems. I get why Russell's solution is unappetising. Believe me, I am with you on that. But I'm still waiting to hear anyone suggest a credible alternative. [1] SPF might have suffered a setback when Meng Wong announced in 2004 that he and Microsoft Corporation had agreed to merge SPF and Microsoft's vaguely similar 'Caller ID' scheme. This intended merger failed when, IIRC, it turned out that 'Caller ID' suffered from undisclosed patent problems. Subsequently, 'Caller ID' got deservedly ignored by the broader Internet community, and SPF as originally written has persisted but gets less press than the (IMO) hideously overengineered DMARC scheme.

On Wed, Nov 11, 2015 at 04:17:49PM -0800, Rick Moen via luv-talk wrote: DMARC fundamentally broken if it uses the From: header instead of the envelope From address. With the sole exception of generating the envelope (including From_) and any missing required headers when a message is first sent (e.g. Message-Id, Date), from the POV of an MTA, headers are and should be nothing but irrelevant comments. And there is never **any** excuse for munging the Reply-To: header. Doing that is broken, and has been known to be broken for many years. I guess i'm not telling you anything you don't already know, but SPF works because it doesn't care about headers. SPF checks compare the envelope From against the published SPF DNS record of the domain. if it comes from one of the servers allowed by the SPF record, then accept. otherwise reject (or accept and tag). That's all a mailing list needs. It re-sends the mail with its own envelope-from address (necessary for VERP as well as SPF), so the recipient server can check SPF to see if the sender is allowed to be sending from, e.g., luv.asn.au. The From: header inside the message is irrelevant, that's for end-user MUAs to handle. If the list admin is worried about subscriber forgery then the list server can check SPF on incoming messages. If list subscribers are worried about forgery of their addresses, then they (or their domain admins) can add SPF records to theri domains. SPF works, and it works better the more domains that use it. It was designed as a domain forgery prevention tool, and it works extremely well at doing just that...who knows better than the domain admins which servers are allowed to claim to be sending mail from that domain? How about a domain-keys derivative that restricts itself to checking the envelope From address as it should? From: headers are comments, not addressing information. But that would probably be indistinguishable from SPF and therefore not give giant corporates like yahoo or google the control over all mail that they are working towards. craig ps: and this "Real Name via luv talk" appendage is ugly and wrong. I'll have to edit it out of every reply I send from now on - when I hit 'g' in mutt for a group-reply, I am *not* sending the message to "Rick Moen via luv-talk", I am sending the message to "luv-talk". If I want to reply to Rick Moen privately, I'll hit 'r' for private reply...except, I can't now because the list munges the From: header and my procmail has for many years partially undone the idiocy of Reply-To by renaming Reply-To headers to X-Old-Reply-To Speaking of mutt, this From: munging also breaks mutt's ability to detect list messages sent by me, and to highlight them by showing them in a different colour: color index cyan default '~P' # mail from myself -- craig sanders <cas(a)taz.net.au>

On Wed, 11 Nov 2015, Rick Moen via luv-talk wrote: Wouldn't the market solution to be to continue doing things from the technically correct point of view on infrastructure we maintain, and ignore companies that don't? yahoo senders, sending to a mailing list can't have their mail received by google recievers, and it's because yahoo are being silly regarding choice of standards? Easy. If the yahoo customers care about being listened to, then they'll change providers. -- Tim Connors

On Thu, Nov 12, 2015 at 04:51:22PM +1100, Tim Connors via luv-talk wrote: and if yahoo cares about keeping their users, they'll change what they're doing. pandering to them gives them no reason to change. craig -- craig sanders <cas(a)taz.net.au>

On Thu, 12 Nov 2015, Mark Trickett via luv-talk wrote: Not a problem for a linux mailing list. -- Tim Connors

Quoting Russell Coker via luv-talk (luv-talk(a)luv.asn.au): And I sincerely apologise in advance for arguing on both sides of the issue. ;-> But I did also seriously, on reflection, think it's worth pondering what actual examples of deliverability problems on account of DKIM or DMARC validity _actually_ exist. Maybe you have examples in hand. Over hear across the wide pond, I've not yet seen any, FWIW. Personally, I don't borrow trouble. OTOH (and there I go again, arguing both sides), I am slow to criticise fellow listadmins seeing things differently. -- Cheers, A host is a host, from coast to coast. Rick Moen And nobody talks to a host that's close, rick(a)linuxmafia.com Unless the host that isn't close is busy, hung, or dead. McQ! (4x80)

On Thu, Nov 12, 2015 at 08:56:57PM +1100, Russell Coker via luv-talk wrote: 1. how many yahoo.com subscribers does LUV have? 2. is there any evidence that proves it is worth breaking standards in order to cater to yahoo subscribers? except that it doesn't work well, because it breaks both the From: header and the Reply-To: header. craig -- craig sanders <cas(a)taz.net.au>

On 2015-11-16 06:26, Tim Connors via luv-talk wrote: +1 Adding "via luv-list" is a mistake, regardless of the merits, or otherwise, of munging the headers. I understand it's a way of saying "look! I've screwed with the headers" but is not necessary. Anders

On Mon, 16 Nov 2015 06:02:09 PM Anders Holmström via luv-talk wrote: From: "Anders Holmström via luv-talk" <luv-talk(a)luv.asn.au> From: "Anders Holmström" <luv-talk(a)luv.asn.au> So you're saying that of the above 2 options you prefer the second? Mailman doesn't appear to give that as an option, or if it does it's a site wide option not a per-list option. If I had been given a free choice between the 2 I would probably have chosen the second. But it might be that hacking the source is the only way of getting it. If there is a consense of opinion that the second option is the best way to do it then I will consider the option of hacking the source - but note that it's in a language that I don't use much and it may be beyond my skills in that area. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

Quoting Craig Sanders via luv-talk (luv-talk(a)luv.asn.au): FWIW, I do agree with Craig on this, Russell, though I applaud your benevolent intentions. -- Cheers, <blazemore> omg i love this song Rick Moen <blazemore> Now playing: Unknown Artist - Track 2 rick(a)linuxmafia.com @ 128 Kbps. (0:47/3:24) McQ! (4x80) <Javi> blazemore: Yeah, that's a bad-ass song.

Quoting Mark Trickett via luv-talk (luv-talk(a)luv.asn.au): That is not the problem space. LUV's mailing list server is able to check DKIM/DMARC or SPF records to validate that the IP source matches published data in the incoming mail's claimed sending domain's DNS saying 'trust these IPs as authorised senders for this domain'. That works just fine. However, when the mail is relayed through Mailman and redirected out to subscribers' own SMTP hosts, those SMTP hosts will tend to fail the mail on the same tests that it passed on the initial transmission, because a new IP -- LUV's IP -- is now purporting to _also_ be an authorised sender IP for the sender's domain (even though LUV's IP _isn't_ listed in the sending domain's DKIM/DMARC or SPF DNS records). Russell's solution amounts to 'change the sender upon retransmission to the subscribers'. That solution runs headlong into those of us who consider having our correct 'From: ' information included in our SMTP headers to be a feature rather than a bug, e.g., it has even _worse_ effects than do the infamously wrong-headed idea of 'Reply-To munging' on the intended existence of reply-sender alongside reply-all response modes. Like Reply-To munging on steroids, the current solution makes reply-sender not work at all, and also conceals the sender's contact e-mail address entirely. Which is pretty serious collateral damage, which should IMO not be taken lightly. Making the retransmitted mail still pass DKIM/DMARC creates more damage and is even more of a problem than just making it pass SPF, because DKIM/DMARC tests the internal 'From: ' SMTP header while SPF does not. (Thus, Russell feels compelled to rewrite the 'From: ' header and substitute a -- pardon the expression -- forged sender of &#39;luv-talk(a)luv.asn.au&#39; with GECOS field '[firstname] [lastname] via luv-talk'.) Unfortunately, if Russell wants traffic transiting through LUV's servers to always be able to continue to pass DKIM/DMARC even after retransmission, he'll need to rewrite the transmitted copies pretty much the way he's doing it. The trivial detail of the ' via luv-talk' suffix on the GECOS field can be fixed easily enough by just finding the *.py file that does that and editing it. However, that's cosmetic. E.g., it's slightly less bad if my SMTP headers get forged to say 'From: Rick Moen &lt;luv-talk(a)luv.asn.au&gt;&#39; instead of 'From: Rick Moen via luv-talk &lt;luv-talk(a)luv.asn.au&gt;&#39; -- but that GECOS suffix isn't the real problem numerous of us have raised. It's that it's _supposed_ to continue to say 'From: Rick Moen &lt;rick(a)linuxmafia.com&gt;&#39;m>', the way I sent it, without intermediate software interfering and purporting to change who sent the mail or from where. Ultimately, LUV either messes with user's mail contents in a fashion that conceals and falsifies the original 'From: ' information the sender specified, with additional damage of sabotaging reply-sender operations, or it doesn't. I appreciate what Russell is trying to do, and would not dream of impugning his good intentions, but would rather he not make LUV's server rewrite my (for one) mail in this way. The specific reason I question the necessity is that I'm not seeing all the other mailing lists being driven to this measure. Indeed, I'm not seeing that elsewhere at all, including on the numerous technical mailing lists *I* administer in the San Francisco Bay Area. Once again, I have the highest of respect for what Russell is trying to pull off. I just don't think the side-effects are justifiable. -- Cheers, <BombScare> i beat the internet Rick Moen <BombScare> the end guy is hard rick(a)linuxmafia.com -- seen on IRC McQ! (4x80)

Quoting Russell Coker (russell(a)coker.com.au): {sigh} Um, I'll be charitable and assume you really didn't follow what I said. Here are the main SMTP + envelope headers from a post I sent yesterday to the main Silicon Valley Linux User Group discussion mailing list, as originally sent: From rick(a)linuxmafia.com Mon Nov 16 17: 2:17 2015 Date: Mon, 16 Nov 2015 17:22:17 -0800 From: Rick Moen &lt;rick(a)linuxmafia.com&gt; To: svlug(a)lists.svlug.org Subject: Re: [svlug] procmail (was: Some pretty serious parsing) Here, for comparison, is the retransmitted set of SMTP + envelope headers upon retransmission by SVLUG's Mailman instance and MTA: From svlug-bounces+rick=linuxmafia.com(a)lists.svlug.org Mon Nov 16 17: 2:51 2015 Date: Mon, 16 Nov 2015 17:22:17 -0800 From: Rick Moen &lt;rick(a)linuxmafia.com&gt; To: svlug(a)lists.svlug.org Subject: Re: [svlug] procmail (was: Some pretty serious parsing) Readers will please note that Mailman encodes the relaying information into the envelope 'From ' header but leaves all of my original SMTP headers alone. That is the way MLMs function all over the planet, and nobody in my experience is having DKIM or DMARC-related deliverability problems at this time. If it were otherwise, I would know, on account of being listadmin in a number of places where we'd be in trouble. I'm reasonably certain that the SVLUG, BALUG, and CABAL mailing list's, among many others I'm on, have significant numbers of Yahoo Mail users (even though Yahoo Mail has become really terrible), and we simply are not hearing of, or seeing deliverability problems. [...] I am having a very difficult time believing you do not comprehend _exactly_ what Craig, I, and many others are saying. You would appear to be pretending to not understand, as surely we have all been abundantly clear and lucid. Now, I respect your right to make technical decisions you feel are right with LUV's mailing lists. I've described the damage that the current course of action is, in my view, doing. Please don't try to tell me I'm not seeing what I'm in fact seeing. Rather, if you wish to go ahead and do what you want, good luck to you. I'm going to leave it at that. [rest snipped]

On Wed, Nov 18, 2015 at 11:22:32AM +1100, Trent W. Buck wrote: Or there may already be a satisfactory solution. ========================================================================== TL;DR mailman 2.1.18 recommends using dmarc_moderation_action instead of from_is_list from 2.1.16 http://wiki.list.org/DEV/DMARC ========================================================================== I'd like to thank those who have already advocated (justifiably) against munging the "From:" field, allowing me to avoid the religious debate thus far. Although, since there's potentially a more widely acceptable solution, I must express that this munging can, and should be avoided. Accepting the consensus arrived at by RFC5323 §3.6.2: "In all cases, the "From:" field SHOULD NOT contain any mailbox that does not belong to the author(s) of the message." – [1] Where "SHOULD NOT" as defined by RFC2119 §4 means: "there may exist valid reasons in particular circumstances when the particular behaviour is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behaviour described with this label." – [2] I'm not convinced there's any "valid reasons" to munge the "From:" field in either of the following circumstances. 1) DKIM signing messages sent by the mailing list Unless there's a DMARC policy for a domain, there's no requirement for any parameter in the DKIM-Signature header field to match any other header field. In particular, the DKIM "d=" domain, and the domain in the "From:" field, aren't required to match in the absence of DMARC, (and ADSP, or DK) policy. Since this list's domain doesn't currently (or presumably, intend to) publish a DMARC policy record, I don't see DKIM signing of list mail as a valid reason to munge the "From:" field. 2) Lists receiving mail from domains with a restrictive policy The real issue here is that domains like Yahoo's publish a "p=reject" DMARC policy rule, and receiving domains (like Google's) are politely enforcing that policy by, you know, rejecting the mail that doesn't pass DMARC. Although rewriting the "From:" header on those posts is an option, (such as appending .INVALID to the domain, as per RFC2606) it's an *abysmal* choice, for reasons previously enumerated on this thread. Possible Solution: mailman 2.1.18 recommends using dmarc_moderation_action, instead of from_is_list from 2.1.16 [3] * Unset the from_is_list [4] "Munge From" which causes *all* of messages sent by the list to be munged. * Set dmarc_moderation_action [5] to one of; + Munge From - rewrite the From: and Reply-To: as in from_is_list + Wrap Message - wrap the message as in from_is_list Which should *only* affect messages received from domains with a restrictive DMARC policy (p=reject,p=quarantine) It seems as though this would be palatable to more people, and is probably a reason it was even implemented in 2.1.18 I'd personally prefer the latter, which preserves the original message (and header fields) in an RFC822 MIME sub-part (attachment), much like digests. FWIW: I'd paint the bike shed the following (different) colour: * Although wrapping the post to preserve the original header fields works, it's still a kludge, with those ugly from_is_list header fields, so I'd just reject the post(s) and send (a) sensible DMARC failure report(s) instead * Rejecting the post, as has already been stated by others on this thread, may provide the feedback to signers (via failure reports) to determine a better "market solution" * Disclaimer: I'm aware this solution has been deemed inappropriate for these lists; but in my own defence, during a past life I've endured the insanity of being a large site's postmaster. [1] https://tools.ietf.org/html/rfc5322#section-3.6.2 [2] https://tools.ietf.org/html/rfc2119#section-4 [3] http://wiki.list.org/DEV/DMARC [4] http://list.org/mailman-admin/general-personality.html [5] http://list.org/mailman-admin/sender-filters.html See also; DomainKeys Identified Mail (DKIM) Signatures https://tools.ietf.org/html/rfc6376 DKIM and Mailing Lists https://tools.ietf.org/html/rfc6377 DMARC https://tools.ietf.org/html/rfc7489

"Joel W. Shea via luv-talk" &lt;luv-talk(a)luv.asn.au&gt; writes: IIRC I think somebody said that Mailman breaks the DKIM on the original message. Which provokes the question - why not fix Mailman so it doesn't break the DKIM on the original message? IIRC this is because of the design of Mailman; it interprets the headers and writes them out again, so it can't write them back exactly the same as they were before. I don't have the references handy, I apologise in advance if I got the above wrong. Which seems to be saying we need to mangle the From: header due to poor design decisions in Mailman. Also see the "Annotations by mailing lists" section in https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail - It seems to suggest other options are available, e.g. using SPF to whitelist DKIM, or to use the Sender: header instead of the From: header. -- Brian May &lt;brian(a)linuxpenguins.xyz&gt; https://linuxpenguins.xyz/brian/

On Thu, 19 Nov 2015, Rick Moen via luv-talk wrote: Isn't cascading Received: from .* (.*) by headers completely normal in mail? Isn't a mailing list just another MTA? Can't it just prepend another Received: line and forward it on? If that breaks DMARC/SFP, doesn't that then mean DMARC/SFP is fundamentally broken? -- Tim Connors

Quoting Tim Connors (tim.w.connors(a)gmail.com): You might have noticed that SPF RRs in DNS sometimes have include directives so as to say 'I expect to be at least sometimes forwarding my outboudn mail through _this_ domain's outbound mail servers, so you should include the IPs pointed to that domain's A record, or MX records etc." But, in any case, you have to be able to publish in your domain's SPF RR, in some way or other, all of the places deemed legitimate handlers of your domain's outbound mail, or there's no point. Example: GMail. $ dig -t txt gmail.com +short "v=spf1 redirect=_spf.google.com" $ dig -t txt _spf.google.com +short "v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all" $ I can't speak to how this is handled in DKIM / DMARC, but it's an implementation of the same core concept.

On Mon, Nov 23, 2015 at 11:49:39AM +1100, Peter Ross via luv-talk wrote: and don't forget your Internet-of-Things toaster, microwave, fridge, washing machine and Winston Smith brand telescreen. and, of course, your web-cam enabled toilet-roll holder. craig -- craig sanders &lt;cas(a)taz.net.au&gt;

Quoting Joel W. Shea via luv-talk (luv-talk(a)luv.asn.au): I think you're right. Upon some review reading, my understanding is that MARC is an elaboration (expansion in scope) of DKIM (which in turn was an elaboration of DomainKeys), and DKIM implements digital signing & vetting of various SMTP headers and the body text. It doesn't attempt to validate the envelope, as does SPF. Yes, I do believe you've summarised the matter well.

Rick Moen via luv-talk &lt;luv-talk(a)luv.asn.au&gt; writes: I don't think DKIM actually cares what the sender's address is. Unlike SPF. Just as long as there is a good signature, and it can look up the signature in DNS, I think that is all that matters. -- Brian May &lt;brian(a)linuxpenguins.xyz&gt; https://linuxpenguins.xyz/brian/

On Fri, Nov 20, 2015 at 07:29:04AM +1100, Brian May wrote: [...] I was referring to the mailing list making it's own signatures, but you're correct, in some circumstances mailmain will break signatures, for instance; by rewriting the From/Subject fields where they were signed by the original sender. Mailman can already sensibly handle DKIM signatures. I recall there being a long discussion thread on the mailman list, late last decade, regarding the handling of DKIM signature edge-cases, but I think things have improved since then Only if we want forwarded mail to pass DMARC policy, which is a seperate issue (the second one on my original post) [...]

On Fri, Nov 20, 2015 at 08:05:48PM +1100, Russell Coker wrote: Thanks, I'd forgotten about &lt;201510172046.45136.russell(a)coker.com.au&gt; http://lists.luv.asn.au/pipermail/luv-main/2015-October/008489.html libmail-dkim-perl behaves correctly in that scenario, since the header specified "c=simple/simple", where any change in that header will also break the signature, IIRC. Perhaps using "c=relaxed/relaxed" (as Google does) would resolve that issue? However, that would depend on the sender's signing policy (over which not all authors will have control), so the following... ...seems to be the ideal solution. Even if that weren't the case; it seems like the previous header mangling issue *might* be because of the Python standard library, and I'm unsure how easily it can be fixed. Assuming the header issue was also fixed... ...or there were also a way to disable it (or revert the change that introduced it) I'll take a *guess*, and assume it's to avoid shaving a yak; the 2.1.x maintainers don't have the time (or can justify) working around DKIM edge cases by also having to work around the Python standard library. I agree, where "sensibly handle" means "preserve" (rather than "remove") In which case, I don't think there's a solution that doesn't require at least a DKIM aware mailing list manager (of which it appears Mailman 2.1.x is not) In using the standard library to parse the message, that's exactly what appears to be happening. No, seems you were right, on both counts. Yes, unless DKIM signatures could be preserved; but again, only for those messages sent from a domain publishing a restrictive DMARC policy

Russell, thank you for the response, and for the maintenance you are providing for LUV infrastructure. On Fri, Nov 20, 2015 at 01:27:18PM +1100, Russell Coker wrote: [...] [...] [...] If that's the case, either: from_is_list is also set, or dmarc_moderation_action is broken; as it should only be munging the "From:" field on messages arriving from domains with a restrictive DMARC policy. I concede the following point may be subjective; but I think wrapping is actually less-ugly, since it gives a clearer picture of what's happening, than simply munging the header; and agree it's more difficult for readers, in that it adds an extra keystroke for me to open the message in my MUA. (but I actually prefer that in the case of digests) [...] I agree that's a fair interpretation, but (IMO) also implies that the forwarder of that message now claims authorship. [...] [...] I should clarify this circumstance; it's only relevant where luv.asn.au is adding its own DKIM-Signature, it does not require the domain in the "From:" header field to align. I reluctantly agree, but *only* for messages sent from a domain with a DMARC policy of reject/quarantine. Hence dmarc_moderation_action being preferred *instead of* from_is_list. Yes, I was adding my support to the consensus against rewriting that field. Again, I reluctantly agree it's probably the only solution that meets LUV requirements, but *only* where it's absolutely necessary, and not on every message forwarded by the mailing list(s). I'd contend that corporations implementing standards in their own interests, and ignoring legitimate real world usage; are making things difficult. Ideally, I'd also like to see the standard improved to allow receivers to defer to the additional signatures of "trusted" third-party relay/forwarders; although, determining trust is another problem.