Replies from Outlook to CiviMail messages are unreadable

[ctarascio]

   First, I'd like to say that we are not implying that this is an issue with CivMail. We just need help finding the source of the problem. At the moment, we can't tell if it's an Outlook issue... a CiviMail issue... both... or something else. We use CiviSMTP, but we do not think it is the "culprit".
     The problem is that when Outlook is used to reply to messages sent from CiviMail, the reply message arrives "encoded". Replies from Outlook 2003, while they have a bit of "code" in them, are at least readable. Replies from Outlook 2007 are totally encoded in what we think is base64. They look like this:
"UmVwbHkgaXMgc2VudCBmcm9tIGN0dGVzdEBhZnNjLm9yZyB0byBjdGFyYXNjaW9AYWZzYy5vcmcN
Cg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEFGU0MgTG9zIEFuZ2VsZXMgW21h
aWx0bzpjdGFyYXNjaW9AYWZzYy5vcmddIA0KU2VudDogV2VkbmVzZGF5LCBGZWJydWFyeSAxMywg
MjAwOCAxMjoyNSBQTQ0KVG86IEN5bnRoaWEgVGVzdGFjY291bnQNClN1YmplY3Q6IEN5bnRoaWEg
dGVzdC0xMjIyDQoNCnRlc3QgMTIyMg0KDQoNCjYzNCBTLiBTcHJpbmcgU3QuLCAzcmQgRmxvb3IN
CkxvcyBBbmdlbGVzLCA5MDAxNDoNCm9wdE91dC4xMTE3MS43Ni44NTIuODNhMDc1Y2VkYmFmYWM0
ZTRmZjQ1YmVhODVmNGJhZjY5Mzk1ZDZmN0Bwc3dyLmFmc2Mub3JnOg0KdW5zdWJzY3JpYmUuMTEx
NzEuNzYuODUyLjgzYTA3NWNlZGJhZmFjNGU0ZmY0NWJlYTg1ZjRiYWY2OTM5NWQ2ZjdAcHN3ci5h
ZnNjLm9yZzogDQoNCg0K"

   If anyone can provide any clues about what might be going on here and what is adding this extra level of encoding, we would appreciate it because we are really "stuck".

Thanks,
Cynthia 
 

[Donald Lobo]

We've not had this issue in the past I suspect it is an outlook problem. You might want to cc another email address in the "reply" and see the contents of the email. If the copy also is uuencoded it means outlook 2007 is doing so. I suspect it is some settings in either the email and/or outlook preferences that triggers something like this. You might want to do a google search on this and that might give u a few clues

I'm pretty sure the civicrm code base does not have any code to uuencode an email. I also doubt that amavis would uuencode any email it filters

lobo

[Piotr Szotkowski]

Well, uu/base64-encoded emails are proper citizens of the SMTP world, so we (CiviMail) should handle them transparently (from the user’s point of view). That said, I’m surprised neither Postfix nor AMaViS decoded them.

Cynthia, can you please forward (ideally, as an attachment) such an email to me (shot@civicrm.org), so I can take a look how our default (Postfix → AMaViS → CiviMail) stack handles this?

[Piotr Szotkowski]

After discussing this with Cynthia off-forum I finally understood it.

The issue is that if an user replies to a CiviMail/CiviSMTP generated email with Outlook 2007, the email gets uu-encoded (there’s nothing wrong with this). Unfortunately, something along the way borks the email in an unfriendly way and makes it not uu-decodable.

Cynthia Cc:-ed me from Outlook 2007, and the email is perfectly decodable (and readable) on my end, so my guess would be that something strips/alters the headers that are required for decoding.

Shane, does CiviSMTP do any altering/munging of the Content-Transfer-Encoding: header? It seems to be set to base64 in Cynthia’s example and my Mutt decoded it just fine.

[shane]

we do not take any action to change or modify headers in the qpsmtpd.  maybe qpsmtpd is doing something, I will look into it when I get a chance.  I am not able to prioritize this because I am *not* getting any other complaints from other users and we are sending tens of thousands of emails / day.  maybe just no one is complaining, I really do not know at this point.

-Shane

[ctarascio]

Shane,
    I know you are busy and this is not a critical issue, so get to it when you can. Meanwhile, we'll research qpsmtpd and if we come up with anything useful, we will post it here.

Thanks,
Cynthia

[Piotr Szotkowski]

The last thing that came to my mind is whether there can be a test setup like this:

1. A mailing done with me (shot@civicrm.org) as the contact that replies get forwarded to.
2. A reply to that mailing from the same Outlook 2007 that encodes the emails – with the reply also directly Cc:-ed to shot@civicrm.org

This way I can get two copies of an email, one direct (which my mail program should decode just fine) and one that goes through CiviMail+CiviSMTP (which, as I understand it, comes munged).

I’d diff these two emails and see what changed and borked the decoding.

Cynthia, would it be possible to run the above test with your resources?

[ctarascio]

Yes. Very doable. I'll set it up a little later this morning, so you should have the emails by 10:30EST. Let me know how it goes.

Thanks Piotr,
Cynthia

[Piotr Szotkowski]

Perfect, thanks.

I received both emails and (despite the differences in the headers) it looks to me like the problem lies with the mis-feature that either CiviCRM or qpsmtpd pads the body with two empty lines, one before the load and one after it – this is the diff between the ‘right’ and ‘wrong’ versions of the same email (I edited any valuable email addresses out):

Code: [Select]

--- 1204537477.12537_2.durance:2,S 2008-03-03 10:44:37.000000000 +0100
+++ 1204537476.12534_2.durance:2,S 2008-03-03 10:44:36.000000000 +0100
@@ -1,44 +1,38 @@
-Return-Path: <ctaras…@afsc.org>
+Return-Path: <do-not-re…@pswr.afsc.org>
 X-Original-To: shot@localhost
 Delivered-To: shot@localhost
 Received: from localhost (localhost [127.0.0.1])
- by durance.shot.pl (Postfix) with ESMTP id 2F556E3BF7
- for <shot@localhost>; Mon,  3 Mar 2008 10:44:37 +0100 (CET)
+ by durance.shot.pl (Postfix) with ESMTP id DBA3EE3BF7
+ for <shot@localhost>; Mon,  3 Mar 2008 10:44:36 +0100 (CET)
 Delivered-To: shot@civicrm.org
 Received: from npomail.electricembers.net [209.209.81.73]
  by localhost with POP3 (fetchmail-6.3.2)
- for <shot@localhost> (single-drop); Mon, 03 Mar 2008 10:44:37 +0100 (CET)
-Received: from smtp1.electricembers.net (smtp1 [209.209.81.67])
- by npomail1-in.electricembers.net (Postfix) with ESMTP id BC46511657
- for <shot@civicrm.org>; Fri, 29 Feb 2008 07:24:36 -0800 (PST)
-Received: from mail.afsc.org (mail.afsc.org [209.120.230.100])
- by smtp1.electricembers.net (Postfix) with ESMTP id EC6936781F
- for <shot@civicrm.org>; Fri, 29 Feb 2008 07:24:17 -0800 (PST)
-X-MimeOLE: Produced By Microsoft Exchange V6.5
-Content-class: urn:content-classes:message
+ for <shot@localhost> (single-drop); Mon, 03 Mar 2008 10:44:36 +0100 (CET)
+Received: from smtp3.electricembers.net (smtp3 [209.209.81.69])
+ by npomail1-in.electricembers.net (Postfix) with ESMTP id B85AC115C4
+ for <shot@civicrm.org>; Fri, 29 Feb 2008 07:24:35 -0800 (PST)
+Received: from civismtp.uas.coop (civismtp.uas.coop [216.240.135.165])
+ by smtp3.electricembers.net (Postfix) with ESMTP id 89A71E2BEA
+ for <shot@civicrm.org>; Fri, 29 Feb 2008 07:24:20 -0800 (PST)
+Received: from mail.afsc.org (HELO localhost) (209.120.230.100)
+  (smtp-auth username losangeles, mechanism cram-md5)
+  by civismtp.uas.coop (qpsmtpd/0.40) with ESMTPA; Fri, 29 Feb 2008 07:23:25 -0800
 MIME-Version: 1.0
-Content-Type: text/plain;
- charset="UTF-8"
-Content-Transfer-Encoding: base64
-Subject: RE: Cynthia-mailing with replies set to go to Piotr
-Date: Fri, 29 Feb 2008 10:24:16 -0500
-Message-ID: <B91F108ADD81414AB2C9CB7449D3CEC7044D93C9@DELTA.AFSC.local>
-In-Reply-To: <20080229152214.063C119F008B@civismtp.uas.coop>
-X-MS-Has-Attach:
-X-MS-TNEF-Correlator:
-Thread-Topic: Cynthia-mailing with replies set to go to Piotr
-Thread-Index: Ach65wQyvBaAcWbSQlC1f2nYqo/59AAABCGA
-References: <20080229152214.063C119F008B@civismtp.uas.coop>
-From: "Cynthia Testaccount" <ctt…@afsc.org>
-Sender: "Cynthia Tarascio" <CTaras…@afsc.org>
-To: <reply.11171.94.889.2abef568dfe62bced630b267e58cdaf0777738e2-cttest=afsc.org@pswr.afsc.org>
-Cc: "Piotr Szotkowski" <shot@civicrm.org>
+Subject: Re: Cynthia-mailing with replies set to go to Piotr
+To: shot@civicrm.org
+From: "cttest1 cttest1" <ctt…@afsc.org>
+Reply-To: ctt…@afsc.org
+Content-Type: text/plain; charset="ISO-8859-1"
+Content-Transfer-Encoding: 7bit
+Date: Fri, 29 Feb 2008 10:24:19 -0500
+Message-Id: <20080229152325.D8AD819F008B@civismtp.uas.coop>
 X-ElectricEmbers-MailScanner-Information: Send questions or false-positive reports to h…@electricembers.net
 X-MailScanner: Found to be clean
-X-Spam-Status: not spam, SpamAssassin (score=-3.6, required 6,
- BAYES_00 -2.60, FROM_ORG -1.00, SPF_PASS -0.00)
-X-Envelope-From: ctaras…@afsc.org
-X-Bogosity: Ham, tests=bogofilter, spamicity=0.023662, version=1.0.1
+X-Spam-Status: not spam, SpamAssassin (score=-1.185, required 6,
+ BAYES_40 -0.18, FROM_ORG -1.00)
+X-Envelope-From: do-not-re…@pswr.afsc.org
+X-Bogosity: Ham, tests=bogofilter, spamicity=0.204023, version=1.0.1
+
 
 DQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBQaW90ciBbbWFpbHRvOnNob3RA
 Y2l2aWNybS5vcmddIA0KU2VudDogRnJpZGF5LCBGZWJydWFyeSAyOSwgMjAwOCAxMDoyMyBBTQ0K
@@ -52,3 +46,4 @@
 Ljg4OS4yYWJlZjU2OGRmZTYyYmNlZDYzMGIyNjdlNThjZGFmMDc3NzczOGUyQHBzd3IuYWZzYy5v
 cmc6IA0KDQo=
 
+

Shane, does this look in any way familiar?

[shane]

does not look familiar. 

I belive that CCRM is adding the extra newlines (but I am not totally sure).  I checked the civismtp "de-queued" table and I see that the bodies of the emails are preceded by a single new line and they are followed by a single new line and a single space.  Can the single space be causing problems?

[shane]

ahh,  I just realized that we do XML escape the body to make it safe for SOAP.  I wonder if that is adding the new lines.

Piotr,  r u able to find anywhere in the CCRM code where the newlines might get added to replies?

-Shane

[Piotr Szotkowski]

Hm. I did some digging around and it seems when I compose a mailing in text-only mode, the PEAR Mail_mime::get() method does add an empty line to the end of the body. In other words, plaintext emails that end in civicrm_mailing_spool seem to have one more line than the text template. No extra space, though, and no extra line at the beginning.

I’m not sure this is the culprit, but if it’s indeed on CiviCRM’s side, then (if I’m not mistaken on how CiviSMTP works) you’ll need to bullet-proof it on UAS’s side anyway until all of the clients upgrade, right? (I’m not saying this will solve the problem, as I’m more than willing to fix it on CiviCRM’s side if this is where it lies, just asking.)

I guess my question is, how does forwarding (of replies) actually works? Do you send the received response to the given instance’s SOAP interface and then the instance puts it in its spool table (and so it returns to CiviSMTP), or is there some short-circuit for this?

[Dennis Gray]

Democrats Abroad is also a client of UAS. We have been experiencing this same garbled email problem. What is the status of this now? Has it been determined that the XML escape is the culprit?

Thanks,

Dennis Gray
Democrats Abroad

[shane]

to answer piotr's question.  the only thing we do is parse the email address to figure out what soap call to make and then we make a soap call to the appropriate server according to the api.  we do not alter anything about the emails that come through our system.

Dennis,  I am currently investigating to see if the xml escaping is doing anything to the emails. more soon.

[Dennis Gray]

Any findings yet?