CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Replies from Outlook to CiviMail messages are unreadable
Pages: 1 [2] 3 4

Author Topic: Replies from Outlook to CiviMail messages are unreadable  (Read 30058 times)

shane

  • I post frequently
  • ***
  • Posts: 265
  • Karma: 16
Re: Replies from Outlook to CiviMail messages are unreadable
March 26, 2008, 12:31:02 pm
not yet,  we are in the middle of a migration.  you should have received an email about it this morning.  let me know if you did not.

when the migration is over we will investigate this further.

thanks for your patience,

-Shane

Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Re: Replies from Outlook to CiviMail messages are unreadable
April 01, 2008, 05:40:50 pm
Hi Shane,

We are trying to get a group of users together with various email clients to  see if the problem may be client related. In the meantime, let us know if you have found anything on the CiviSMTP side.

I got my beta site working a while ago and will be sending out various mailings to test and confirm things. So far I can only reply using Horde but will try Evolution and Outlook 2007 later.

Thanks,

Dennis

shane

  • I post frequently
  • ***
  • Posts: 265
  • Karma: 16
Re: Replies from Outlook to CiviMail messages are unreadable
April 02, 2008, 07:41:08 am
ok,  all looks OK on our end with the XML escaping. so that is not an issue.

please let me know what you find out.

-Shane

Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Re: Replies from Outlook to CiviMail messages are unreadable
April 03, 2008, 02:05:43 pm
Has anyone definitely been able to narrow this down to Outlook 2007? If that is the case, I can at least narrow my research.

Piotr Szotkowski

  • Moderator
  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Replies from Outlook to CiviMail messages are unreadable
April 04, 2008, 12:41:38 am
My gut feeling is this happens with any base64-encoded emails – it’s just that Outlook 2007 sends these more often than other clients (i.e., is more easily – always? – triggered into sending base64 instead of ~plain text).
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

ctarascio

  • I post frequently
  • ***
  • Posts: 334
  • Karma: 30
    • American Friends Service Committee
  • CiviCRM version: 4.1.3
  • CMS version: Drupal 6.26
  • MySQL version: 5.5.20
  • PHP version: 5.3.13
Re: Replies from Outlook to CiviMail messages are unreadable
April 04, 2008, 06:06:04 am
I can't state for sure that it is definitely just an Outlook 2007 issue. What I can say is that we've tested replies from various other clients and they are not encoded. Replies from Outlook 2003 have minimal encoding as well, but the message is at least readable. If you need help testing this just let me know.

Thanks,
Cynthia

Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Re: Replies from Outlook to CiviMail messages are unreadable
April 04, 2008, 04:41:18 pm
I just performed a test sending through CiviSMTP and replying using an Evolution client. Here is the resulting source of the garbled reply.

Return-Path: <do-not-reply@lists.democratsabroad.org>
Delivered-To: 6-dmgray0325@fareast.com.au
X-Spam-Checker-Version: SpamAssassin 3.1.9 (2007-02-13) on
   ip-208-109-30-20.ip.secureserver.net
X-Spam-Level:
X-Spam-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00,UPPERCASE_25_50
   autolearn=ham version=3.1.9
Received: (qmail 31898 invoked from network); 5 Apr 2008 10:29:27 +1100
Received: from civismtp.uas.coop (67.212.170.242)
  by ip-208-109-30-20.ip.secureserver.net with SMTP; 5 Apr 2008 10:29:27 +1100
Received: from lists.democratsabroad.org (HELO localhost) (208.69.4.60)
  (smtp-auth username da-ccrmu, mechanism cram-md5)
  by civismtp.uas.coop (qpsmtpd/0.40) with ESMTPA; Fri, 04 Apr 2008 18:29:12 -0500
MIME-Version: 1.0
Subject: Re: Mail to test Mantis Issue #341
To: dmgray0325@fareast.com.au
From: "Dennis Gray" <dmgray0325@fareast.com.au>
Reply-To: dmgray0325@fareast.com.au
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
Date: Fri, 04 Apr 2008 19:29:27 -0400
Message-Id: <20080404232912.99E39124C090@civismtp.uas.coop>



--=-YRzFqvp5mNvHJaklbK68
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

This is an HTML formatted reply sent from Evolution.

--=-YRzFqvp5mNvHJaklbK68
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

&amp;lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"&gt;
&amp;lt;HTML&gt;
&amp;lt;HEAD&gt;
  &amp;lt;META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8"&gt;
  &amp;lt;META NAME="GENERATOR" CONTENT="GtkHTML/3.14.3"&gt;
&amp;lt;/HEAD&gt;
&amp;lt;BODY&gt;
This is an HTML formatted reply sent from Evolution.
&amp;lt;/BODY&gt;
&amp;lt;/HTML&gt;

--=-YRzFqvp5mNvHJaklbK68--




torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Replies from Outlook to CiviMail messages are unreadable
April 04, 2008, 05:16:10 pm
Careful examination of the headers shows that some step along the way is stripping out the content type headers.

Multipart MIME wants a header something like this:
Quote
Code: [Select]
Content-Type: multipart/alternative;
              boundary="----=_NextPart_000_0046_01C89645.927685D0"
The "boundary" part varies from message to message, and is used to deliminate the parts of the MIME package.

In the garbled emails, what you have instead is something like this:
Quote
Code: [Select]
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
This is an explicit instruction to display the email as simple text.  The MIME related parts of the packaging will get ignored, and you get a garbled email.

I'm going to take a quick look at the CiviMail sources, but it would not surprise me at all if they are simply nailing down the Content-Type without checking its suitability when handling an email reply.

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Replies from Outlook to CiviMail messages are unreadable
April 04, 2008, 05:35:56 pm
I took a look at the CiviCRM 1.9 sources.  The following from CRM/Mailing/BAO/Reply.php is almost certainly wrong:
Code: [Select]
        $message =& new Mail_Mime("\n");
        $headers = array(
            'Subject'       => "Re: {$mailing->subject}",
            'To'            => $mailing->replyto_email,
            'From'          => $from,
            'Reply-To'      => empty($replyto) ? $dao->email : $replyto,
            'Return-Path'   => "do-not-reply@{$domain->email_domain}",
        );
        $message->setTxtBody($body);
        $b = $message->get();
        $h = $message->headers($headers);

If you look at the code in packages/Mail/mime.php,  you'll see that the Mail_Mime object sets its headers for a text message unless you explicitly ask it to do otherwise.  In addition, the body of the email is passed on to the object via the setTxtBody message.  This is fine if your message is text.  I'm guessing that there isn't a check to make sure that the body was really text.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Replies from Outlook to CiviMail messages are unreadable
April 04, 2008, 05:50:01 pm

since you are on the right track towards a solution, wanna try to figure out what we need to change to fix the problem. We obviously need to check the type and then extract the previous email text/html bodies properly. would be great if you could figure it out and submit a patch:)

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Replies from Outlook to CiviMail messages are unreadable
April 04, 2008, 06:04:49 pm

An easy option might be to include the mail as a mime attachment instead of using the TxtBody (which is obviously wrong).

So using this function in Reply.php from mime.php

Code: [Select]
    /**                                                                                                                     
     * Adds a file to the list of attachments.                                                                               
     *                                                                                                                       
     * @param  string  $file       The file name of the file to attach                                                       
     *                             OR the file data itself                                                                   
     * @param  string  $c_type     The content type                                                                         
     * @param  string  $name       The filename of the attachment                                                           
     *                             Only use if $file is the file data                                                       
     * @param  bool    $isFilename Whether $file is a filename or not                                                       
     *                             Defaults to true                                                                         
     * @return mixed true on success or PEAR_Error object                                                                   
     * @access public                                                                                                       
     */
    function addAttachment($file, $c_type = 'application/octet-stream',
                           $name = '', $isfilename = true,
                           $encoding = 'base64')

lobo


A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Replies from Outlook to CiviMail messages are unreadable
April 05, 2008, 03:31:39 pm
Lobo --

It's an issue of cost/benefit here:  I'm not familiar enough with your implementation to make this fix without investing an engineering day or more just learning the call graph for a reply.  This is necessary because I do not believe your fix will work at all unless the headers of the incoming message is parsed to get its original content type.  Otherwise, you are just attaching or forwarding a chunk of text rather than simply sending the chunk of text directly.  This would not be a fix.

If you can tell me (or have one of your engineers) tell me who is ultimately calling the "reply()" function (which is passed the message body only;  I'm guessing the caller or its caller has the information I need), then I can start trying out fixes.  But I don't think I can tell a client in good conscience that it's a good cost/benefit to them to pay for the prep I'd need to do to get the that information myself.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Replies from Outlook to CiviMail messages are unreadable
April 05, 2008, 08:53:18 pm

The code is called from amavisd (approx line 16667) via the SOAP interface. So the arguments are marshalled and sent in via Perl/SOAP

regarding the cost/benefit issue, i'm a bit unsure how to answer that. It will take us some time also to figure out the code, replicate the problem and fix it. As an open source project we do appreciate contributions and patches to existing issues

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

torenware

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 4
Re: Replies from Outlook to CiviMail messages are unreadable
April 05, 2008, 10:19:01 pm
Quote from: Donald Lobo on April 05, 2008, 08:53:18 pm
regarding the cost/benefit issue, i'm a bit unsure how to answer that. It will take us some time also to figure out the code, replicate the problem and fix it. As an open source project we do appreciate contributions and patches to existing issues

lobo

It's an issue of knowledge of the code base.  I know parts of the source very well, and in those areas, I contribute.  Many of the areas I work in are not documented at all, so there's a lot of time that was spent reading sources, working code through debuggers, and so on.  I'm happy to help other developers with those parts of the code, and have.

But CiviMail is a largely independent part of the project, and there are limits to what what makes sense to as far as investing in the knowledge of that source base.  For a fix like this, the actual fix is a reasonable sized project, either for me as a pure volunteer, or as a consultant who contributes back something that a client pays for.  But getting set up for the fix is a prerequisite,  and in this case, setting myself to set up the full CiviMail stack and hook it into my development environment is a much bigger job.  I'm guessing a couple of days work.  This is especially true since the client uses Shane Hill's very cool service.  So I don't even have a complete set up to work with for this purpose.

I can justify the time on the fix, but not the set up.  Since Piotr wrote most of that code AFAIR,  he can tell me enough about the internals so that I can set up a simple test program that will just simulate the processing of a reply.  This is a reasonable amount of work.

This is a basic truth in open source projects:  what gets documented gets worked on.  Often, reading the source is enough documentation on a project like this.  Just not in the case of CiviMail.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Replies from Outlook to CiviMail messages are unreadable
April 05, 2008, 11:31:33 pm

Please file an issue and we'll take care of this in the 2.1 release cycle

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

Pages: 1 [2] 3 4
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Replies from Outlook to CiviMail messages are unreadable

This forum was archived on 2017-11-26.