CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail »
  • Using CiviSMTP (Moderator: shane) »
  • Is CiviMail conforming to RFC 5322 when generating the Message-ID?
Pages: [1]

Author Topic: Is CiviMail conforming to RFC 5322 when generating the Message-ID?  (Read 2066 times)

Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Is CiviMail conforming to RFC 5322 when generating the Message-ID?
June 20, 2013, 04:08:58 pm
CiviCRM 4.1.5

I'm not sure where the Message-ID is specified in RFC 5322 is generated but I am posting here first. The problem has to do with both messages disappearing and how his client, Mac OS X Mail, groups mail in threads. My first reply gives more detail as the user described it.

The following is a comment from one of our users about problems with Mac OS X Mail clients using threading to organize their CiviMail inbox. One of the more technical users has suggested that CiviMail violates RFC5322.

Read on:


HI all,

Inge asked me to have a look at this.  I think the problem is that CiviMail is violating the email RFCs, specifically paragraph 3.6.4 of RFC 5322 and of course every paragraph that ever mentions the Message-ID field in every RFC ever.

It seems that rather than generating unique message-ids for every message, CiviMail is now generating a single message-id for all messages from a particular sender in the system.  For the DA-UK chair this constant message-id is

> Message-Id: <fbl-chair-186-fbl@civismtp.com>

For DA-UK secretary it is:

> Message-Id: <fbl-dauksecretary-186-fbl@civismtp.com>

I think this gives us enough information to see what the pattern is.

Duplicate message-ids cause serious problems with the vast majority of standards obeying Mail User Agents.  This should be fixed as a matter of urgency, probably by ensuring that the CiviMail system adds a random element to the message-id.

Kind Regards,

-Alan
« Last Edit: June 22, 2013, 05:32:11 pm by Dennis Gray »

Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Re: Problem is Msg-ids per RFC5322
June 20, 2013, 04:10:58 pm
Here is the original email that prompted this:

As so often happens, I believe that "what we have here … is failure to communicate."

 

Yes, I understand how to deactivate threading (view as conversation) in Mail. I submit that this is merely a workaround to the problem, rather than a fix.

 

I further submit that evidence suggests something has changed on DA's side of the service interface to help create a problem. I will try to outline my thinking and request that you ask the IT team to reconsider.

 

 

Observation 1: The vast majority of email client interfaces now default to a threading (group by conversation) function. So anything that causes different DA mailings to "clump" together in a single thread can be a barrier to communication with an unknown number of our members.

 

Observation 2: Before the CiviMail upgrade was concluded, messages sent by DAUK were grouped by my software based on the identity of the individual mailing. That is to say, a thread was comprised of the original mail received from CiviMail together with responses to that mailing and that mailing only.

 

Observation 3: After the the CiviMail upgrade was concluded, it appears that nearly all messages sent by DAUK are now threaded (grouped/clumped) into a single conversation. My inbox now reports a single "conversation" starting in March 2013 which includes 85 emails, even though these actually represent about 7 or 8 differing mailings (with different subject lines) and their corresponding replies.

 

Observation 4: The sheer size of the "conversation" is starting to cause anomalies in my email client. I anticipate that this will become worse over time.

 

Inference 1: The problem reported by Nancy Platt sounds suspiciously like her software is exhibiting the same symptoms that I see in my software. (E.g., confusion over which email is relevant - difficult to open current email.)

 

Observation 5: I believe that everything being threaded together (clumped) by my email client has the common factor of being related to a message originally sent out via CiviMail "from" the same address. That is, the clumping phenomenon may be triggered by a new implementation of the common "from" field.

 

Observation 6: The CiviMail upgrade also seems to have solved a problem where my Mac OS X Mail client sometimes failed to resolve the html in the body of the email, and mistakenly displayed the underlying html code as plain text. Now emails are displaying clearly in html.

 

 

Hypothesis 1: This "clumping" problem is solely the result of recent upgrades to Mac OS X Mail client?

 

Testing Hypothesis 1: Older emails sent by DAUK prior to CiviMail upgrade work (which still reside in my email box) do not "clump" in the same way. They are being threaded differently. One assumes that if this were merely there exult of a change in client software, the mis-threading (clumping) would exhibit in older emails as well.

 

RESULT: Hypothesis 1 rejected (for the time being).

 

 

Hypothesis 2: Something in DA's upgraded implementation the CiviMail has changed and fools my email client software into believing that all emails from DAUK are part of the same "conversation"? (Perhaps something to do with the way messages are coded? Perhaps something to do with the imposed-standard "from" field? Something else we do differently that is a hook for email client software? Perhaps the system is taking a shortcut in composing the blast and "copying" some aspect of previous mailings from the same person?)

 

I am unable to test Hypothesis 2 because I am not sufficiently familiar with email coding standards, and I have no knowledge of the CiviMail implementation.

 

REQUEST: Please ask someone if they can test Hypothesis 2.

alleyla

  • I post occasionally
  • **
  • Posts: 98
  • Karma: 1
  • CiviCRM version: 4.0.4
  • CMS version: Drupal 7
  • MySQL version: 5
  • PHP version: 5.2
Re: Problem is Msg-ids per RFC5322
June 20, 2013, 04:31:16 pm
Hi Dennis,

We are looking into this issue on our end. Will post an update once we find a solution.

Kind regards,

-Leyla
CiviSMTP Support

Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Re: Is CiviMail conforming to RFC 5322 when generating the Message-ID?
June 22, 2013, 05:30:53 pm
Here are some more details from the user:

Dennis & Co

I have conducted a quick examination of full messages headers on the server-side of my mailbox. (My webmail application continues to display multiple messages that have the same non-unique message id.) I think I can narrow the timeline of when the non-unique message id fault first began.


OBSERVATIONS:

1. Messages from the CiviMail system that I have examined with Date: "Tue, 26 Feb 2013 18:51:24 -0500" or EARLIER appear to exhibit unique message id's.

2. Messages from the CiviMail system that I have examined with Date "Tue, 26 Feb 2013 23:55:50 -0500" or LATER appear to exhibit NON-unique message id's.

3. An email was sent by the DA IT upgrade team early on February 27 describing a series of modifications to the DA CiviMail implementation.


WORKING HYPOTHESIS:

1. The NON-unique message id problem began sometime during the course of the day on February 26, 2013, most probably during an EMEA overnight maintenance window.

2. The probable cause of the NON-unique message id fault was something that was introduced during the CiviMail modification of February 26-27, 2013.


CAVEATS:

1. I have not conducted a comprehensive examination of all message headers in my inbox.

2. The fault of non-unique message id's seems to relate to some message generated by CiviMail, not all. For example, TEST messages sent in advance of a mailing do not seem to exhibit the non-uinque message id fault. So prior to the fault, all messages had unique message ids. After the fault, many (but not all) emails exhibit the non-unique message id fault. Therefore it is a little bit difficult to determine exactly when the fault began to arise.

alleyla

  • I post occasionally
  • **
  • Posts: 98
  • Karma: 1
  • CiviCRM version: 4.0.4
  • CMS version: Drupal 7
  • MySQL version: 5
  • PHP version: 5.2
Re: Is CiviMail conforming to RFC 5322 when generating the Message-ID?
June 25, 2013, 10:58:50 am
We have found a problem in our code that was sometimes resulting in non-unique message ids. We released a fix last night so the issue should be fixed from our end.

However this might not be the only issue if you also have some custom code running on your side that's modifying the IDs before sending to us.

Please test and let us know if you're still seeing this issue.

Kind regards,
-Leyla
CiviSMTP Support


Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Re: Is CiviMail conforming to RFC 5322 when generating the Message-ID?
June 25, 2013, 04:33:48 pm
Thanks for the quick response, Leyla. We will observe the results in our future mailings.
« Last Edit: July 03, 2013, 08:10:58 pm by Dennis Gray »

Dennis Gray

  • Ask me questions
  • ****
  • Posts: 472
  • Karma: 1
  • CiviCRM version: Various. See post.
  • CMS version: Drupal, Wordpress and Joomla. See post.
  • MySQL version: TBA
  • PHP version: TBA
Re: Is CiviMail conforming to RFC 5322 when generating the Message-ID?
July 03, 2013, 08:11:26 pm
Looks like everything is okay now

alleyla

  • I post occasionally
  • **
  • Posts: 98
  • Karma: 1
  • CiviCRM version: 4.0.4
  • CMS version: Drupal 7
  • MySQL version: 5
  • PHP version: 5.2
Re: Is CiviMail conforming to RFC 5322 when generating the Message-ID?
July 03, 2013, 08:54:36 pm
Excellent, thank you for letting us know!

Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail »
  • Using CiviSMTP (Moderator: shane) »
  • Is CiviMail conforming to RFC 5322 when generating the Message-ID?

This forum was archived on 2017-11-26.