CiviCRM Community Forums (archive)

Hi, I just went ahead and checked this bug against 3.0.1

If you create a HTML message in CiviMail that contains special characters, e.g. "ěščřžýáé", some of the letters are converted to entities ("ěščřžýáíé").

Unfortunately, the "entitized" letters are then lost on the automatic generation of a plain-text version of the message (if no plain-text version is directly added in the wizard).
Moreover, if you do this in "simple mail", the plain-text body is not generated at all.

Maybe this is an easy fix that could make it to the next release. I am quite surprised Piotr hasn't encountered this issue before as we both use these these weird-accented languages

code from CiviMail:
...
Content-Type: multipart/alternative;
boundary="=_aea2b3b4038814efa74b3546d37fb406"
Date: Tue, 27 Oct 2009 23:16:04 +0000

--=_aea2b3b4038814efa74b3546d37fb406
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8"

Sample Header for HTML formatted content.

ěčřž

ĚČŘ

<div class="location vcard"><span class="adr"><span class="street-address">S 15S El Camino Way E</span>
<span class="locality">Collinsville</span>, <span class="region">CT</span> <span class="postal-code">6022</span>
<span class="country-name">United States</span></span></div>http://localhost/drupal/index.php?q=civicrm/mailing/optout&reset=1&jid=2&qid=2&h=ec3ec6d908adc1a2 Sample Footer for HTML formatted
content.
--=_aea2b3b4038814efa74b3546d37fb406
Content-Transfer-Encoding: 8bit
Content-Type: text/html; charset="utf-8"

Sample Header for HTML formatted content.
<p>ě&scaron;čřž&yacute;&aacute;&iacute;&eacute;</p>
<p>Ě&Scaron;ČŘ&Yacute;&Aacute;&Eacute;</p>
<p>&nbsp;</p>
<p><div class="location vcard"><span class="adr"><span class="street-address">S 15S El Camino Way E</span><br /><span class="locality">Collinsville</span>, <span class="region">CT</span> <span class="postal-code">6022</span><br /><span class="country-name">United States</span></span></div>http://localhost/drupal/index.php?q=civicrm/mailing/optout&amp;reset=1&amp;jid=2&amp;qid=2&amp;h=ec3ec6d908adc1a2</p>
Sample Footer for HTML formatted content.
--=_aea2b3b4038814efa74b3546d37fb406--

and from "simple mail"
Date: Tue, 27 Oct 2009 23:09:54 +0000

--=_1ef0b528da8b4bbd36aa84a27a601a39
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8"


--=_1ef0b528da8b4bbd36aa84a27a601a39
Content-Transfer-Encoding: 8bit
Content-Type: text/html; charset="utf-8"

<p>ě&scaron;čřž&yacute;&aacute;&iacute;&eacute;</p>
--=_1ef0b528da8b4bbd36aa84a27a601a39--


And I just noticed that the default {domain.address} token is not 'plaintextized' at all. The issues above seem to be more serious to me, tho.
Can someone please file a bug report, Piotr?

Thank you, Boris

The problem is that the entities are created by the FCKeditor by default and htmlToText then discards entities.

The solution is to change the default FCKeditor behavior (which as I found out on the net has been quite a frustration for many users for some time).
It's in fckconfig.js on line 65 - should be FCKConfig.ProcessHTMLEntities = false;
The two lines below should also be set to false (Latin and Greek entities).
I suggest this customization becomes the default setting for FCKeditor in CiviCRM.

Regarding "simple mail" not having any content at all in the plaintext section, I'll investigate that later. Most probably it's a different issue.

Hi Piotr, not sure if I understood your note correctly, but shouldn't it be the other way around? If there's no plain text, try to fetch and convert HTML section into plain text and use that (that's what CiviMAIL does and currently not "simple mail"), instead of discarding the whole plaintext section (that's what I think you're saying you just implemented)?

I created CRM-5334 to track this.

I fixed the entities issue in r24966 – if you want to, you can apply the patch or overwrite your packages/html2text/class.html2text.inc file with the one from http://svn.civicrm.org/civicrm/branches/v3.0/packages/html2text/ and verify whether this works for you.

I’ll tackle the HTML → text conversion when the text part is missing sometime later this week.

Thanks, works great. With the html_entity_decode() call this workaround doesn't even require changing the default setting for entities in FCKeditor. (Good for the future - no need to carry over customizations for two packages but just one).

Right, that was the reason to fix it on the html2text’s side. Also, if any other WYSIWYG editor decides to escape non-US-ASCII characters we’ll handle that automatically.

Please also consider removing the ending line break ("\n") ) on the <td>...</td> conversion line:
        "\t\t\\1\n",                            // <td> and </td>
That would make tables look great even in plaintext.

Ok, after some consideration (to the tune of ‘but what if the <td>…</td>s all are strung together on one line?’) I did that in r25064, but as it’s not very urgent, I’m a bit reluctant to make the fix on the 3.0 branch (don’t want to change people’s emails when they upgrade from 3.0.2 to 3.0.3).

You can test this by getting class.html2text.inc from http://svn.civicrm.org/civicrm/trunk/packages/html2text/