CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Discussion (deprecated) »
  • Alpha and Beta Release Testing »
  • 2.1 Release Testing »
  • Loss of diacritical mark when data is exported
Pages: [1]

Author Topic: Loss of diacritical mark when data is exported  (Read 5147 times)

ctarascio

  • I post frequently
  • ***
  • Posts: 334
  • Karma: 30
    • American Friends Service Committee
  • CiviCRM version: 4.1.3
  • CMS version: Drupal 6.26
  • MySQL version: 5.5.20
  • PHP version: 5.3.13
Loss of diacritical mark when data is exported
September 18, 2008, 01:09:28 pm
Hi,
   The one little snag that appears in the CiviMember export, which may or may not be due to CiviCRM, is the loss of the correct diacritical mark for committee member Steve Rañales. In the export, his last name appears as Rañales. I think I recall similar things happening in an export from Surveymonkey, so this may not be a problem with CiviCRM. On the other hand, if I paste “Rañales” into the spreadsheet, it appears fine. So, I’m not sure where the problem is occurring.

Does anyone have any ideas?

Thanks,
Cynthia

Piotr Szotkowski

  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Loss of diacritical mark when data is exported
September 18, 2008, 10:18:01 pm
What you see (ñ) is the ISO-8859-1’s (and CP-1252’s) rendering of the ñ character when it’s encoded in UTF-8. We use UTF-8 internally, hence this is the encoding we use in the export.

What do you import this file into? I’m pretty sure you’ll see the proper letter if you try opening the file with OpenOffice.org Spreadsheet (and choose UTF-8 as the encoding, if it’s not the default).
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Loss of diacritical mark when data is exported
September 20, 2008, 01:29:45 am
Hi,

(tested in office 2003 on vista).

The problem is that office doesn't offer any filter, just open the csv, considering that the encoding is latin1 (well most probably win).

To make it work: save as ".txt" instead of the .csv
open from excel, then it offers the filter, choose unicode (utf-8) as the charset and it works.

Haven't tried to see if you can set the default charset to utf8 in office, please share if you find a way.

X+
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result

Piotr Szotkowski

  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Loss of diacritical mark when data is exported
September 22, 2008, 02:03:26 am
Quote from: xavier on September 20, 2008, 01:29:45 am
To make it work: save as ".txt" instead of the .csv
open from excel, then it offers the filter, choose unicode (utf-8) as the charset and it works.

Thanks for sharing this workaround!

Quote from: xavier on September 20, 2008, 01:29:45 am
Haven't tried to see if you can set the default charset to utf8 in office, please share if you find a way.

I’m sorry, but I don’t have a Windows box handy, much less one with Office 2003.

I googled a bit and found this quote from Microsoft about the MS08-014 (943985) patch:

Quote
Known issue with this security update
[…]
You may experience any of the following issues after you install security update 943985 on a computer that has Microsoft Office Excel 2003 installed.
• The contents of the following files may be displayed incorrectly when you open the files in Excel 2003:
   • Tab-delimited (*.txt) files that use ANSI encoding
   • Comma-delimited (*.csv) files that use UTF-8 encoding
To resolve these issues, install hotfix 952208. For more information about hotfix 952208, click the following article number to view the article in the Microsoft Knowledge Base:
952208 Description of the Excel 2003 hotfix package: May 10, 2008
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

ctarascio

  • I post frequently
  • ***
  • Posts: 334
  • Karma: 30
    • American Friends Service Committee
  • CiviCRM version: 4.1.3
  • CMS version: Drupal 6.26
  • MySQL version: 5.5.20
  • PHP version: 5.3.13
Re: Loss of diacritical mark when data is exported
September 22, 2008, 11:19:24 am
 Thank you both for this information. At least now we have a work-around. What a pain!!  I did quite a bit of research and there is no easy (automated) way around this issue. Excel just plain does not like utf-8  but, as I said, we can get around this without too much trouble.

Thanks again,
Cynthia

Piotr Szotkowski

  • I live on this forum
  • *****
  • Posts: 1497
  • Karma: 57
Re: Loss of diacritical mark when data is exported
September 23, 2008, 02:16:25 am
I thought a bit about this and there’s not much we can do – we do have a defined ‘legacy’ encoding setting for importing stuff from Excel (or other places which generate non-UTF-8 content), but we can’t use it for export, as we can’t assume all data will ‘fit’ into the legacy encoding (as soon as you have a name with Polish diacritics you can’t export it in ISO-8859-1/CP-1252/Latin 1, for example).
If you found the above helpful, please consider helping us in return – you can even steer CiviCRM’s future and help us extend CiviCRM in ways useful to you.

ctarascio

  • I post frequently
  • ***
  • Posts: 334
  • Karma: 30
    • American Friends Service Committee
  • CiviCRM version: 4.1.3
  • CMS version: Drupal 6.26
  • MySQL version: 5.5.20
  • PHP version: 5.3.13
Re: Loss of diacritical mark when data is exported
September 23, 2008, 07:34:35 am
Dear Piotr,
   Thank you for giving this a second thought. We are quite satisfied that there is a work-around that will only require a couple of extra steps. Maybe someday I can talk my boss into switching to OpenOffice  ;)

Cynthia

xavier

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4453
  • Karma: 161
    • Tech To The People
  • CiviCRM version: yes probably
  • CMS version: drupal
Re: Loss of diacritical mark when data is exported
September 24, 2008, 01:38:40 am
Hi,

Well, if you have a good sells speech that explains that it's excel fault, not civicrm, please share ;)

X+
-Hackathon and data journalism about the European parliament 24-26 jan. Watch out the result
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Discussion (deprecated) »
  • Alpha and Beta Release Testing »
  • 2.1 Release Testing »
  • Loss of diacritical mark when data is exported

This forum was archived on 2017-11-26.