CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site

This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Discussion »
  • Internationalization and Localization (Moderators: Michał Mach, mathieu) »
  • Problem with frontend search fields with Polish characters (Joomla Civicrm)
Pages: [1]

Author Topic: Problem with frontend search fields with Polish characters (Joomla Civicrm)  (Read 2936 times)

lukasht

  • I’m new here
  • *
  • Posts: 9
  • Karma: 0
  • CiviCRM version: 4.4
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.4.19
Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 13, 2013, 03:49:25 am
Hello,

I have a similar problem to the one described here: http://forum.civicrm.org/index.php/topic,10313.0.html. Have you already managed to resolve it somehow? Database civicrm tables encoding is utf-8 unicode, here is the link to the menu position (Profile Search form): http://oirp.lublin.pl/lista-radcow-prawnych. When searching, for example, for Jusińska Agnieszka, writing Jusińska gives me nothing; but writing Agnieszka in the Imie field displays her among others. Also, writing her company number, which is LB-1643, gives the proper result. So the problem is when searcing the word with Polish sign. Any idea?

I'll be very grateful for any help.

Lukasz Kaminski

mathieu

  • Administrator
  • Ask me questions
  • *****
  • Posts: 620
  • Karma: 36
    • Work
  • CiviCRM version: 4.7
  • CMS version: Drupal
  • MySQL version: MariaDB 10
  • PHP version: 7
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 13, 2013, 06:20:17 am
Hmm, to make things even weirder, searching for "Jusinska" does work.
This has to do with the MySQL "collate" option that determines how strings are treated.

Can you check:

1- Does the quicksearch of CiviCRM work? i.e. the search box in the upper-left corner of the screen. I tried creating a contact with the same name, and it worked for me. (you can also try to reproduce on the demo site: http://joomla.demo.civicrm.org/administrator/).

2-  "Show create table civicrm_contact;" and see what kind of collate/charset it returns? For example:

Code: [Select]
show create table civicrm_contact;
[.... very long table description ..]
ENGINE=InnoDB [...] CHARSET=utf8 COLLATE=utf8_unicode_ci

In utf8_unicode_ci collate, mysql considers that "ń" and "n" are the same letter, so it explains why "Jusinska" would work, but does not explain why "Jusińska" does not.

Reference: http://stackoverflow.com/questions/3506338/mysql-and-polish-words

Warning: I do not recommend changing the "collate" of the table unless you really know what you are doing, but if you search on the topic, you will inevitably find some people who may recommend it. I suspect the problem is elsewhere, since I cannot reproduce the error on a demo site.

CiviCamp Montréal, 29 septembre 2017 | Co-founder / consultant / turn-key CiviCRM hosting for Quebec/Canada @ SymbioTIC.coop

lukasht

  • I’m new here
  • *
  • Posts: 9
  • Karma: 0
  • CiviCRM version: 4.4
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.4.19
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 13, 2013, 12:28:50 pm
Thank you for your quick reply. Backend searching works fine (both "n" and "ń" characters are recognized as same). The problem is only in the front end searching. Changing the database collate to utf8_polish_ci doesn't change anything  :(

mathieu

  • Administrator
  • Ask me questions
  • *****
  • Posts: 620
  • Karma: 36
    • Work
  • CiviCRM version: 4.7
  • CMS version: Drupal
  • MySQL version: MariaDB 10
  • PHP version: 7
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 13, 2013, 01:04:37 pm
Can you try to reproduce the problem on the demo site with a profile? http://joomla.demo.civicrm.org/administrator/

(sorry I don't have time to do more tests locally, but if we can confirm that it's a bug with CiviCRM, and not the specific hosting/site, we can open a bug report and ask more people to look into it)
CiviCamp Montréal, 29 septembre 2017 | Co-founder / consultant / turn-key CiviCRM hosting for Quebec/Canada @ SymbioTIC.coop

lukasht

  • I’m new here
  • *
  • Posts: 9
  • Karma: 0
  • CiviCRM version: 4.4
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.4.19
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 14, 2013, 12:56:13 pm
I can't reproduce the problem on the demo site, as the demo admin panel doesn't allow to create menu item. The problem is only in the front end search engine (menu item Search Profile), not in the admin part od civicrm. No idea at all...

lukasht

  • I’m new here
  • *
  • Posts: 9
  • Karma: 0
  • CiviCRM version: 4.4
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.4.19
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 15, 2013, 06:28:03 am
I managed to simulate the problem on the demo site, exactly in same steps as I get into it on my own page. So I created a Profile with only one field - Surname. Then I added an Individual person - Jusińska. Then I entered this link: http://joomla.demo.civicrm.org/administrator/index.php?option=com_civicrm&task=civicrm/profile&reset=1&gid=13

On your demo site, searching for "Jusińska" returns proper result, the same link on my site returns nothing.

So there must be a problem with my website configuration... Now what...?

mathieu

  • Administrator
  • Ask me questions
  • *****
  • Posts: 620
  • Karma: 36
    • Work
  • CiviCRM version: 4.7
  • CMS version: Drupal
  • MySQL version: MariaDB 10
  • PHP version: 7
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 18, 2013, 12:54:15 pm
Seems to be something the server is doing, maybe enforcing a character set somewhere. Some hosting providers will enforce it in Apache, but I would assume that the browser sends the character set when submitting a form, so it should not make a difference.

Could you try adding the following at the bottom of your civicrm.settings.php?

Code: [Select]
mysql_set_charset('utf8');
(or in your Joomla settings file)

Otherwise, it would be necessary to log the value received from the form, somewhere in PHP/CiviCRM code. The simplest way is probably to add "die(print_r($_POST, 1));" in your civicrm.settings.php just before you submit the form (NB: this would affect all visitors, so should be done on a test server, or put it there only for a few seconds).

Although I guess that before going into further debugging, you may want to talk to your hosting provider about the issue, just to be sure they don't have something converting character sets.
CiviCamp Montréal, 29 septembre 2017 | Co-founder / consultant / turn-key CiviCRM hosting for Quebec/Canada @ SymbioTIC.coop

lukasht

  • I’m new here
  • *
  • Posts: 9
  • Karma: 0
  • CiviCRM version: 4.4
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.4.19
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 18, 2013, 01:57:03 pm
Adding this code "mysql_set_charset('utf8');" at the bottom of civicrm.settings.php (in the components catalog) doesnt change anything.

But:

Just a few moment ago I have googled a similar problem (dated somewhen in 2009). The point (then and in my case as well) was to add in main joomla index.php this line: setlocale(LC_ALL, 'pl-PL.utf8', 'pl_PL.UTF8', 'pl_PL.utf8', 'pl_PL.UTF-8', 'pl_PL.utf-8', 'polish_POLISH.UTF8', 'polish_POLISH.utf8', 'pl.UTF8', 'polish.UTF8', 'polish-pl.UTF8', 'PL.UTF8', 'polish.utf8', 'polish-pl.utf8', 'PL.utf8');

That resolved the problem. I am not quite sure what I have done, because Joomla itself and SQL tables were already configured as UTF-8, but after adding the above line of code it all works now:)

mathieu

  • Administrator
  • Ask me questions
  • *****
  • Posts: 620
  • Karma: 36
    • Work
  • CiviCRM version: 4.7
  • CMS version: Drupal
  • MySQL version: MariaDB 10
  • PHP version: 7
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 18, 2013, 04:09:29 pm
Wow, that's weird. Thanks for sharing the fix.

One caveat, however, is that you may run into a bug if you use address geocoding.

LC_ALL sets LC_COLLATE (for regular expressions), LC_CTYPE (case-sensitive comparisons), LC_MESSAGES (string translation), LC_MONETARY, LC_NUMERIC (ex: decimals format) and LC_TIME.

I'm guessing in your case, it was LC_CTYPE that was causing problems. Could you test to see if just setting that, instead of LC_ALL, still fixes the issue?

Another option: try using "native gettext", if your host supports it (it should, if the pl_PL setlocale works).

http://wiki.civicrm.org/confluence/display/CRMDOC/i18n+Administrator%27s+Guide#i18nAdministrator%27sGuide-Usingnativegettextsupport/setlocale

In short:

* in your civicrm.settings.php, add:
Code: [Select]
define('CIVICRM_GETTEXT_NATIVE', 1);* move civicrm/l10n/pl_PL/civicrm.mo to: civicrm/l10n/pl_PL/LC_MESSAGES/civicrm.mo

It should also provide a good performance boost.
CiviCamp Montréal, 29 septembre 2017 | Co-founder / consultant / turn-key CiviCRM hosting for Quebec/Canada @ SymbioTIC.coop

lukasht

  • I’m new here
  • *
  • Posts: 9
  • Karma: 0
  • CiviCRM version: 4.4
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.4.19
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 18, 2013, 11:52:51 pm
Changing LC_ALL to LC_CTYPE works fine, still fixes the issue, thanks:)

However, the "native gettext" method returns proper value (jusińska) in search engine, but everything is back in English:( (looks like this fix could change the translation language back to English)

mathieu

  • Administrator
  • Ask me questions
  • *****
  • Posts: 620
  • Karma: 36
    • Work
  • CiviCRM version: 4.7
  • CMS version: Drupal
  • MySQL version: MariaDB 10
  • PHP version: 7
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 19, 2013, 12:06:52 am
Great to hear that the issue is narrowed to a very specific fix (LC_CTYPE). Also reminds me that I really need to work on making "native gettext" easier to enable.

If the strings are in English, can you double-check if you moved:

Code: [Select]
* from civicrm/l10n/pl_PL/civicrm.mo
* to   civicrm/l10n/pl_PL/LC_MESSAGES/civicrm.mo

Thanks,
Mathieu
CiviCamp Montréal, 29 septembre 2017 | Co-founder / consultant / turn-key CiviCRM hosting for Quebec/Canada @ SymbioTIC.coop

lukasht

  • I’m new here
  • *
  • Posts: 9
  • Karma: 0
  • CiviCRM version: 4.4
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.4.19
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 19, 2013, 12:41:04 am
You are right - I made a mistake moving file civicrm.mo to the wrong catalog. Now, after modyfing civicrm.settings.php (both from administrator and components catalog) and moving file civicrm.mo, along to your instructions, all works fine! Thank you very much for your support!

mathieu

  • Administrator
  • Ask me questions
  • *****
  • Posts: 620
  • Karma: 36
    • Work
  • CiviCRM version: 4.7
  • CMS version: Drupal
  • MySQL version: MariaDB 10
  • PHP version: 7
Re: Problem with frontend search fields with Polish characters (Joomla Civicrm)
November 19, 2013, 10:31:36 am
Great, thanks. This confirms that 'native gettext' is the right way to fix many issues we're having with localization.
CiviCamp Montréal, 29 septembre 2017 | Co-founder / consultant / turn-key CiviCRM hosting for Quebec/Canada @ SymbioTIC.coop

Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Discussion »
  • Internationalization and Localization (Moderators: Michał Mach, mathieu) »
  • Problem with frontend search fields with Polish characters (Joomla Civicrm)

This forum was archived on 2017-11-26.