CiviCRM Community Forums (archive)

I did a quick test to manage CiviCRM translations in the Drupal database, rather than to use the gettext ".mo" files.

Background: I have noticed that crm_translate() can be fairly slow on many types of servers, especially if PHP "APC" caching is not used. You can see this if you trace a request using Xdebug. The cause may be that the translation file is big (1 meg) and has to be parsed on every request. However, managing translations in a database, as Drupal does, sounds less efficient since every translation string causes an SQL query.

So I wrote a small module to test this:

* https://github.com/mlutfy/civicrm_l10n
* get a copy using git: "git clone https://github.com/mlutfy/civicrm_l10n.git"

Then read the installation instructions in the README.txt (in short: enable the module, but also import the .po files into Drupal).

Conclusions:

* Benchmarks I did on a server with APC gave me aproximately 25-30% performance increase (although I have to admit my benchmarks were not the most scientific, so take with a grain of salt). I suspect that without APC the difference will be even more important.
* It's pretty useful to also be able to translation strings from custom Smarty templates or extensions.

I'd really like to get some feedback from other people: how it performs, whether the implementation makes sense.

Mathieu (bgm on IRC)

Ok, so it turns out that
CRM/Core/I18n.php

isn't using gettext and the constructor seems to load all civicrm.mo indeed. I'm a bit confused as for how the others are imported.

Anyway, how do you benchmark? IMO going native if gettext is enabled is a big big win.

But we'd need to mesure.

@Xavier: Here's how to use native gettext:

* apply the patch (which is really just a quick test, do not use in production)

* rename your civicrm/l10n/fr_FR/messages.mo to civicrm/l10n/fr_FR/LC_MESSAGES/civicrm.mo

IMPORTANT: your operating system locale must support the language. On Debian/Ubuntu, "dpkg-reconfigure locales". Also check /etc/locales.gen.

EDIT: use patch2

Not sure about the memory usage, but I'd expect it to be similar or a bit better.

I opened an issue for the implementation in core:
http://issues.civicrm.org/jira/browse/CRM-9976

bgm, I can verify it's true.

Especially when we using memcache for drupal "cache" table. The string not in database, is always at memory, the performance will significant increase. A single page will query 100-200 string (on file or on database), now on the memory.

I'd really like to get some feedback from other people: how it performs, whether the implementation makes sense.

Mathieu (bgm on IRC)

Quick test in 20 mins.

Environment
Visit /civicrm/admin?reset=1 by pressing F5 reload page.
Using drupal devel module.
CiviCRM 3.4 + drupal 6
with other sites on the same box. (but sure the concurrent only running this one.)

First, with memcached localized string + APC - no significant between db cached string. But I believe it will have different when concurrent page view increase.
try 1:
Page execution time was 483.59 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=8.46 MB.
try 2:
Page execution time was 473.7 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=8.46 MB.
try 3:
Page execution time was 404.17 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=8.45 MB.


Second, database cached localize string + APC
try 1:
Page execution time was 413.77 ms.
Memory usage: Memory used at: devel_init()=0.35 MB, devel_shutdown()=8.27 MB.
try 2:
Page execution time was 403.42 ms.
Memory usage: Memory used at: devel_init()=0.35 MB, devel_shutdown()=8.27 MB.
try 3:
Page execution time was 589.74 ms.
Memory usage: Memory used at: devel_init()=0.35 MB, devel_shutdown()=8.27 MB.

Third, original civicrm mo file + APC + Drupal memcached
try 1:
Page execution time was 616.8 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=18.36 MB

try 2:
Page execution time was 666.08 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=18.36 MB.

try 3:
Page execution time was 753.31 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=18.36 MB.


Forth, Native gettext from CRM-9976 patch (to CiviCRM 3.3)
try 1:
Page execution time was 392.88 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=8.44 MB.

try 2:
Page execution time was 458.07 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=8.44 MB.

try 3:
Page execution time was 407.58 ms.
Memory usage: Memory used at: devel_init()=0.42 MB, devel_shutdown()=8.44 MB.

Another try with apache ab in 4 different setup. Don't think this number can prove anything, because every try have high login failure attempt.

Command
ab -c 10 -t 10 -C SESSf9bc9ce58ed8526f33fead6b70414073=f1b26c701d5b47f63c09da878c5e3e1f https://xxxxxxxxx/civicrm/admin?reset=1

1. translation cached into memcache( drupal memcache enable)
=====================================
Document Path:          /civicrm/admin?reset=1
Document Length:        100265 bytes

Concurrency Level:      10
Time taken for tests:   10.286 seconds
Complete requests:      91
Failed requests:        36
   (Connect: 0, Receive: 0, Length: 36, Exceptions: 0)
Write errors:           0
Total transferred:      9223351 bytes
HTML transferred:       9189679 bytes
Requests per second:    8.85 [#/sec] (mean)
Time per request:       1130.363 [ms] (mean)
Time per request:       113.036 [ms] (mean, across all concurrent requests)
Transfer rate:          875.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       19   46  37.3     33     185
Processing:   858 1012  74.5   1005    1189
Waiting:      813  967  71.6    963    1148
Total:        881 1058  91.9   1054    1324


2. translation cached into drupal db (drupal memcache disable)
=====================================
Document Path:          /civicrm/admin?reset=1
Document Length:        100265 bytes

Concurrency Level:      10
Time taken for tests:   10.026 seconds
Complete requests:      79
Failed requests:        67
   (Connect: 0, Receive: 0, Length: 67, Exceptions: 0)
Write errors:           0
Total transferred:      8040389 bytes
HTML transferred:       8011109 bytes
Requests per second:    7.88 [#/sec] (mean)
Time per request:       1269.152 [ms] (mean)
Time per request:       126.915 [ms] (mean, across all concurrent requests)
Transfer rate:          783.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       22   62  41.7     47     207
Processing:   824 1154 103.1   1143    1407
Waiting:      804 1093  98.2   1083    1341
Total:        889 1216 107.7   1217    1441



3. original civicrm(drupal memcach enabled)
=====================================
Concurrency Level:      10
Time taken for tests:   10.035 seconds
Complete requests:      46
Failed requests:        3
   (Connect: 0, Receive: 0, Length: 3, Exceptions: 0)
Write errors:           0
Total transferred:      4629115 bytes
HTML transferred:       4612279 bytes
Requests per second:    4.58 [#/sec] (mean)
Time per request:       2181.618 [ms] (mean)
Time per request:       218.162 [ms] (mean, across all concurrent requests)
Transfer rate:          450.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       18   51  40.5     32     168
Processing:  1476 1902 207.7   1902    2417
Waiting:     1419 1840 201.6   1842    2324
Total:       1506 1954 210.1   1945    2486



4. native gettext(drupal memcache enabled)
======================================
Document Path:          /civicrm/admin?reset=1
Document Length:        100265 bytes

Concurrency Level:      10
Time taken for tests:   10.399 seconds
Complete requests:      92
Failed requests:        35
   (Connect: 0, Receive: 0, Length: 35, Exceptions: 0)
Write errors:           0
Total transferred:      9348551 bytes
HTML transferred:       9314513 bytes
Requests per second:    8.85 [#/sec] (mean)
Time per request:       1130.302 [ms] (mean)
Time per request:       113.030 [ms] (mean, across all concurrent requests)
Transfer rate:          877.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       18   43  32.7     32     161
Processing:   691 1009 100.8   1001    1337
Waiting:      644  965  97.7    959    1304
Total:        738 1052 103.7   1048    1380

In our case the combination of APC and Native text shows a impressive results.

Thanks Mathieu!!

**********************************************************************************************
ENVIROMENT
**********************************************************************************************
CiviCRM 4.1.2 + Drupal 7.14 + Multilanguage (Spanish Castilian)
Debian 6 + Apache + MySQL 5.1.61-0+squeeze1 + PHP    5.3.3-7+squeeze8
RAM 1.5 GB

**********************************************************************************************
1.- Original civicrm.mo file + APC (Drupal memcache DISABLED)
**********************************************************************************************
ab -c 10 -t 10 -C SESSb07ce7c584.... site.org/index.php?q=civicrm/admin?reset=1

Document Path:          /index.php?q=civicrm/admin?reset=1
Document Length:        9497 bytes

Concurrency Level:      10
Time taken for tests:   11.367 seconds
Complete requests:      21
Failed requests:        0
Write errors:           0
Non-2xx responses:      21
Total transferred:      208992 bytes
HTML transferred:       199437 bytes
Requests per second:    1.85 [#/sec] (mean)
Time per request:       5413.092 [ms] (mean)
Time per request:       541.309 [ms] (mean, across all concurrent requests)
Transfer rate:          17.95 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:  3599 4052 294.9   4103    4674
Waiting:     3532 3972 291.2   4033    4573
Total:       3599 4053 295.0   4103    4675

Percentage of the requests served within a certain time (ms)
  50%   4016
  66%   4208
  75%   4246
  80%   4247
  90%   4391
  95%   4487
  98%   4675
  99%   4675
 100%   4675 (longest request)
 
**********************************************************************************************
2.- Native gettext + APC 3.1.9 (Drupal memcache DISABLED)
**********************************************************************************************
ab -c 10 -t 10 -C SESSb07ce7c584.... site.org/index.php?q=civicrm/admin?reset=1

Document Path:          /index.php?q=civicrm/admin?reset=1
Document Length:        9497 bytes

Concurrency Level:      10
Time taken for tests:   10.101 seconds
Complete requests:      172
Failed requests:        0
Write errors:           0
Non-2xx responses:      173
Total transferred:      1721696 bytes
HTML transferred:       1642981 bytes
Requests per second:    17.03 [#/sec] (mean)
Time per request:       587.242 [ms] (mean)
Time per request:       58.724 [ms] (mean, across all concurrent requests)
Transfer rate:          166.46 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:   413  571 101.3    547     922
Waiting:      389  537 103.2    508     879
Total:        413  571 101.4    547     922

Percentage of the requests served within a certain time (ms)
  50%    547
  66%    566
  75%    581
  80%    594
  90%    679
  95%    851
  98%    882
  99%    890
 100%    922 (longest request)
 

**********************************************************************************************
3.- Native gettext (APC 3.1.9 DISABLED) (Drupal memcache DISABLED)
**********************************************************************************************
ab -c 10 -t 10 -C SESSb07ce7c584.... site.org/index.php?q=civicrm/admin?reset=1
Document Path:          /index.php?q=civicrm/admin?reset=1
Document Length:        9497 bytes

Concurrency Level:      10
Time taken for tests:   19.636 seconds
Complete requests:      30
Failed requests:        0
Write errors:           0
Non-2xx responses:      39
Total transferred:      388128 bytes
HTML transferred:       370383 bytes
Requests per second:    1.53 [#/sec] (mean)
Time per request:       6545.394 [ms] (mean)
Time per request:       654.539 [ms] (mean, across all concurrent requests)
Transfer rate:          19.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.3      0       3
Processing:  1050 3198 2202.7   2370   11142
Waiting:      924 3081 2206.0   2242   10991
Total:       1050 3199 2202.2   2374   11142

Percentage of the requests served within a certain time (ms)
  50%   2374
  66%   4579
  75%   4667
  80%   4734
  90%   6123
  95%   6212
  98%  11142
  99%  11142
 100%  11142 (longest request)