apache segmentation faults

[SteveM]

Hey all,

I've had plenty of experience installing and using CiviCRM, but this one is new to me. When I enable CiviCRM in a new Drupal 5.2 install, any attempts to use the site fail because apache segfaults. When I disable CiviCRM, it runs fine. Has anyone else seen this sort of thing?

CiviCRM 1.8.11248 Drupal PHP5
PHP 5.1.6 (cli) (built: May  9 2007 11:28:53)
Apache 2.2.2-1.3
MySQL 5.0.27-1.fc5

We do not have any sort of PHP accelerator enabled. This is one of a few Xen instances that run on the same physical box, with identical versions of the core software ... and CiviCRM runs fine on the other instances.

A sample (unhelpful) httpd log:

[Thu Sep 06 13:26:51 2007] [notice] Apache/2.2.2 (Fedora) configured -- resuming normal operations
[Thu Sep 06 13:26:56 2007] [notice] child pid 31038 exit signal Segmentation fault (11)
[Thu Sep 06 14:31:47 2007] [notice] child pid 31042 exit signal Segmentation fault (11)

[SteveM]

I should note that the other Xen instances are using slightly older versions of CiviCRM 1.8 stable. Did something odd happen in this latest release?

[Donald Lobo]

apache/php seg faults are the most difficult to debug (IMO). Its a matter of going thru the code and figuring out where it is dying. In all likelihood this is because of some recursion in civicrm that blows the process up. We have not seen / encountered this issue and hence its hard to tell u where to start
 
I would start with CRM_Core_Invoke::main( ) and trace it from there, putting exit statements as u go along till u find out where it dies. You need to be a bit methodical to track this down. Not having a good understanding of the code base makes it more difficult (I suspect)

If you get on IRC, we can chat and help u get going on this

lobo

[Chris Burgess]

I'm seeing a similar thing. Will try and trace it from CRM_Core_Invoke::main() and see what I can find.

For us, the issue seems isolated to specific contacts; I can change the email on my own contact without problems, but when changing another contact it will die like this.

[Chris Burgess]

I tracked the issue (for us) down to the UTF-8 detection in CRM_Utils_String::isUTF8()

This regexp throws a segfault parsing a number of UTF-8 documents. The original culprit was a resultset from Google Maps (sample available via this link) but on further testing I found that the same function threw a segfault on various UTF-8 docs from around the web.

Looks like this can be traced back to a bug in the PHP preg library which is fixed in PHP5.2.5. So if you're running an earlier version of PHP than the one released last month, and seeing intermittent segfaults, this may be your issue.

I made a little sample suite which demonstrates an alternative approach (or two) and tests the UTF-8 detection against some sample documents. Forum won't let me post .tgz, so it's here: http://bushi.net.nz/tmp/civicrm-utf8test.tgz

It tests the UTF-8 goodness of the following documents. Note that I'm not very knowledgeable about UTF-8 generally, so I'm trying to get a reasonable sample set. I was surprised by how many of these upset the existing function.

Test Source	Comment	Special Characters?	New code result	Old code result
00 My original Google Maps culprit	claims to be UTF-8	no special chars	validates	segfaults
01 A Unicode test page	should be UTF-8	some special chars	validates	segfaults
02 Another UTF-8 test page	should be UTF-8	some special chars	invalid	invalid
03 The Wikipedia page on UTF-8	should be UTF-8	some special chars	valid	segfaults
04 Front page of UTF-8.com	should be UTF-8	no special UTF-8 chars	valid	segfaults
05 Front page of ASCIItable.com	should be ASCII	no special chars	valid	segfaults
06 Front page of Microsoft Korea	should be UTF-8	some special chars	valid	segfaults
08 A gzipped tarball of the above files	should not be UTF-8	some special chars	invalid	invalid
09 Some rubbish from /dev/urandom on Linux	should not be UTF-8	some special chars	invalid	invalid
10 An empty file	nothing to see here, move along		valid	valid

I tested this on PHP4 (4.4.0 and 4.4.7) and PHP5 (5.2.0, 5.2.3, 5.2.4 and 5.2.5). Only PHP5.2.5 ran the code in CiviCRM's current CRM_Utils_String::isUTF8() function consistently without segfault.

Here the tests I ran (including a couple of alternative approaches, one of which MAY correctly determine UTF-8ness without segfault on earlier versions of PHP - but I'm not expert enough in character encoding to claim that it does anything other than look pretty. Which I don't, cos it's ugly.)

Hope this helps someone else spend a few hours looking for that sneaky segfault that stole half my Monday

[Chris Burgess]

Update:

PHP5.2.5 on windows didn't segfault, but when I later tested it on the same distribution I found it still segfaulted. So I swapped the function in CRM_Utils_String::isUtf8() to use one of the custom versions instead, and it now saves contact details without error.

Another fix for this intermittent behaviour (suggested by Jedi) is to disable GeoCoding altogether ...

[Chris Burgess]

Opened a ticket against CRM_Utils_String::IsUtf() - CRM-2467. Lobo has posted a fix which prevents the Google geocoder from calling that function, thus avoiding the segfault.

However, it should be noted that other UTF-8 strings could still cause similar segfaults if handed to CRM_Utils_String::IsUtf8(). If it's possible to check UTF08 validity via some other method, that may be a more stable approach for CiviCRM to take until CiviCRM's users have a chance to migrate to a fixed PHP version.

If you'd like to contribute, please try this sample code on your PHP and document your results here?

Below are the different platforms I tested the CRM_Utils_String::IsUtf8() segfaulting with the example Google Maps result string above. We should be able to pass this upstream to PHP developers and get the segfault resolved - I'll post a ticket here when this is done.

 * PHP4.3.10 on SuSE did not segfault with the existing regexes.
 * PHP4.4.4-8+etch4 on Debian segfaulted.
 * PHP 5.2.3-1ubuntu6.1 on Debian segfaulted.
 * PHP 5.2.3 on OSX segfaulted.
 * PHP 5.2.4 on OSX segfaulted.
 * PHP 5.2.5-0.dotdeb.1 on Ubuntu segfaulted.
 * PHP 5.2.5 on Windows (binaries from PHP.net) did not segfault.

[Chris Burgess]

Looks like the issue is dependent on the length of the input string - the regex works (testing only on a single machine, 128MB ram) with strings under 4000 chars, and dies with strings above 5000 chars.

When I get a chance, I'll submit an alternate version of IsUtf8() which handles this properly (eg by splitting the string into nice chunks for checking, and/or using alternative methods where they are more dependable).