WSOD Using Importer

[ColborneMMX]

I'm at my wits' end, and I'm hoping someone can help.

I have a brand new, out of the box CiviCRM 4.4.5 install hosted in Wordpress 3.9. I'm getting ready to import contacts from different data sources in. So far, I've been able to successfully import and map custom fields against a data set that's roughly 250 columns wide but I have another data set that's roughly 350 columns wide that generates a white screen even if I reduce the import down to a single row under the column headers. The URL it bombs out at is:

.../wp-admin/admin.php?page=CiviCRM&q=civicrm/import/contact&_qf_MapField_display=true&qfKey=db9ab82800c2719f416e6298fdcda0b4_2339

This happens near-immediately if I just make it a two row source (column headers + 1 row of data), so I don't think I'm timing out on anything.

I confirmed that the data source, a CSV file, is UTF-8 encoded. I've also tried changing the line breaks (file is just CR; tried CR+LF, no effect). I'm able to import the CSV using PHPMyAdmin into a table in MySQL, so I know the server is able to read the data without issue. I've also tried clearing out quotation marks from the column headers; this had no effect. I've also tried importing the data using a SQL query against the table I imported via PHPMyAdmin (e.g. "SELECT * FROM mycivi_database.importedTable;") - this also gives me the White Screen of Death.

The really frustrating part is that, even if I enable CiviCRM debugging, even if I enable PHP debugging, even if I enable MySQL debugging, even if I enable Apache LogError debug, I'm not getting any errors, warnings, messages, or anything else indicating what's going on.

The good news is I have root on the box it's being hosted on (CentOS 6.5, Apache 2.2, PHP 5.3.26, MySQL 5.5.36 Community), so I've been able to play with some configuration options. Here's what I have so far:

PHP.ini Settings in VirtualHost Directory
upload_max_filesize=256M
post_max_size=256M
display_errors=On
display_startup_errors=On
html_errors=On
log_errors=On
error_reporting=E_ALL
memory_limit=2048M
max_execution_time=30000
max_input_vars=4000
realpath_cache_size=2M

Current MySQL Settings

max_allowed_packet	1073741824
max_binlog_cache_size	18446744073709547520
max_binlog_size	1073741824
max_binlog_stmt_cache_size	18446744073709547520
max_connect_errors	10
max_connections	151
max_delayed_threads	20
max_error_count	64
max_heap_table_size	33554432
max_insert_delayed_threads	20
max_join_size	1000000000
max_length_for_sort_data	1024
max_long_data_size	1073741824
max_prepared_stmt_count	16382
max_relay_log_size	0
max_seeks_for_key	18446744073709551615
max_sort_length	1024
max_sp_recursion_depth	0
max_tmp_tables	32
max_user_connections	50
max_write_lock_count	18446744073709551615

Apache, meanwhile, is set with a Timeout of 300.

Anyway, I'm running out of possible reasons why the Import page might be bombing out. One idea I had was that maybe the query was too long, but when I enabled Slow Query logging in MySQL and turned max_packet_size to the maximum (1GB), the longest query (UPDATE civicrm_cache SET group_name = 'CiviCRM Session', path='CiviCRM_CRM_Contact_Import_randomtext, data=Wall of text!; ) completed successfully and there were additional CiviCRM queries completed after it.

So, what's going on? Is there a way for me to get this stuff in without splitting the file in half lengthwise and hoping deduplication fills in the remaining columns in the "duplicate" entries? Do I need to spend the next couple of days reading up on the API (I'd rather not if it can be helped)? Is there some server setting I should be trying somewhere that I'm not aware of?

[JonGold]

Hey there,

There should be a ConfigAndLog folder somewhere in your webroot's hierarchy - I'm actually not 100% sure how that directory is set, but it's usually in the same folder as the templates_c folder, which is set in civicrm.settings.php.  The log files in there will have the PHP errors you're looking for - as will Apache's error.log (though this is obviously dependent on your Apache config, and I don't have a CentOS box handy - try /var/log/httpd?).

350 fields sounds pretty crazy!  That definitely sounds like a job for the API, or at least the CLI-based CSV import - which of course involves breaking up the data.  That said, the import deduplication rules are pretty good if you have an external identifier field to relate all the fields together - I've definitely done it, though I don't envy you that 350 field schema!

[ColborneMMX]

According to civicrm.settings.php:

Code: [Select]

* EXAMPLE - WordPress Installations:
 * If the path to the WordPress home directory is /var/www/htdocs/wordpress
 * the $civicrm_root setting would be:
 *      $civicrm_root = '/var/www/htdocs/wordpress/wp-content/plugins/civicrm/civicrm/';
 *
 * the CIVICRM_TEMPLATE_COMPILEDIR would be:
 *      define( 'CIVICRM_TEMPLATE_COMPILEDIR', '/var/www/htdocs/wordpress/wp-content/plugins/files/civicrm/templates_c/' );
The good news is the ConfigAndLog directory is indeed in the same area. The bad news is, there's nothing in there, even after I enabled debug and backtracing just now and re-ran the import that keeps bombing out on me.

As for Apache, this particular server has CPanel and WHM installed, which is configured to log to /usr/local/apache/logs - I'm not seeing any errors, even with LogLevel=debug. I do see some debug-related messages so I know nothing's overriding that setting anywhere, but if I have a tail -f going on error_log while I'm performing the import, nothing new shows up on screen.

At this point, I think I'm being more stubborn than anything - I mean, something has to be causing it to error out, and I should be able to find out what somewhere, right? So where's the bottleneck?

In the meantime, I'm definitely going to have to break this data down into thinner chunks. Whether I use the CLI or the Import screen, that's going to be a necessity either way, it seems. It's either that or roll my own VM and try to SQL dump everything in or something.

[ColborneMMX]

Just for kicks, I tried to import a single record of the data into the CiviCRM Wordpress Demo site to see if it behaved any differently. The good news is I was indeed able to get to the field mapping screen, which is encouraging. That at least tells me I'm not running into some sort of hardwired limit in CiviCRM with the column import.

[Donald Lobo]

a wsod in most cases means:

a. PHP/Apache is aborting

c. PHP is running into a fatal error (memory limits etc)

with 350 fields, i suspect we are hitting the POST limit maybe of apache?

lobo

[linkx]

I am seeing the exact same behavior. My data is 2 columns and 400 rows.

I have determined that my WSOD issue is not in the data, because I can import the full data set no problem.

The problem occurs when I try to import into a custom field group that uses "Addresses" in the "Used for" setting of the custom field group.

If I map my CSV column to fields in a custom field group configured for "Addresses", I get a WSOD immediately when I start the import.

If I map the same CSV column to a field in a different custom field group, one that has "Used for" configured to "Organization," then the import succeeds.

I need the "Addresses" setting because I need the custom data to be available to my Mailing Label format configuration found under /civicrm/admin/setting/preferences/address and I don't know of any other way to use tokens from custom data fields in mailing labels.

[JonGold]

Hi Linkx,

I'm pretty sure I recall this being a problem that was fixed somewhere in the 4.3 series.  You don't have your CiviCRM version listed on your forum profile - could you please tell us what version you're usng?

[linkx]

Thanks @JonGold I was on 4.2 so I ran the upgrade to 4.4 and it works now. Thanks again for your reply.