CiviCRM Community Forums (archive)

Hey there,

We recently experienced a few problems with timeouts and servers crashing and much wailing and gnashing of teeth.  I thought that this reply to the client might be useful for others as well.  Kudos to Eileen for starting the MIH on performance - I think that is a sensible solution and hope it catches on...

"The export process can become quite memory intensive, especially when aggregating information from different parts of the system (greetings, addresses, tags, groups, etc.).  The extra time taken to compute the export can increase dramatically with each extra column that you add, and at some point, this ends up crashing the system. 

One way of getting around this is to split the export up by columns into two different exports: one export contains the first half of the columns (September 2012 Appeal - part 1), and the other contains the second half (September 2012 Appeal - part 2).

FYI, the second of these exports took by far the longest.  Most likely this is because the data that you are exporting includes information on groups, tags, etc, so if you do encounter problems again (which is likely to happen as your dataset gets bigger and more complex (more groups, more records, etc.), try splitting up the second part of the export.

I've attached the exports created with these to this email.  Both exports contain and are sorted by the internal contact ID show you should be able to copy and paste these together quite easily in a spreadsheet.

Two longer term and more appropriate solutions to this problem are

1) put the site on a higher powered server (though as you scale more and more, this becomes impractical / prohibitive)
2) improve the performance of the code that runs these queries. see http://civicrm.org/civicrm/contribute/transact?reset=1&id=44 and http://civicrm.org/blogs/eileen/making-it-happen-bit-faster-performance-and-your-civi-site.
"