CiviCRM Community Forums (archive)

I'm unable to get my installation to process 7,000 records in a reasonable time frame. I've observed all of the tips in the FAQ/Import post, e.g., bumped up my server's PHP memory limit, script execution time, etc. Only thing that I can think of is that my installation isn't tuned out of the box and can't process input quickly enough.

I am able to get through to the field mapping table. Upon clicking Continue, browser hangs waiting for a response from server. I have successfully imported 100 records into my installation and that took 2-3 minutes. That said, roughly put, 7,000 records imported into my installation would take 3-4 hours.

I've run 7,000 records containing around 40 fields through drupal.demo.civicrm.org. The demo installation performed the field checking and created the Import Error csv very quickly, in just a few minutes. I've also successfully imported the same records on a local development box in my office. Also only took a few minutes.

Firebug isn't showing anything in the console except the usual GET/POST messages. From what I can tell the file has successfully uploaded at /sites/default/files/civicrm/upload though it obviously hasn't been parsed and handed off to the CiviCRM tables. Any thoughts?

I'm on virtual dedicated/Virtuozzo box at media temple. Pretty decent as these things go. I've got 1.5 gigs of RAM, more than adequate throughput, and plenty of CPU. If it's a resource issue, would recommend moving to dedicated hardware? Seems like overkill to me but I'm open to any/all ideas.

Is there any way to console into Drupal and see where things are failing or whether the request is even hitting the script?

2.1.4 yes.

Erm...CPU load on the virtual box is around 75%?   

My local box is just a Mac mini running 10.5.6.



Guess I'll let it run overnight and see what happens.

Seems like this should be a pretty routine operation.

Only asking as it has been a problem in past for me - but I have found imports using Safari more reliable that with Firefox. I have no idea why. But it helps.

I would love to hear more about this issue, and resolutions.

We are on a dedicated server, and are running 2.0.2

I have been uploading data, started with about 25000 contributions with org and ind contacts. When I started uploading i could do 1000 contact records under a minute each upload, usually around 30 seconds... it seems once I hit the 30-40k contact records it has just started slowing down incredibly.

When an 1000 record/24 column  upload started hitting 3+ minutes, I broke the file size in half to 500, but it was almost as slow at 2.5 minutes. I keep pluygging along and the 500 upload is now taking over 4 minutes, when just yesterday I could do 1000 in less than that. I have over 125000 records to upload so Im about half way, my fear is Im going to have to keep reducing the file size to keep from timing out.

Any thoughts?

Thanks for the help, thats good to know. We have quite a few more instances similar to this to do too.

John

We run virtual servers (using both xen and vservers) and have found disk through put to often be an issue. Given that importing into a database is disk i/o heavy, it might be something worth looking at.

While doing the import on a linux machine, you can run:

vmstat 1

There's a lot of numbers, but keeping track of the second column (number of blocked process due to input/output) could be a clue. You can also run top and pay attention to the percent of time your CPU is in the Wait state (it's also refers to disk i/o).

I raise this issue because even if you have a dedicated virtual server, that server is probably sharing the same disk with many other virtual servers. On the other hand, importing on your desktop computer, even if it's a slow CPU, might have dedicated access to your disk.

jamie