File upload size

[ewanmclean]

Hello

How do I increase the allowed CSV file import upload size limit? It is set by default to 2 MB. Please note this is not a PHP limit, it's in Civi somewhere. My PHP limit is 10MB as confirmed by phpinfo and drupal.

Thanks

[Eileen]

http://drupal.demo.civicrm.org/civicrm/admin/setting/misc?reset=1

[ewanmclean]

I thought this was the jackpot but changing the value here hasn't changed it on the import screen and its still rejecting larger uploads.

[ewanmclean]

I've upgraded to 4.4 today with the result that the import limit has decreased to 1MB. I can trace this variable back to CRM/Core/Config/Variables.php but altering it doesn't seem to do anything. There is a //FIXME in CRM/Import/DataSource/CSV.php saying why limit it to 8MB if the config limit is different. I can play around with these lines and get the limit to change on screen, but then the upload fails.

[ewanmclean]

Fixed this by adding an extra line like $size=$size*8 which gave me an 8MB limit. Same fix was needed across various Datasource.php files for activity, contribution, etc.

[gharris]

YIKES!     Did you report this as a bug?  I'm seeing the same behavior on a WP based install.

[gharris]

AHA!  The math is wrong!  Instead of dividing by 1024 * 1024, it should be 1024 * 8.  Now to try and figure out how to get the bug reported and fixed.

[gharris]

Well, apparently I tested the wrong file.     That only changed what's posted on the form to the correct value.  It did not change the actual upload size.

[Interpolat]

It seems to me that there may be an influence from the environment at play here.  So I downloaded CiviCRM 4.4.4 and installed on Joomla 3.2.  The default limit was set at 8MB in the contacts import ui.  I looked at CiviCRM 4.4.4 on Joomla 2.5.8, and there, the default is 1MB.  Also, another difference I noted between the two setups, which may or may not be the cause of the difference in the default limit size, was the version of PHP installed.  The Joomla 3.2 was running on an environment with PHP 5.4.22.  The Joomla 2.5.8 install was on PHP 5.2.19.  Hope this sheds some light.  A patch would be real handy right now, until 4.5 comes out.

[Max Hunter]

I'm also seeing this behavior after an upgrade to 4.4 -- the max upload size in /civicrm/admin/setting/misc?reset=1 and php.ini are both 8MB, but all uploads larger than 1MB are failing.  Has anyone found a workaround?

[sonicthoughts]

I have tried changing several variables but can't even hack the code .  I see this logged as: CRM-14199 but tagged as trivial.  I know most guru's use other import tools but this seems to be more that trivial - no? 

[Jeremy Proffitt]

I don't know that I'd consider it trivial, but in the meantime, here are a couple of other methods:

- Create a DB table and import the CSV file to the table. You can then use the SQL import option in the importer.

- Create a custom script using the SPLFileObject to iterate through the CSV file and use the Civi API to import each record. This actually gives you a lot more flexibility than the importer does.

http://wiki.civicrm.org/confluence/display/CRMDOC/Using+the+API#UsingtheAPI-ExploringtheAPIanddocumentation
http://wiki.civicrm.org/confluence/display/CRMDOC/Bootstrap+Reference

[gharris]

Jeremy,

I understand that it gives a great deal more flexibility, but the CSV routines seem to take care of a lot of behind-the-scenes mapping and auto-generation that I didn't get the idea that the API does.  Did I miss something?     I'm thinking that the CSV routines seem to be much more comprehensive on the first pass and that the API would be better for filling in additional details.  Am I off?  Does the API provide everything that the CSV does and then some?

[sonicthoughts]

Thanks - This is very helpful but it is really not a valid workaround.  I'm surprise more people haven't flagged it.  Perhaps it is specific to my environment.

There are big differences with the API method since the standard import workflow includes many related fields (like custom fields, address) that require multiple passes.  Also, many users expect the standard workflow to work, without requiring shell access.  Also, the error message does not actually say that it is related to the file size so it requires pulling hair out a bit . 

It's no longer an issue for me, but If someone can point me to the right file, I'd be happy to help - hope it get attention in the issue queue for the sake of others....