Cleaning House

[Denver Dave]

My space usage was going through the roof for the domain containing CiviCRM, so I finally decided to figure out where it is going.

Currently using 184 MB of space which does not include the MySQL Databases.

The initial CiviCRM upload was 36 MB which is a mystery how any application could have that much script, but there is 36 MB.

30 MB stored in Emails for bounces and replies that I have not figured out how to process yet.

4 MB for phplist install

1 MB for old email list system with the data (about 2/3 of current CiviCRM contacts)

This leaves 113 MB un accounted for, so I started looking.  Eventually found 72 MB in /drupal/files/civicrm/upload.  Seems that every import upload that we have done was stored there - so deleted them.

Maybe everyone else knows about the upload directory, but thought I'd pass this on.

[Matt2000]

The initial CiviCRM upload was 36 MB which is a mystery how any application could have that much script, but there is 36 MB.

Because it includes a ton of third party packages for templating, forms, javascript, and DB abstraction, instead of using the tools provided by the CMS.

With what host does 184MB constitute 'through the roof' space usage? Sounds pretty tame to me. I have clients who use a Gigabyte of space for email storage alone.

[Denver Dave]

I'm not sure the term "through the roof" isn't accurate.  I do have the current space and more, but let's look  at the changes:

Number of contact records increased 50% from 2,000 to 3,000 over the last 5 months

Space requirements without databases went up from 40 MB to 186 MB or 450 %, but after deleting upload directory, now only 40 MB to 285%

phpmyadmin exports started out at 3 MB and are now 40 MB for an increase of 1,300 %  At some point, I may not be able to do a backup and I may already not be able to load the backup with phpmyadmin in under 30 seconds without a parsing script like "Big Dump".

There are extensive mysql log files of activities that may have to be cleaned up.   On the other side, a complete dump of the contact records with custom fields is only 650 KB.

[Denver Dave]

I'm checking back a year later and the accumulation of records continues to be more of a problem.  We now have over 3/4 of a million records in the CiviCRM database, but only 5,300 contact records.  Some of the issue has to do with the CiviMail activity where we tend to accumulate mailing events from many people.  However, the lack of a "house cleaning" or pruning utility is a serious consideration.

We are currently running v 1.9 and one benefit to a export and reload rather than upgrade to new version may be that we can clean house - I'll discuss this with upgrade strategies.

Please let me know if you have an interest in this area.

Thanks. Dave

[Donald Lobo]

We definitely need an archival process to either delete old entries in the event table and/or move them to an "archive" type table in the same/different DB. We probably need to do the same for activities

Ideally someone in the community would step up and contribute the code and/or sponsor the feature.

lobo