CiviCRM Community Forums (archive)

*

News:

Have a question about CiviCRM?
Get it answered quickly at the new
CiviCRM Stack Exchange Q+A site
This forum was archived on 25 November 2017. Learn more.
How to get involved.
What to do if you think you've found a bug.



  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Removing old mail data ( e.g civicrm_mailing_event_queue )
Pages: [1]

Author Topic: Removing old mail data ( e.g civicrm_mailing_event_queue )  (Read 19067 times)

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Removing old mail data ( e.g civicrm_mailing_event_queue )
March 19, 2013, 04:13:46 pm
We've talked about culling old CiviMail data before but never pinned it down to actual impacts.

It seems to me that

Code: [Select]
DELETE meb FROM civicrm_mailing_event_bounce meb
LEFT JOIN civicrm_mailing_job mj
 mj.id = meb.job
 WHERE mj.end_date < '2011-01-01';
Will delete all delivered, clicked through, bounce, forward, confirm, reply, opened information for the report but leave activities untouched

I did a screen dump before & after running
Code: [Select]
DELETE q FROM civicrm_mailing_event_queue q
LEFT JOIN civicrm_mailing_job j ON j.id = q.job_id
WHERE mailing_id = 895;
« Last Edit: July 02, 2013, 10:34:03 pm by Eileen »
Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

Eileen

  • Forum Godess / God
  • I’m (like) Lobo ;)
  • *****
  • Posts: 4195
  • Karma: 218
    • Fuzion
Re: Removing old mail data
March 19, 2013, 04:52:12 pm
NB - I checked this on an older version of CiviCRM & the behaviour seems a bit wrong

Intended recipients on the front page shows the number currently in the group
Clicking through shows the number of people in the with entries in the event_bounce table

I couldn't see where the data from civicrm_mailing_recipients had an impact - but I think this may be irrelevant to a later version?

Make today the day you step up to support CiviCRM and all the amazing organisations that are using it to improve our world - http://civicrm.org/contribute

isatyam

  • I’m new here
  • *
  • Posts: 7
  • Karma: 0
  • CiviCRM version: 4.4.2
  • CMS version: Joomla 2.5.16
  • MySQL version: 5.5
  • PHP version: 5.3
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 03, 2013, 12:43:59 am
Hi,
I m having some problems with the old mail data. My client sends around 50k mails in a month. I think the mail data is filling up the disk space. I have no idea which files can be safely deleted in the /media/civicrm directory.
Please help.

Satyam 

jern

  • I post occasionally
  • **
  • Posts: 46
  • Karma: 0
    • SEN Magazine
  • CiviCRM version: 4.5.8
  • CMS version: Joomla 2.5.28
  • MySQL version: 5.5.40
  • PHP version: 5.4.33
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 13, 2013, 03:56:39 am
Hello. Anyone any ideas on this? I have the same problem as isatyam. My civicrm_mailing_event_queue table is now at 120MB and still growing every time we send out a mailing.

I assume that most of the 700,000 records are redundant, but I daren't start thinning the table without really understanding what effect it will have. Thanks for any suggestions.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 13, 2013, 02:04:02 pm

you can use eileens solution and also delete the activities if u want to reduce the size of that table

however, civimail does need a way to archive and clean up old mailings. Would be good if folks would either write an extension to do so (or if u dont have development skills, sponsor someone to do so)

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

robinhood

  • I post frequently
  • ***
  • Posts: 153
  • Karma: 6
  • CiviCRM version: 4.5.5
  • CMS version: Drupal 7.34
  • MySQL version: 5.1.56
  • PHP version: 5.3.5
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 13, 2013, 07:11:16 pm
Quote from: Eileen on March 19, 2013, 04:13:46 pm
It seems to me that

Code: [Select]
DELETE meb FROM civicrm_mailing_event_bounce meb
LEFT JOIN civicrm_mailing_job mj
 mj.id = meb.job
 WHERE mj.end_date < '2011-01-01';
Will delete all delivered, clicked through, bounce, forward, confirm, reply, opened information for the report but leave activities untouched

This doesn't work - there is no job field in civicrm_mailing_event_bounce.  Anyway the biggest tables in that group, in my install, are civicrm_mailing_event_queue and civicrm_mailing_event_delivered, at about a quarter million records each.  civicrm_mailing_recipients is also large at 183K records.  Any suggestions?

jern

  • I post occasionally
  • **
  • Posts: 46
  • Karma: 0
    • SEN Magazine
  • CiviCRM version: 4.5.8
  • CMS version: Joomla 2.5.28
  • MySQL version: 5.5.40
  • PHP version: 5.4.33
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 15, 2013, 12:39:57 pm
Sorry, I just found this old thread: http://forum.civicrm.org/index.php/topic,6079.30.html

So it looks like it's a well-known issue, and I suppose that everyone has found their own way of keeping bloated CiviMail tables under control. The archiving solution seems the best idea, so I will try that one day when I am feeling brave.

I don't have the skills to write an extension, but I would be happy to sponsor someone else to do it - it's a serious time-bomb for anyone doing lots of mailing. How would I go about sponsoring this?

Thanks.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 19, 2013, 01:30:50 pm

depending on how extensive the archival spec is, i suspect its a 20-50 hour project.

Our consulting rates are on the wiki:

http://wiki.civicrm.org/confluence/display/CRM/Consulting+Services+Agreement

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions

jern

  • I post occasionally
  • **
  • Posts: 46
  • Karma: 0
    • SEN Magazine
  • CiviCRM version: 4.5.8
  • CMS version: Joomla 2.5.28
  • MySQL version: 5.5.40
  • PHP version: 5.4.33
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 19, 2013, 01:51:31 pm
That's outside our budget, but happy to chip in for a share. I imagine that it's an issue which will affect a lot of installations sooner or later, so it seems important to find a solution. I only noticed the huge tables when our backups started to fail.

Donald Lobo

  • Administrator
  • I’m (like) Lobo ;)
  • *****
  • Posts: 15963
  • Karma: 470
    • CiviCRM site
  • CiviCRM version: 4.2+
  • CMS version: Drupal 7, Joomla 2.5+
  • MySQL version: 5.5.x
  • PHP version: 5.4.x
Re: Removing old mail data ( e.g civicrm_mailing_event_queue )
July 19, 2013, 10:02:58 pm

Would be good if you can spearhead an MIH for this for the 4.5 release cycle

http://wiki.civicrm.org/confluence/display/CRM/Make+it+Happen

Its a bit hard to convince people to fund things that are important for the project but does not impact them immediately

lobo
A new CiviCRM Q&A resource needs YOUR help to get started. Visit our StackExchange proposed site, sign up and vote on 5 questions
Pages: [1]
  • CiviCRM Community Forums (archive) »
  • Old sections (read-only, deprecated) »
  • Support »
  • Using CiviCRM »
  • Using CiviMail (Moderator: Piotr Szotkowski) »
  • Removing old mail data ( e.g civicrm_mailing_event_queue )

This forum was archived on 2017-11-26.