Loading activities tab with mailings is 100x slower after upgrade from 4.1 - 4.4

[Sean Madsen]

After upgrading from 4.1 to 4.4 I'm experiencing some new and peculiar slowness...

On a contact's record, loading the activities tab takes a very long time. For example, on a random contact I just tried, loading the initial page with the summary tab took less than 1 second (timing with FireBug). Then loading the activities tab (showing 25 out of 38 activities) took 109 seconds. Waiting almost 2 minutes to see the activities info is problem for my staff.

The request that is taking so long is a POST to /civicrm/ajax/contactactivity?snippet=4&context=activity&cid=40675

Same behavior on all contact records that I've tried. I don't have any errors in my Civi error log. What else can I do to troubleshoot this issue?

[Sean Madsen]

Still trying to figure this out. I've narrowed it down slightly though. I've noticed many records which do not exhibit this slowness, and noticed a pattern: contacts who have not received any mailings through CiviMail consistently have very speedy activity tabs, even when they have lots of activities. But contacts who have bulk mail activities consistently exhibit this slowness. Any the slowness seems to be, so some degree, proportional to the quantity of bulk mailing activities -- the more mailings the contact has received, the slower the activity tab is to load. Some contacts who have received lots of mailings are so slow to load that the request times out.

I've also monitored my server during the loading of a slow activity tab. During the first half of the time spent waiting, mysqld is the most intensive CPU process. Then the 2nd half is apache. I'm grasping to ways to troubleshoot this issue. Any recommendations?

[Donald Lobo]

There is a patch for this in 4.4.4

its a bit of a hack for now, but mailing activities are suppressed (since they contribute to the load time signifcantly)

lobo

[Fabian_SYSTOPIA]

This problem could indeed be linked to the number of contacts involved with the activities that are displayed (we suspect, all contacts involved in all activities are loaded in the background).

In general, there are a couple of issues connected to bulk activities - please have a look at this discussion: http://forum.civicrm.org/index.php/topic,31486.0.html.

At the moment we are trying to work out a way to properly address those - no displaying bulk activities is not really a feasible long term solution. Maybe we could also find a way to work at this together...

Best Regards
Fabian

[mielert]

For me it is really important to see the bulk activities.

Example:
- Customer gives us a call: he doesn't receive our newsletters anymore.
- With the changes made by @kurund I can't help because I can't see whether newsletters have been sent or not.

So I've downgraded CRM/Activity/BAO/Activity.php. Yes it is extremely slow, but we need the bulk data.

Cu,
Fritz

[joanne]

I agree with mielert, being able to confirm that last week's newsletter was sent to that person is very important for us. 

But it is important for the most recent newsletter and perhaps for newsletters going back for three to six months, for newsletters sent earlier than that we really don't care.

It is probably a clumsy and code intensive suggestion, but a configurable tick box when creating a bulk email to "delete mail activity for targets contacts after X months" where X is a system setting would work for us. People could always go to the archived mail report if they absolutely had to know if Mary Jane was sent the August 2012 newsletter.

My colleague has begun deleting those activities directly in the DB to try to get the activities tab to load in a reasonable time and we haven't noticed any adverse effects so far. 

[Sean Madsen]

Just upgraded to 4.4.4 and and my staff are happier. The slowness is no longer a problem. Yes, the mailings activities no longer show up, but fortunately that's totally fine for us.

[joanne]

For anyone finding this thread, in 4.4 (and possibly 4.3 - we are still on 4.2) at Administer>CiviMail>CiviMail Component Settings there is an tick box " Enable CiviMail to create activities on delivery".

If it is ticked then an activity is created for each contact for each bulk email and you get a horrible delay when loading activities for a contact. (4.2 standard behaviour)

If it is not ticked then a new tab called Mailings is created for each contact and the bulk emails they have received are stored there.  It works retrospectively too in that unticking the box means all previous mailings are shown on that Mailings tab.

This means that the patch gives you a faster loading activity tab and you can still see if someone has been sent a bulk email.

Note that the implementation in 4.4.4 has a bug - the activity count on the tab will include the bulk emails.  The fix will be in 4.4.5. if you need to patch before then it is https://github.com/civicrm/civicrm-core/commit/eacfbd68058161e4cda8d35a86c8613770860ad2#diff-0

[mudnik]

Hmm..Nice solution Joanne.

[joanne]

Thanks for the sentiment, but it belongs to Kurund who supplied the answer in a comment on Github. 

All I did was add it to this topic so people could find it without leaving the forum. I should have added info on the count bug which I will do now via an edit to that post.

[Fabian_SYSTOPIA]

Dear all,

we are still working on a more comprehensive solution for the whole problem regarding activities with a large number of contacts.

However, since we are not happy with not seeing Newsletter etc. in the activities tab we made a quick and dirty patch. When in the contact's activities tab it prevents laoding the contacts in the "with" column.

As a result, you will be able to see all activities, including newsletters and other mass activities but obviously you will not be able to see the contacts involved in the "with-column" (you will see n/a instead). This applies only to the contact activities tab, not other search results.

As described above, this is more of a quick and dirty solution - thought I would share the patch anyway, just in case anybody is interested. We tested it on 4.3.8 but it should also work on 4.4.4.

Best Regards
Fabian

[Fabian_SYSTOPIA]

Hi,

we created another hot fix which is still far from perfect but better than the one before. Here's what it does:

• If only one contact's name would be shown in the with-column a link to that contact will be shown instead
• If more than one contact's name would be shown in the with-column the count of contacts is shown instead
• If (for any unlikely reason) no contact would be shown in the with-column n/a is shown instead

We tested this one on 4.3.8 and 4.4.3 - feel free to use this patch and let us know if you encounter any problems

[Fabian_SYSTOPIA]

And another, slightly improved patch - also tested on 4.4.4 & 4.4.5.

Regards
Fabian