Author Topic: New mailing - recipients select box is s-l-o-w [FIXED] (Read 1462 times)

ken · March 13, 2015, 01:16:32 am

In 4.6.beta3 the Recipients select box seems to be trying to do too much.

We have hundreds of groups and thousands of mailings so having the browser build a select list with include groups, exclude groups, include mailings and exclude mailings takes a long time. My Firefox browser greys out for 15 seconds *multiple times* if I type a few characters into this box.

I don't regularly include or exclude mailings. So waiting while they are retrieved doesn't excite me. But if I were to use them I think I'd find this screen frustrating as I'd have to scroll past the "include groups" etc to get to them.

May I suggest separate fields?

Coleman Watts · March 13, 2015, 06:38:14 am

I have to agree. While I applaud the GSCO folks who invented this combo widget for their innovation, I think it's either too clever, or not clever enough, to surmount the usability problems it creates.

totten · March 15, 2015, 10:59:50 pm

Quote from: ken on March 13, 2015, 01:16:32 am

We have hundreds of groups and thousands of mailings so having the browser build a select list with include groups, exclude groups, include mailings and exclude mailings takes a long time. My Firefox browser greys out for 15 seconds *multiple times* if I type a few characters into this box.

Based on the code (but without testing), I'll agree that there's a scalability problem -- e.g. it prefetches the full list of mailings+groups rather than fetching the list incrementally as you search. (Put another way: the current implementation is optimized for "simple code" rather than "scalability".) IMHO, the most obvious fix is to reimplement the crmMailingRecipients directive using AJAX searches instead of prefetching.

To help give us some concrete reference points, could you describe some give more precise metrics, e.g.

Desktop/laptop/workstation specs - CPU model (or approx year+family), amount of RAM, OS version, browser version
Mailings - "Thousands" is a wide-range. Does that mean 1-2k? 2-5k? 5-10k? 10k+?
Groups - "Hundreds" => 100-200? 200-500? 500-1000?

Quote from: ken on March 13, 2015, 01:16:32 am

I don't regularly include or exclude mailings. So waiting while they are retrieved doesn't excite me. But if I were to use them I think I'd find this screen frustrating as I'd have to scroll past the "include groups" etc to get to them.

I agree that scrolling is frustrating. The v4.6.beta3 UX is primarily a reaction to the traditional (pre-4.5) UX - which produced a million scrollbars. (Well, eight scrollbars. But eight feels like a million.) The simplest solution to scrolling is to use a typing/searching UX -- which is exactly what v4.6.beta3 does. But it sounds like you're not getting the benefit of the typing/searching UX because of the performance issue.

Quote from: ken on March 13, 2015, 01:16:32 am

May I suggest separate fields?

Generally, a major goal has been to make the UI more amenable to customization. v4.6 goes fairly far (in that it uses components and remixes them with several different layouts) but doesn't go all the way (in that we haven't built an API or UX for defining custom layouts or swapping components). I would patchwelcome an option to use the four-field layout -- and possibly acquiesce to it as default for pre-existing sites -- but oppose four-field layout as a default for new sites.
Nitpick: If one does four fields... then... in the typical case where the user just wants to include a group, they would have to scroll past several things they don't care about.
Switching the UI/layout to four-fields doesn't mean we go back to the old implementation with the old performance dynamic - it means we do a new implementation (with new performance dynamic) which looks like the old one. There are several differences between the old implementation and the new implementation which could explain the performance issue -- one difference is #fields. Other candidates (off the top of my head): using Angular, integrating Angular+select2, the per-item visualizations, the prefetching policy, and the decision to prioritize "simple" over "fast". The odds are that the problem involves one or more of the other items; if we only change the #fields, then we're likely to repeat the problem.

Quote from: Coleman Watts on March 13, 2015, 06:38:14 am

I have to agree. While I applaud the GSCO folks who invented this combo widget for their innovation, I think it's either too clever, or not clever enough, to surmount the usability problems it creates.

The old four-field UI was no gem. I think we're at a stage where we need to be more precise about problems. This last comment seems overly vague. Ken raised some specific problems - which is great.

ken · March 16, 2015, 04:27:23 am

@totten,

1. Regarding your questions ...

Computer
Machine -   DELL Inspiron laptop, 2013
Operating system -    Ubuntu Linux 14.10
Browser version -    Firefox 36.0.1
Kernel and CPU -   Linux 3.16.0-31-generic on x86_64
Processor information -    Intel(R) Core(TM) i3-3217U CPU @ 1.80GHz, 4 cores
Real memory -   3.64 GB total
Virtual memory -    3.91 GB total
Local disk space -   396.99 GB total / 272.61 GB free / 124.38 GB used

Mailings - 4000 mailings

Groups - 500 enabled groups

2. Regarding the one field performing four functions (include/exclude group/mailing), I'm still not convinced of the merits of this. Let me try and explain my concerns ...

The 'grammar' of the 4 fields approach is "verb type instance" - eg "include group" (by choosing the box) and "newsletter" (by typing in the box and choosing the right item) to include the Newsletter group in the mailing
The grammar of the single field approach is "instance verb type" - eg "newsletter" (by choosing the box and typing in it) and "include group" (choosing the right item)
So in one sense both approaches are equally efficient - both include the steps 'choose box', 'type in the box' and 'select item'.
But in another sense the 4 field approach is more efficient. While in the "include group" box I can select additional groups to include without having each time to select whether to include/exclude a group/mailing.
I could also imagine someone under deadline pressure getting confused between include/exclude and group/mailing
Finally, groups and mailings are different beasts and it seems odd to put them together

Coleman Watts · March 16, 2015, 08:05:23 am

Quote

This last comment seems overly vague. Ken raised some specific problems - which is great.

Agreed, and apologies, my own thoughts were a bit jumbled and instead of detangling them all I just wanted to briefly give Ken some encouragement in hopes he would say more and it would help me clarify my own thinking. Both of your follow-up comments have helped so I think I can finally articulate myself

By "usability problems" I specifically mean 3 things:

The current implementation doesn't scale well for 2 reasons. 1 is the prefetching of all data into memory. This obviously has limits (with an equally obvious solution of using ajax).
The other reason is that combining 2 types of data into one makes it hard for the user to discover everything the field contains. For people who prefer scrolling, it would take a very long time to get to the end of a 500 group list (esp when every group is shown twice). And for people who type into search fields, the first couple of letters they type will probably turn up 10 groups, and if they type a complete group name then it will probably show just the one group they were searching for. So both types of users might spend years using CiviCRM without ever realizing that field contains anything but groups!
Regardless of what widget is used, I think we could do a little better at making the and/not logic apparent to the user. While the "live preview" of recipients is awesome (really), I still find myself a little confused by how the logic will work in the combo box, e.g. it's not completely obvious (to me anyway) that the order doesn't matter, and placing an "exclude group" before an "include group" will have the same result as the other way 'round because they will be processed in reverse order by the server. The four fields wasn't perfect either but at least the order is clear.

It's a complex problem with maybe no perfect solution, but while I was typing the above a compromise came to me which might strike the right balance:
I'm going to advocate for 2 ajax fields, one for include groups/mailings and the other for exclude groups/mailings. (mockup attached)

Here's my thinking:

As Tim said, using ajax is the obvious solution to the performance issue
2 fields is not going to overly burden the ui by taking up too much space
Breaking it apart by include/exclude will emphasize and clarify the and/not logic.
Excluding is something "advanced" users do, but even they do it rarely. Breaking the exclude options into another field gets them out of the way and prevents the type of user error that Ken is concerned about.
Showing each field only once simplifies the widget code, possibly to the point where we can re-use the existing EntityRef code for this (although I'm not sure how to show 2 types of entities in the same entityRef).
Ken I agree that putting groups and mailings in the same box is a little weird, but I think it would be less weird this way because at least now they are always serving the same purpose within their box.

So this is a compromise but I think it has something for everyone. I will say though that the "something" it has for us developers is "more work". From a developer's point of view, implementing 4 ajax boxes that look exactly like the 4.5 mailing UI would be a lot easier. Developing a custom entityRef that can hold 2 types of entities will take a day at least. Just straight-up reusing the existing entityref code for 4 separate fields would be quick, easy, and would meet every need except compactness.

totten · March 16, 2015, 08:16:27 pm

Quote from: ken on March 16, 2015, 04:27:23 am

So in one sense both approaches are equally efficient - both include the steps 'choose box', 'type in the box' and 'select item'.
But in another sense the 4 field approach is more efficient. While in the "include group" box I can select additional groups to include without having each time to select whether to include/exclude a group/mailing.
I could also imagine someone under deadline pressure getting confused between include/exclude and group/mailing
Finally, groups and mailings are different beasts and it seems odd to put them together

True - If you need to enter multiple includes and multiple excludes, then that is more efficient.

But I feel like this is optimizing for an edge-case. The typical case is to include one group. The next level up is to include multiple groups. Anything dealing with exclusions is advanced-user territory. There's a segment of the userbase for whom sending mail to one group is a challenge. (Unfortunately, I don't have stats - only anecdotal experience from my time running a Civi SaaS where mailing was an important selling-point. We did all kinds of workarounds to help people avoid that screen.) Philosophically, I think that the normal case should be trivial, and the advanced-case should be possible. Anything better than that is "icing on the cake".

At the risk of being obstinate/pedantic, let me try to provide some of the thinking behind the current 4.6 layout. Anyone who has ever used email has seen the same form with these input fields: "From:", "To:", "Subject:", and the implicit "body." It doesn't matter if you use Gmail, Outlook, Thunderbird, or Pine -- they all follow the same layout. Each deviation from this model is something that users have to study. The current "Recipients" layout tries to get as close as possible to the "To:" field. However, accomplishing this required two conceits:

1. More nouns. In standard email, you type nouns into the "To:" box -- those nouns are names or email-addresses. In CiviMail, the nouns are group-names or mailings (names/dates). If we had time, I'd happily add support for more nouns (e.g. contact names, email addresses, event names).
2. Exclusions. This conceit is particularly difficult because it's not applicable to standard, one-on-one email. I feel like the coloring/styling makes the behavior unambiguous. I personally expected we'd have to do an entirely custom widget to accomplish this (something like "drop-down-sections.png" or "popup.png", attached) - but was surprised by how much mileage we got out of select2.

The thing I like about the current layout is that it makes the normal case easy -- there's only one field to understand, and its first recommendation is the normal thing ("Include Group X"). If you have the wherewithall for advanced behavior, then you can dig into it.

Quote from: Coleman Watts on March 16, 2015, 08:05:23 am

It's a complex problem with maybe no perfect solution, but while I was typing the above a compromise came to me which might strike the right balance:
I'm going to advocate for 2 ajax fields, one for include groups/mailings and the other for exclude groups/mailings. (mockup attached)

I can kind of go along with the 2 ajax fields. The labels in the screenshot made me worry, but if we tweaked the labels (like in "two-auto-completes.png", attached), then I'd be more comfortable.

FWIW, I'm not sure that the 2 ajax fields really addresses Ken's point; anything that crams a list of 5,000 mailings into a select2 dropdown is basically unbrowseable (for mouse-people). As another possible compromise, let me suggest this: the recipient field has 1 ajax autocomplete. By typing in it, you can select groups to include. (This satisfies my concern about making the normal case trivial.) It doesn't have any support for mailings or exclusions. However, next to it, we put a button with a list-icon. Clicking the icon opens a more powerful dialog (like "popup.png") which makes all the include/exclude group/mailing nuances manifest and browsable.

totten · March 16, 2015, 08:25:07 pm

(Aside: To view the attached images at recommended size, I suggest left-clicking the image to expand and then right-clicking to view on a standalone page.)

ken · March 17, 2015, 06:51:26 pm

@totten, I like the look of your 2nd diagram "drop-down-sections.png"

defaults to the base use case "include group"
clearly presents other options (eg "exclude mailing")
avoids scrolling

On the other hand "popup.png" seems busy to me.

totten · March 17, 2015, 09:36:28 pm

Yeah, "drop-down-sections.png" is kind what I had in mind when it was handed over to me... but it was so much easier to use <optgroup>... will need to think about this a bit...

For a little geeking out... part of me really likes prefetching (because it has better latency for small/medium datasets), so I did some benchmarking to see if it was an obvious/correctable issue in our code. Interesting (but unscientific) results:

* Setup:
+ Single-core Windows XP VM with 1.5GB RAM and Firefox 31. The host CPU is a 2011 i7. Using VMWare Fusion.
+ ~2000 mailings and ~2000 groups (aka ~4000 unique records; aka ~8000 visible rows with include+exclude)
+ No other processes or tabs were visibly active
* Test Procedure
+ Enable or disable some feature we've added
+ Reload the page
+ Open the dropdown. Watch the clock.
+ Run a search. Watch the clock.
+ Backspace the search text. Watch the clock.
* select2 is almost (but not) fast enough to render 8000 rows - opening and searching each took about 4-6sec. Unfortunately, backspacing totally blew the numbers (12sec delay). These are the best numbers I got in this round of testing.
* Using <optgroups>, using formatted dates, and calculating the visual appearance didn't have any observable affect on performance.
* Rendering the icon and CSS-based coloring did affect performance. For open/search, the penalty seemed to be within the margin-of-error (extra 1-2sec penalty; MoE is high without proper measurements), but for backspace it was pretty bad (extra 5sec penalty).

Code: [Select]


                                                open     srch     bkspc
0. crmMailingRecipients: status quo             ~6sec    ~8sec    ~17sec
1. crmMailingRecipients: formatItem dummy       ~6sec    ~5sec    ~12sec
2. crmMailingRecipients: disable formatItem     ~6sec    ~6sec    ~13sec
3. select2 standalone                           ~6sec    ~4sec    ~12sec

totten · March 17, 2015, 11:53:40 pm

Did another experiment -- tried rendering the full list of ~4,000 data records using a basic Angular UI (with a few of the layouts we've seen/discussed). It generally did about 3x better than select2 -- ~1.5sec for filtering and ~3sec for backspacing. (It's not apples-to-apples yet, though, because it's missing several features we'd want.)

https://github.com/civicrm/civicrm-core/compare/4.6...totten:4.6-poc-grpml-list?expand=1

Coleman Watts · March 18, 2015, 11:03:33 am

Tim, I also like the tabbed interface, and with my experience of tricking out select2 to create the entityRef widget I'm pretty sure I could implement it.

One caveat: select2 v4 has been released (or is about to be) and it is a significant rewrite with a lot of apis changing. All of the crazy stuff we've done to customize it will need to be upgraded at some point

But I have heard the new version is more performant

And a question: loading all groups into clientside memory seems "okay" because the number of groups in a db doesn't grow linearly and even orgs with huge numbers of contacts will still have a reasonable number of groups. # of mailings on the other hand does grow linearly, with no limit. An org that puts out 2 mailings a week will have 520 of them after 5 years, 1040 after 10 years, and so on.
With possibly one mitigating factor - if the widget excludes "archived" mailings (does it?) and the org is in a regular habit of archiving mailings over 1 year old, then the number should always be manageable.
Alternately, the selector could hard-code a time or # limit (say, 2 years or 500 mailings) and simply not show the rest.
This would be the rock-bottom simplest solution to this problem, and IMO well worth considering, because:

With an ever-changing contact database, the recipient list for mailings over 2 years old really isn't worth re-using.
The change would fix Ken's problem, you'd get to keep your fancy widget, and would probably be a 5-minute patch.
We could mark this issue as fixed and get on with releasing 4.6, and revisit other options for the next release.

totten · March 18, 2015, 04:11:13 pm

Quote from: Coleman Watts on March 18, 2015, 11:03:33 am

Tim, I also like the tabbed interface, and with my experience of tricking out select2 to create the entityRef widget I'm pretty sure I could implement it.

Cool. It seemed to be getting pretty deep into the nitty-gritty of select2, and I couldn't find/grok an interface in the docs that looked appropriate. +1 for you figuring it out.

Quote from: Coleman Watts on March 18, 2015, 11:03:33 am

One caveat: select2 v4 has been released (or is about to be) and it is a significant rewrite with a lot of apis changing. All of the crazy stuff we've done to customize it will need to be upgraded at some point
But I have heard the new version is more performant

The hard part, of course, is understanding it. Since you've already done a lot of that in select2 v3, it might be worth taking a stab at in v3.

Quote from: Coleman Watts on March 18, 2015, 11:03:33 am

And a question: loading all groups into clientside memory seems "okay" because the number of groups in a db doesn't grow linearly and even orgs with huge numbers of contacts will still have a reasonable number of groups. # of mailings on the other hand does grow linearly, with no limit. An org that puts out 2 mailings a week will have 520 of them after 5 years, 1040 after 10 years, and so on.

Yeah. Related thought... it would neat if, e.g., we could search groups with a prefetch and then fill-in mailings based on AJAX. However, I suspect that's a bit complicated.

Quote from: Coleman Watts on March 18, 2015, 11:03:33 am

With possibly one mitigating factor - if the widget excludes "archived" mailings (does it?) and the org is in a regular habit of archiving mailings over 1 year old, then the number should always be manageable.
Alternately, the selector could hard-code a time or # limit (say, 2 years or 500 mailings) and simply not show the rest.
This would be the rock-bottom simplest solution to this problem, and IMO well worth considering, because:
With an ever-changing contact database, the recipient list for mailings over 2 years old really isn't worth re-using.
The change would fix Ken's problem, you'd get to keep your fancy widget, and would probably be a 5-minute patch.
We could mark this issue as fixed and get on with releasing 4.6, and revisit other options for the next release.

That's pretty good analysis.

I like the idea of constraining the #mailings, especially archived mailings and otherwise-old mailings.

Not sure if it's a coincidence, but in benchmarking my Win XP VM, I've found that the break-even point where prefetch and AJAX-based UI's feel equally laggy seems to be around 2,000 rows (aka 500 mailings and 500 groups). If we keep the current UI, then 500 seems like a proper limit. If we changed the implementation to one which doesn't require a 4x multiple (like this Angular POC, https://github.com/civicrm/civicrm-core/compare/4.6...totten:4.6-poc-grpml-list?expand=1#diff-6b6b2d2bf6e0b3c5f07720a7ac5d73a5R28 ), then I think we could safely bump the limit to 1000 or 2000.

Proposed filter for mailings:

Code: [Select]

SELECT id, name, scheduled_date
FROM civicrm_mailing
WHERE is_completed =1 AND $mailingACLs
ORDER BY is_archived asc, scheduled_date desc
LIMIT 500
-- note archived mailings float to the bottom

Coleman Watts · March 19, 2015, 11:32:49 am

That query could probably be represented as api params.
Question: how do AB tests factor into that query? I think we'd want to filter out the As and Bs and just show finals right?

totten · March 19, 2015, 02:26:23 pm

I'd vote for including A+B tests but don't feel strongly about it. To implement that, one could filter on "mailing_type IN ('standalone','winner')".

Coleman Watts · March 21, 2015, 10:54:16 am

Ok will have a go.
https://issues.civicrm.org/jira/browse/CRM-16155

CiviCRM Community Forums (archive)

News:

Author Topic: New mailing - recipients select box is s-l-o-w [FIXED] (Read 1462 times)

ken

New mailing - recipients select box is s-l-o-w [FIXED]

Coleman Watts

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

totten

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

ken

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

Coleman Watts

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

totten

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

totten

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

ken

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

totten

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

totten

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

Coleman Watts

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

totten

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

Coleman Watts

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

totten

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w

Coleman Watts

Re: New mailing - 4.6.beta3 - recipients select box is s-l-o-w