CiviCRM Community Forums (archive)

The Canadian New Democratic Party is looking at alternatives to their current system for voter tracking and GOTV and are considering CiviCRM. While CiviCRM seems fine for their volunteer, donation, sign location, etc tracking, there are concerns about performance, scalability, functionality, and usability, particularly for high-volume data-entry.

It would need to be able to:

- handle all Canadian voters (about 25 million)

- import the data efficiently, since it arrives in 300+ files of approximately 80k contacts, and there will need to be at least two and possibly three merges done over a campaign of 35 - 40 days,

- merge phone company data (files vary in size from 150k – 8 million contacts), which would involve parsing addresses and incorporating heuristic match criteria based on name and address info,

- export data to call centres (current functionality),

- import and merge results back from call centre (current functionality?),

- provide walk lists and call lists,

- support approximately 1,000 simultaneous data entry staff with high availability entering information from these call sheets and other sources during a campaign entering foot canvass results,

- provide a user interface optimized for high-volume data entry (likely UI: entry on a long form with same data to previously printed/exported walk sheets),

- provide fairly simple voter intention tracking, eg party preference, second party preference, definitely not party aversion, likelihood to vote,

- provide virtual call centre functionality similar to what we previously discussed, number of simultaneous users undetermined.

I expect a phased approach to meeting these needs, including a small trial of a few hundred thousand contacts, might meet approval.

In trying to determine whether CiviCRM might be suitable for this, I had a few questions:

What are the largest simultaneous user load environments for CiviCRM?

What are the largest contact databases, and how do they perform under load? (Lobo recently reported that the largest 2.0/2.1 installation has 517,000 records, but there may be larger ones in earlier versions.)

I should mention there is still a strong bias amongst decision-makers to continue with their proprietary systems, given the several hundred thousand dollars invested in it over the last decade, and other alternatives are being actively explored as well.

It would be useful if people could post either here or in Showcase forum about large implementations, either working, considered, or abandoned. Large in this context means either a large volume of data, a large number of users, and especially heavy traffic sites. What have the scalability issues been?

The core team has worked on at least one enterprise implementations with front end apache and backend mysql clusters. How many other projects that are running in this sort of a scaled and high availability configuration? Are there significant issues? What ballpark of server environment would be required to support the user load with high availability: a load balancer, 3 – 5 good web servers, and 2 database servers?

Previously the core team implemented a memcache version of CiviCRM for a consulting project, but I believe that was a one-off for 1.6 or so, and has not been maintained as it is not a core requirement for most installations.

Have any installations attempted to partition their datasets (eg by state/province or region) to improve performance?

Early versions of CiviCRM had some issues importing large numbers of users, in large due to memory leaks that have since been patched, I believe. The new support for SQL query rather than csv imports should help in importing large datasets.

Information about organizations who would need large installations that decided not to implement CiviCRM due to issues around scalability / availability / performance would also be interesting.

Another question down the road is whether there would be enough in the way of interest and resources from large organizations to support an on-going a large / high performance release if it is determined that one is useful/needed.