CiviCRM Community Forums (archive)

I'm wanting feedback on continuing with use of internally developed spooler for asynchronous bulk outbound processing, versus creating a dependency on a Doctrine bundle.

CiviCRM currently has a fairly simple spooler at CRM_Mailing_BAO_Spool::send, with the asynchronous sending of emails handled by the Send Scheduled Mailings scheduled job. CRM_SMS_Form::postProcess also calls into CRM_Mailing_BAO_Mailing::create so it also uses this spooler.

As part of work on Phone Integration we will need to have functionality to spool and asynchronously deliver Voice Broadcast messages. Down the road we will also want to spool more complex interactive voice response (IVR) calls eg to get people to respond to CiviSurvey questions without needing an operator.

Looking at http://knpbundles.com/keyword/queue I see lots of options for bundles that provide queuing for work jobs to be run asynchronously. Many integrate with well-maintained high quality services: RabbitMQ, redis, beanstalkd. Most of them require some sort of package management to install a daemon that would effectively replicate what we are doing with Drupal cron. Some like queue provide a thin layer that allows allow various services to be substituted under the interface.

There are other sorts of functionality that could benefit from asynchronous jobs besides those currently setup as scheduled jobs. Long running imports are the one I think about most - Salesforce does imports this way, sending completion notices out when done.

Any feedback on creating a dependency on queue or an equivalent? I think it would be useful if we implemented our own spooler and scheduled job that did not require installation of any package on servers with restricted access. For installations with higher end needs and greater sysadmin privileges, the external packages could provide scalability, failover, etc. Once this is working for phone blasts, the mailing spool could be refactored to take advantage of the same approach. But this is maybe over-engineering.

Feedback?

Thanks alot. For now we will just use the core queue implementation, though we may get keen and refactor civimail to use too.