Queue Runner documentation?

[lolas]

Now that there is a batch queue runner in CiviCRM is there any documentation on using it?

So far I found these two examples to go by. I am sure there is something in the CiviCRM upgrade code as well.
https://github.com/civicrm/civicrm-core/blob/master/CRM/Admin/Page/ExtensionsUpgrade.php
https://github.com/civicrm/civicrm-core/blob/master/tests/phpunit/CRM/Queue/RunnerTest.php

[totten]

There's an extension called "org.civicrm.demoqueue" which may be easier to read than the others:

https://github.com/civicrm/civicrm-core/tree/master/tools/extensions/org.civicrm.demoqueue
https://github.com/civicrm/civicrm-core/tree/master/tools/extensions/org.civicrm.demoqueue/CRM/Demoqueue/Page

You've probably figured this out already, but the main concepts are:
 * Task: A particular unit of work. (Note: The queue can store anything serializable, but if you want to use the QueueRunner, then the queue should store Task objects.)
 * Queue: The stored list of tasks. This usually stored in a SQL table (CRM_Queue_Queue_Sql).
 * QueueRunner: The worker which consumes the tasks iteratively, executing each. For CLI usage (like drush), one would call runAll() or runNext(). For the web UI, one would call $runner->runAllViaWeb(). runAllViaWeb is a terminal call - executing it will immediately trigger a redirect to the queue UI.

There's a little more documentation about the options accepted by QueueRunner in the docblocks:

https://github.com/civicrm/civicrm-core/blob/master/CRM/Queue/Runner.php#L90

[lolas]

Tim,

Thanks for links! I hadn't figured out all that stuff out yet because I probably won't get to try to use the Queue until next week. Perhaps I can write up a couple of notes on the wiki after I do.

[jaapjansma]

I know that this post is a bit old though I am still having the same question. What about documentation and tutorial on using the CiviCRM Queue system? Is that available and where do I find it?

[jaapjansma]

I have written a tutorial on how to use the CiviCRM Queue mechanism. See http://bit.ly/1x8IqYQ

[totten]

Cool! Thanks, Jaap.

== Suggestion #1: Interactive vs background use-case ==

There are two steps here:
 * Step 3 create a page which executes all tasks in the queue
 * Step 7 create a cron job

These seem like two alternative user-experiences to me -- you'd pick one depending on the use-case. If you want to run an interactive import and display a progress bar, then you'd do something like step 3. If instead you wanted to defer a task so that it runs in the background (headless), then you'd setup a cron job. So maybe it would make step 3 would be more like:

 * Step 3: Decide how to execute the tasks
   + Step 3a: Execute the tasks interactively (web-based runner)
   + Step 3b: Execute the tasks in the background (cron-based runner)

== Suggestion #2: Structure vs Indirection ==

Comparing the wiki tutorial with org.civicrm.demoqueue, they take a different trade-off between being "structured" vs "simple". The tutorial (4-5 files, 11-12 functions) puts in a lot of structure but adds indirection and boilerplate which makes it harder to trace; the demoqueue (2 files, 6 functions; (https://github.com/civicrm/civicrm-core/blob/d7c8cf03e9d/tools/extensions/org.civicrm.demoqueue/CRM/Demoqueue/Page) is the opposite.

The greater structure seems useful in developing a generic/multi-purpose tool (like CiviRules), but it's more complex than I would want to write in a typical case. The demoqueue has a similar fault -- it's too simple; real code would be more structured.

How would about splitting responsibilities like this: https://gist.github.com/totten/dd9e9f02bc5ed68597db

(Random aside; the demo-queue uses CRM_Core_Page and sets up the queue in run(). I think for an extension author, they're more likely to use CRM_Core_Form and set up the queue in postProcess(). I was just too lazy to write out that example.)

[jaapjansma]

Hey Tim,

Thank you for your feedback. Indeed step 3 and step 7 are two different approaches.

About the structure it is indeed more structured but I tried to follow the standard civix method of generiting extensions. So this structure shouldn't be too difficult to follow because I have incporporated each civix step.

Jaap