Complex financial entities

[Eileen]

Hi all,

Joe & I discussed the api & financial stuff (technical term) today on IRC & after the discussion we are both in favour of all financial transactions passing through the contribution.create api - ie. if a participant is to be created with a payment then the call would be

civicrm_api3('contribution', 'create', array(
  'total_amount' => 50,
  'financial_type_id' => 4,
  'payment_instrument_id' => 'Check',
  'contact_id' => 7,
  'participant => array (
    array(
      'event_id' => 8,
    ),
   'membership' => array(...)
   'pledge_payment' => array(...)
  )
);

The api would create the contribution, line items, financial transaction records, & participant record(s)

This is a break with the previous pattern but my ensures that the process creating the financial records has 'all the information' & as Joe points out means that in failure the whole thing is rolled back & we are no longer asking people to create the complex entities. It also saves devs the pain of constructing participant_payment records & figuring out which entity the line items belong to. A few notes

1) Ideally the line items would go under contribution even when they are participant line items - not sure if the BAO can figure that out. We might need to put on the line item 'is_participant'
2) I used a double array on participant as it could be multiple & think we should use that pattern for all
3) contact_id, status_id would be optional on participant
4) I didn't use the api.participant.get as we are implying a much tighter coupling
5) I put membership & pledge_payment in the above eg to think about - it kind of highlighted that for pledge_payment the pledge_payment rather than pledge would be passed in & would need to be pre-existing.

[xavier]

Would make it easier indeed. Do we need to specify the event_id? it seems it's either too generic (ie. you have have several contact_id+event_id participants) or too specific (if the contrib id is given, the api could probably figure out direclty if a participant needs to be updated).

On a tangential note, didn't you work on some generic stuff for the IPN to automatically update the related entities when a contribution is confirmed/canceled?

[Eileen]

In this case I'm looking at the create instance - so event_id is the only compulsory participant field that is not also provided for the contribution. However, I would expect the contact_id or indeed the status_id might differ - so they would be optional params

[JoeMurray]

So we want an API for contribution create that can support all of the post process functionality to complete the purchases on event registration pages and on contribution pages with memberships configured. In other words, we want to end up eventually moving the business logic out of the forms and into a big new function with an API interface. This would allow a REST call to do the same thing as one of those forms, and the db changes would all be done correctly, and emails sent out, etc. That's the goal.

Here's a modification of Eileen's call signature that provides all the needed info for both the financial side and for the non-financial/'business'/component side of creating participants, memberships, etc. As I don't have a good name for this, I just shoved the params for creating the object related to a line_item - a membership, participant, etc - into an array parameter line_item_params.  Although calls from existing forms would not mix participants with memberships, etc., I've put them all into one call for convenience here. I see the API looping through the

civicrm_api3('contribution', 'create', array(
  'total_amount' => 50,
  'financial_type_id' => 4,
  'payment_instrument_id' => 'Check',
  'contact_id' => 7,
  'line_items_with_params' => array (
    line_item_with_param => array(
      'line_item' => array(
        'entity_table' => 'civicrm_participant',
        'entity_id' => NULL,
        ....
      ),
      'params' => array(...), /* to create participant */
    ),
    line_item_with_param => array(
      'line_item' => array(
        'entity_table' => 'civicrm_participant',
        'entity_id' => NULL,
        ....
      ),
      'params' => array(...), /* to create participant */
    ),
    line_item_with_param => array(
      'line_item' => array(
        'entity_table' => 'civicrm_membership',
        'entity_id' => NULL,
        ....
      ),
      'params' => array(...), /* to create membership */
    ),
    line_item_with_param => array(
      'line_item' => array(
        'entity_table' => 'civicrm_contribution',
        'entity_id' => NULL,
        ....
      ),
      'params' => NULL, /* not needed */
    ),
    line_item_with_param => array(
      'line_item' => array(
        'entity_table' => 'civicrm_pledge_payment',
        'entity_id' => NULL,
        ....
      ),
      'params' => NULL, /* not needed */
    ),
  ),
);

The API function will use the name of the entity_table to determine what needs to be done with the params. I put the entity_id in for illustration purposes, but I don't think it should be in there in practice. Once the object if any is created, its ID will be used for the entity_id for the line_item. As appropriate, membership_payment, pledge_payment, etc records will also be created.

Currently, line_item.entity_table is set to 'civicrm_contribution' for memberships, which isn't quite the right semantics of this field as I recall them. To avoid backward incompatibility in case anyone is relying on this ;( we could complexify things by using a new and different arg to indicate that an object needed to be created for each line_item.

On an additional minor note about how the implementation didn't seem quite right to me, I also saw that on a pay_later payment for a single ticket that line_item.participant_count was 0, though I thought it would show 1.

Should we change the call from contribution.create v3 to something else? Don't we still need a hidden lower level _contribution.create to create just the contribution record?

I haven't delved down into the dependencies between the line_item_with_params. For example, I believe that 2nd and later participants set a parent_id to the first participant. I think some intelligence should remain in the form, e.g. if user wants different transaction for membership than for donation, the form would call API twice rather than once. But I don't have strong views on that.

[Eileen]

I think I'm OK with the syntax as you have suggested - although it needs to have the price set it to calculate the field values - so we should add that in. We should just key the line items in the line_items_with_params array by irrelevant keys (e.g sequential).

Currently, line_item.entity_table is set to 'civicrm_contribution' for memberships, which isn't quite the right semantics of this field as I recall them. To avoid backward incompatibility in case anyone is relying on this ;( we could complexify things by using a new and different arg to indicate that an object needed to be created for each line_item.

- I'm not feeling quite happy with this at the moment - but I'd rather focus on the 'outside' - the api wrapper having sensible inputs & outputs & if fns are using the wrapper we can alter this later. I would like to have a separate conversation on this

I think some intelligence should remain in the form, e.g. if user wants different transaction for membership than for donation, the form would call API twice rather than once.

that makes sense to me

[xavier]

Instead of introducing a new special param (always a pain to document/explain....) can't we create a ContributionLine entity an chain?
 

civicrm_api3('contribution', 'create', array(
  'total_amount' => 50,
  'financial_type_id' => 4,
  'payment_instrument_id' => 'Check',
  'contact_id' => 7,
  'api.ContributionLine.create' =>  array(
        'entity_table' => 'civicrm_participant',
        'entity_id' => NULL,
        ....,
        'api.participant.create'= array()
      ),
  'api.ContributionLine.create.2' => array(
      'line_item' => array(
        'entity_table' => 'civicrm_membership',
        'entity_id' => NULL,
        ....,
        'api.participant.create'= array()
      ),
...  ),
);

The API function will use the name of the entity_table to determine what needs to be done with the params. I put the entity_id in for illustration purposes, but I don't think it should be in there in practice.

No matter the syntax, I'm wondering if the chaining logic is in the right order: shouldn't it be Contrib->Participant->LineItem instead of Contrib->LineItem->Participant ?

I think that The LineItem could even be implicit: it you create a related entity within a main contribution.create, you can go ahead and create the needed LineItem.

[Eileen]

My reservation with that is that the chaining syntax is what we use when different BAO calls create different things & they are loosely coupled. In fact, however, the contribution.create call already creates the line items but actually creates them incorrectly if it doesn't know it is creating a participant item. So the coupling is much tighter than the api optional chaining syntax implies. We are really looking to say 'This is the way you should be creating entities with payments' & have one recommended method.

Joe in your syntax suggestion - where would payments fit in?

[xavier]

Good point,

however, if we chain contribution.create->api.participant (or api.membership or...) then you can keep the implicit "hidden" line creation (you are right, the contribution line shouldn't be a "normal" entity chained in that case)

if you call contribution.create without any chaining, it works as today, if you want to have the correct lines, chain with api.participant, api.membership and the contribution.create is smart enough to know which line(s) to create

Would it work?

[Eileen]

No, currently contribution.create is not smart enough to create the correct lines for a participant.create. My argument against using the api.participant.create syntax is that is suggests that passing in the participant is optional. Actually it needs to be required because the way the line items are created needs to be taken care of by the contribution.create call.

I think we should have one supported 'look' for creating contributions with related entities & if you want to wing it you are on your own.

[Coleman Watts]

I'm going to take the example of webform_civicrm since that's what I'm most familiar with. It's also the only software I know of that attempts to create participants, memberships, and a contribution under a single transaction. As you can imagine doing so was a little hairy and I would welcome these proposed improvements.

I would particularly welcome getting rid of the membership_payment and participant_payment join tables, as those seem to be redundant with line-items (or at least they would be if membership line items reported the correct entity).

Re the proposal to add contribution_id to the line item table - this seems like a good change but could also result in some redundant data if not thought through completely.

I have some other questions about this but don't want to get too detailed too early. Maybe we should schedule a more in-depth discussion at some point.

[JoeMurray]

Thanks, Coleman.

In general I agree with you.

Eileen, I see the way that one line item has a one to many relationship with participants when a price set item purchases more than one participant ticket. When you speak of the one participant to many line_item records relationship, can you provide an illustration? My sense is that there may be many financial_item records for a line_item and participant as it goes through pay later, partially paid and paid, etc. But I don't see the use case for a single participant. Is it that there are multiple charges associated with a single ticket, eg taxes? As I understand charitable tax receiptable amounts here in Canada at least, there is no need to put them into a different revenue account, and thus no need for separate line items for the receiptable and non-receiptable portions of ticket price.

[Eileen]

A typical scenario of one participant having many line items is when a price set it used to present dinner options & accommodation options for a conference - causing the total price to vary on what is selected

[JoeMurray]

Interesting. Certainly from a high level business perspective they are all line_items related to the single event participant. I tend to think of the price set element that actually increments the participant count for an event as the only line_item related to participant record. This is probably not as useful.

[Coleman Watts]

I think there is some ambiguity about how line_items should be used. In webform_civicrm I use line_items only to represent a real entity (contribution, membership, or participant). But core civicrm seems to mostly use them to represent price set selections, e.g. choice of soup would be a line_item.
I tend to agree with Joe that "soup" is not an entity worthy of being recorded as a line item, but I'm not sure how we would get from point A to point B with that change.

[JoeMurray]

If there are pricing differences with soup vs oysters for appetizer, then it makes sense to me that they are line_items. I sometimes wish that free/non-cost items didn't end up in same table, but I don't even want to propose changing that. There is a variety of ways people have used price sets to deal with complex events. For example, the RNAO tracks breakout sessions with price set options, and some but not all have extra costs associated with them. With developments in recent versions of hierarchical events I think that one can now set up these as separate sub-events, which would be great since it allows session attendance to be capped by room size for simultaneous breakouts. Their 3.5 day AGM has a 2 or 3 page printed registration form. This is not a use case that Webform addresses so far as I know.

So I'm not sure we need to push changes here, just maybe think about terminology and watch for some edge use cases like Eileen identifies. Are there any reporting difficulties here? I seem to recall a few a few versions ago for the RNAO.