CiviCRM Community Forums (archive)

In our work with Doctrine, some basic questions have arisen: how would a developer use it? And how would we phase out DAO?

For PHP developers writing server-side code, the answer can be straight-forward: start writing code that uses Doctrine directly! Eventually rewrite or deprecate code that relies on DAO.  This has an upside because Doctrine provides much richer interface than DAO.

For developers writing remote code (such as rich Javascript apps), one doesn't call DAO directly; rather, one uses a number of APIs which have been built on DAO/BAO.  Phasing out DAO would mean reimplemeting or replacing these API's.  When tackling this problem, we should continue a trend in our API's evolution: shifting away from complex, adhoc API implementations towards thinner APIs driven by metadata.

One approach -- which I experimented heavily with -- would be to fully replace our DAO+API stack with Doctrine and a REST library (like http://leedavis81.github.io/drest/ , https://github.com/FriendsOfSymfony/FOSRestBundle, or http://hateoas-php.org/ ).  This sounds promising at first because it uses more third-party code (and reduces the amount of inhouse code), but fundamentally these are libraries -- not a fully-formed API.  There's a lot of valuable, generic services in our current API layer -- things like permissioning, authentication, chaining, option-values, and Drush+Smarty bindings -- which would need to be reimplemented or reinvented in some similar-but-different way to use these libraries.  Crucially, reimplementing the API's generic services would fail two goals: it would significantly change API DX and it would still leave us with a new chunk of code to maintain.  I think it would be OK to compromise on one of these for 5.0, but I'm skeptical of compromising both.

The next approach is to keep our basic API DX and runtime but allow incremental progress:

• Use the same basic API layer for Doctrine- and DAO-based APIs -- civicrm_api(), CRM.api(), {crmAPI}, etc.
• API requests with "version=>3" use DAOs; with "version=>4", use Doctrine.
• Initially provide very thin, generic Doctrine-based APIs for a long list of entities.
• Use Doctrine metadata (annotations) and classes to drive APIv4 (instead of XML + GenCode + procedural magic-functions).
• As the Doctrine-based APIv4 matures, deprecate the DAO-based APIv3.
• As the Doctrine-based APIv4 matures, introduces more services/interfaces which take advantage of Doctrine.

This incremental approach requires some changes to implementation of civicrm_api() and its various helpers -- which leads to the final topic: making the the API-layer cleaner and more flexible. I've developed a patch here:

• Overall patch: https://github.com/totten/civicrm-core/compare/civicrm:doctrine...doctrine-api-kernel?expand=1
• Old civicrm_api(): https://github.com/civicrm/civicrm-core/blob/0ce8483729744eef1649ad91f908fda25ee5223f/api/api.php#L13
• New Civi\API\Kernel::run(): https://github.com/totten/civicrm-core/blob/360207acabb39d2c211934978ccfd29b089ea352/Civi/API/Kernel.php#L69

A few things to notice:

• The original civicrm_api() has three different calls to $transaction->rollback for different call-paths. The new kernel doesn't need to know anything about how transactions are implemented. Instead, that's handled by https://github.com/totten/civicrm-core/blob/360207acabb39d2c211934978ccfd29b089ea352/Civi/API/Subscriber/TransactionSubscriber.php -- and there's only one call to rollback().
• The original civicrm_api() looks up a callback function using our magic-function-naming convention (line 49), and -- when executing the callback -- the function signature is adapted depending on how the particular callback was designed (line 77-91). In the refactored version, the magic-function-naming convention has been isolated in its own class ( https://github.com/totten/civicrm-core/blob/360207acabb39d2c211934978ccfd29b089ea352/Civi/API/Provider/MagicFunctionProvider.php#L37 ). We can implement other API providers which work differently (eg generic providers based on Doctrine, the custom-data system, or third-party services).
• To get a high-level summary of features supported by the kernel, look at https://github.com/totten/civicrm-core/blob/360207acabb39d2c211934978ccfd29b089ea352/Civi/Core/Container.php#L192 . Features can be added and removed by adding/removing event subscribers.
• I've been frequently running the api_v3_SyntaxConformanceTest (and spot-checking some other tests); so far, this has avoided several regressions.
• You might expect that adding several new classes and event listeners would impact performance, but some quick performance testing on my laptop suggests that the changes have little affect. (See data the bottom)

The work isn't done yet -- my next step is writing a DoctrineCrudProvider which exposes all Doctrine entities via the API (as long as they have the annotation @CiviAPI\Entity).

---------------
Performance Without Event-Driven Kernel
 * Running SyntaxConformanceTest takes ~139sec. (Three trials: 139sec, 138sec, 139sec). ((NOTE: This is a good test of "running a huge number of API calls within one request".))
 * Running "drush cvapi contact.get" 50 times takes 59 sec. ((NOTE: This is a good test of "running a small number of API calls but bootstrapping frequently".)

Performance With Event-Driven Kernel:
 * Running SyntaxConformanceTest takes ~139sec (Three trials: 139sec, 142sec, 139sec)
 * Running "drush cvapi contact.get" 50 times takes 59 sec.

Tim - I must say I like this approach as it moves the API 'intelligence' down to the Doctrine layer as much as possible - and therefore all core code also benefits from it. The API then becomes a 'thin translation layer' between our various external interfaces and the Doctrine core/annotations/classes. This is appealing from an architectural point of view as it improves consistency and minimizes reimplementation between core and API.

So, putting aside 'odds & sods' - I think the existing query & dao get types are the ones we need to plan ahead for. I don't feel like other crud items have any systematic differences (ie. some of them are a bit odd but there is no fundamental reason for them to be unstandardised).

Putting it another way - the two methods do one of 2 things
1) expose advanced search / search builder functionality to the api
2) expose fairly basic systematic access to the api

So, I guess the first question is - are these going to converge or remain as 2 strands. Note that the search builder functionality goes quite far beyond anything systematic in some cases - e.g the relationship based critieria involves constructing temp tables to deal with the potentially unknown relationship direction.

The basic strand is somewhat inadequate currently - e.g. address api will not apply contact based acls.

What happen to the BAO in this picture? Some actions replies on it (eg the create one).
Some API users relies on some of the BAO features (eg triggering the hook pre/post).

Can doctrine cover both the DAO and the BAO (eg. with the annotations), are we going to migrate BAO to use doctrine?  Do you see any easy way out?

X+

Hey Xavier,

I would argue that doctrine replaces the BAO logic as far as I understand BAO. Let me explain what doctrine does. It is a wrapper for storing and retrieving objects (not relational data) and that is why I think doctrine handles the BAO logic. because the BAO logic should be in the entity class. For example we should have an Entity class Contribution. This class holds information about the amount, receive date, status etc. but also a link to the source object (which could be an event, membership) and a link to the contact. Al those links are object-relationships. Then we have a standard repository class which we can override for the contribution. So we get an ContributionRepository. This repository is responsible for retrieving objects. Standard doctrine comes with functions as FindById, FindAll, FindAllById, everything after the FindBy and FindAllBy are properties of the entity class. So without any extra code one can do ContributionRespostiro->FindAllByContact($contactObject).
With lazy loading doctrine will load all linked objects to the contribution object.

To conclude the business logic of an entity goes into the entity class (BAO), the logic for fetching objects from a store (database) goes into a repository class (DAO).

@xavier: what we should do is to use validation rules on an entity and event listeners. So we can use the validation rule for checking an email address already extists. And the event listener on the event pre update of the e-mail adress entity to make sure there is one primary e-mailadress.

Yes that is a very good approach. In my opinion there is a difference between a Doctrine Entity and an BAO for example. The BAO checks the business logic is chekced in BAO as soon as the object is saved also there are busniess logic methods to achieve something.. E.g. in the case the BAO Mergecase function is used to reassign a case to another client. In Doctrine I would assume that the function setClient, addClient, removeClient etc are used on the entity. And the logic for chaning something is hidden between those setters and getters. Also in Doctrine we have to think of valid entities in the system (not in the database). So one of the powerfull rules is that one can add validation constraints on an entity. And with the event listener one can insert their own constraints.

So we should start with writing down the entities (and the data they contain) and make their validation as simple as possible, as soon as it becomes a bit too complicated we should start thinking of putting those validation in their own 'Bundle' and inject them with event listeners and dependency injections.

A good rule of thumb here is that one should write down the responsibility of a class in a sentence of max 25 words. If the sentence becomes longer or to many 'and' or 'or' are used then one should split up the responsibility into different classes. This will result in many class files... (harder to find where a functionality is defined) but the functionality self is clearer to understand and much more maintainable. Also this creates loose coupled code.

As far as I am concerned I think at this moment the BAO has to much responsibility in the system. And it is not always clear which BAO is repsonible for something (e.g. I was making my own search the other day and I had to use the Contact BAO for the validation of the search form, while my search was not searching for contacts but for documents).

+1 Agree that listeners/hooks would be really appealing and would loosen coupling.

There is a lot of caching involved in Doctrine. A few important caches:
 * Metadata cache. By storing the list of annotations (etc) in cache files, they avoid the need to use reflection and to parse annotations at runtime.
 * Within a given entity manager, there should only be one instance of the same entity. (IIRC -- Hibernate definitely does it this way, I'm 66% sure Doctrine does too). For example, in the following snippet, $a, $b, $c, and $d would refer to the same PHP object (rather than different objects with the same values).

$em = CRM_DB_EntityManager::singleton();
$a = $em->find('Contact', 1); // first read; hits DB
$b = $em->find('Contact', 1); // load from in-memory cache
$c = $em->find('Contact', 3)->getEmployer(); // load from in-memory cache
$d = $em->createQuery('SELECT c FROM Contact c WHERE c.id = 1')->getFirstResult(); // query DB then check in-memory cache