DB_DataObject dependencies

[totten]

There has been renewed interest in replacing DB_DataObject:

http://civicrm.org/blogs/xavier/database-layer-evaluation

One important question in planning such an effort (among several) is this: To what extent does CiviCRM actually depend on the features of DB_DataObject? How many times is it (directly or indirectly) instantiated? Which functions are actually used -- and how often?

TECHNIQUES

In statically-typed languages (like Java), one can rely on static-analysis of the source code files to assess dependencies on functions and classes. (This is why things like automated refactoring work in Java... if you use an IDE like Eclipse/Netbeans with Java code, you can get a quick-n-dirty list of dependencies by previewing a "rename" operation.) But this probably isn't reliable in PHP.

In PHP, we *can* do runtime-analysis. There are existing tools for profiling and for aspect-oriented programming in PHP. I don't have much experience with them, so instead I hacked a small wrapper class for DB_DataObject (attached). This class intercepts and logs every function call which would otherwise be handled by DB_DataObject. (I also tried to intercept property-assignments using PHP's magic methods, but that attempt didn't work as I'd expected.)

DATA FORMAT

The attached patch produces a CSV. Each line corresponds to an invocation of a function in DB_DataObject. The columns are:

1. Concrete class: The full name of the concrete class for which the function executed. This should always be a subclass of DB_DataObject.
2. Injection technique: How did the logger receive a notification that the function was called? Was it using a overloaded function? Or PHP's magic function?
3. Operation type: Function "call", property "set", or property "get"
4. Function name or property name
5. Caller file
6. Line of caller file

For example, consider this line:

CRM_Core_DAO_Domain,overload,call,selectAdd,/home/abc/svn/civicrm/CRM/Core/BAO/Setting.php,205

This indicates that line 205 in CRM/Core/BAO/Setting.php depends on DB_DataObject because it calls selectAdd on an instance of CRM_Core_DAO_Domain.

DATA COLLECTION

To do a runtime-analysis, we need to run Civi through a variety of scenarios and accumulate log data. The results of running CiviCRM 3.4 (~3.4.4) through three test suites (api_v2_AllTests, api_v3_AllTests, and CRM_AllTests) are available here:

http://arms.dl.locker10.com/devel/apiv2_apiv3_CRM.csv.bz2

Please note that the coverage of this dataset is only as good as the coverage of the chosen unit-tests.

INITIAL OBSERVATIONS

There are a few hundred unique lines of code which invoke functions in DB_DataObject. The number goes down a bit if you exclude self-invocations (i.e. incidents in which DB_DataObject calls itself):

Code: [Select]

$ grep ,call, apiv2_apiv3_CRM.csv | cut -d, -f5,6 | sort -u | wc
    589     589   33206
$ grep ,call, apiv2_apiv3_CRM.csv | grep -v packages/DB/DataObject | cut -d, -f5,6 | sort -u | wc
    549     549   30933

The non-static DB_DataObject::query function (which accepts arbitrary SQL) is invoked many times, but there's only one line of code with a hard dependency on it:

Code: [Select]

$ grep call,query, apiv2_apiv3_CRM.csv | wc
  34353   34353 2721695
$ grep call,query, apiv2_apiv3_CRM.csv | cut -d, -f5,6 | sort -u | wc
      1       1      46
$ grep call,query, apiv2_apiv3_CRM.csv | cut -d, -f5,6 | sort -u
/home/armsys/svn/civicrm/CRM/Core/DAO.php,149

The non-static DB_DataObject::find function (which constructs SQL based on object-attributes) is also invoked many times, but there are many more files/lines which reference it:

Code: [Select]

$ grep call,find, apiv2_apiv3_CRM.csv | wc
  21591   21591 2088771
$ grep call,find, apiv2_apiv3_CRM.csv | cut -d, -f5,6 | sort -u | wc
    205     205   11567


[xavier]

That's awesome.

Code: [Select]

2. Injection technique: How did the logger receive a notification that the function was called? Was it using a overloaded function? Or PHP's magic function?

What do you mean by magic (working on api/class.api.php that's full of magic, might have a twisted mindset right now)?

Eileen is doing a great job profiling and identifying parts where civi goes crazy on the db:
http://forum.civicrm.org/index.php/topic,20697.0.html

Do you think your code could be extended to log as well what test is running, so we can group by that and see how many db calls per test? (probably not perfect, but surely a good indication of something running amok if you got 100th of calls on a single test).

As an aside, did start using sqlite3 instead of flat files (csv) on some pet project, I'm in love with that thing.

[totten]

Magic methods like __call() or __set() can be used to dynamically define class methods/properties. They're often used for generic programming/bindings.

http://php.net/manual/en/language.oop5.magic.php

I imagine we could capture test names. I'm not a phpunit expert,  but other xunit implementations provide a global variable to track the active test.

Haven't really used sqlite... is it easy to autocreate the db/schema?

[xavier]

Hi,

Wasn't sure you meant magic as in magic methods or as in hairy parts that behave. That's what I used in api/class.api.php To get things like $api->Contact->Get (the official magic, not the accidental one - had fun with it

As for sqlite, you

Code: [Select]


    if ($this->db = new SQLite3 ('/path/to/your/db.sqlite3' )){
      @$this->db->exec ('CREATE TABLE IF NOT EXISTS db_call (id int PRIMARY KEY, class varchar (100)... ) ');

and then do the regular insert...

X+

[totten]

At the risk of taking a hack too far, I cleaned it up a bit more; a revised patch is attached.

FWIW, the class "SQLite3" doesn't seem to be available on PHP 5.2, but PDO seems to work similarly.