Author Topic: Anyone tried running Smarty out of memcached? (Read 3932 times)

torenware · February 14, 2013, 08:54:10 pm

I've recently gotten a large CiviCRM 4.1 site up on AWS. It is generally working, but there are teething issues $:-\$

One annoying problem has been with handling the Smarty cache. We are running a cluster of autoscaled instances on AWS, and for the moment, template_c is sitting under sites/default/files/civicrm, and is really getting served off an NFS server.

Since we also use CiviMail and have started using cli.php for the required scripts, we're starting to have problems where the access rights on some subdirectories of template_c are getting messed up. This is a hard problem to solve with NFS.

A quick look on the web indicates that Smarty _has_ a way of changing the caching layer, and that it's an old part of the Smarty system. CiviCRM appears to be using Smarty v2, and even there, the hook is there. It's documented at http://www.smarty.net/docsv2/en/section.template.cache.handler.func.tpl.

Now, to be the first person to try anything like this on CiviCRM, you have to be kind of stupid. And Lir knows, as far as this things go, I'm definitely dumb enough to try something like this. And we're already using memcache (hell, we're evening using memcache in some CiviMail scripts). So I might be just dumb enough to try this as well.

But no one likes to be that stupid unless it's absolutely necessary. Have any of you actually _tried_ this? Did it even work? If it did, were you happy with it?

Thanks for any answers on this one.

torenware · February 14, 2013, 11:51:01 pm

I've done a quick experiment. And answered my own question.

Short answer is, well, it was a stupid idea :-)

Longer answer is: there's a cache hook, and it does work. But it's not why we have so many Smarty related access problems. The real problem is that Smarty really, really wants template_c to be on the file system, so it can compile files and stick them there. While there's no real reason those files couldn't be in a cache, the assumption that these files are on a file system that understands things like is_writable() and similar.

This is not a necessary architectural decision for Smarty, but it's a very deeply rooted one, at least in Smarty v2. So what I have is a dumb idea that is not likely feasible.

It would be good to get CiviCRM out from under Smarty. I know that some of the core folks are looking at Symphony for various new bits of machinery. This is another place to look for it.

Donald Lobo · February 15, 2013, 08:48:43 am

Any specific reason why you dont put templates_c on a local disk or a RAM disk instead of a NFS partition?

lobo

torenware · February 16, 2013, 12:37:04 pm

I'm open to the idea, and in fact, am currently trying that. I'm calculating the current user name, since we use cli.php and therefore scripts don't always execute as the web server's user, so for now, templates_c is now at

/tmp/USERNAME/templates_c

Each independent web head has its own copy of templates_c, which appears to work.

We're _still_ getting problems with Smarty, even in this configuration (See http://issues.civicrm.org/jira/browse/CRM-11907).

I don't think I'm saying anything too controversial if I say this: Smarty is probably the biggest cause of installation errors with CiviCRM, and has been for a long time. Lots of issues on this forum, since some users have problems getting it set up right from the get-go. But even for folks who have been using CiviCRM for a long time, there are a lot of problems that only resolve themselves if you delete templates_c, which seems to be subject to corruption under a wide range of situations. Or to access problems related to Unix file system access issues.

I know you folks are looking at moving away from it, and I for one will be happy to see Smarty just... go away. Swapping it out won't be a small job, I know. But it will make CiviCRM a lot better.

Donald Lobo · February 16, 2013, 04:33:37 pm

I'm not very sure i agree with most of the statements below. I dont think smarty is a big cause of errors in the recent past. It has its warts and issues, but overall it has been qutie useful / scalable etc

To a large extent some of the issues you are seeing is due to deploying in a different environment with its own set of issues.

lobo

torenware · February 16, 2013, 04:48:52 pm

Quote from: Donald Lobo on February 16, 2013, 04:33:37 pm

I'm not very sure i agree with most of the statements below. I dont think smarty is a big cause of errors in the recent past. It has its warts and issues, but overall it has been qutie useful / scalable etc

To a large extent some of the issues you are seeing is due to deploying in a different environment with its own set of issues.

lobo

Certainly it's a pain point on installs. And while it may scale on a single host well, just about anything that gets written to disk is a problem when you use auto-scaling.

What makes our particular case difficult is that the auto-scaled instances are largely inaccessible. This is becoming more and more common; while the deploy system we're dealing was developed by the DNC (Democratic National Committee) and is a custom system built in python and based on wrapping stuff in to CentOS RPMs, almost any system you're going to use today -- Capastrano, Chef and Puppet among others -- are not really going to let you go in and tweak too much on a deploy's file system. Even the Drupal model for deploying files into sites/default/files is a major PITA, and can be difficult and expensive to implement.

So it's been pretty difficult to debug this stuff; a lot of the work has been instrumenting various calls so we can better see what's going on.

Also, the version of Smarty used in CiviCRM is not very modern -- it's a 2.6 from what I see in your version notes in packages/VERSIONS. So certain things that might make Smarty easier to integrate into cloud deployments aren't yet in trunk.

xavier · February 17, 2013, 03:37:12 am

Hi,

Can't you run the cli as www-data?

Another option to explore (and that might improve performance a lot) might be to disable
http://www.smarty.net/docs/en/variable.compile.check.tpl
so you'd spend less time deleting/recreating the cached files. Not sure how it works for the templates used by civimail.

micah tried to work on it and if I recall it wasn't super sucessful. you might want to ping him on IRC to see if you can pick up where it left it.

X+

CiviCRM Community Forums (archive)

News:

Author Topic: Anyone tried running Smarty out of memcached? (Read 3932 times)

torenware

Anyone tried running Smarty out of memcached?

torenware

Re: Anyone tried running Smarty out of memcached?

Donald Lobo

Re: Anyone tried running Smarty out of memcached?

torenware

Re: Anyone tried running Smarty out of memcached?

Donald Lobo

Re: Anyone tried running Smarty out of memcached?

torenware

Re: Anyone tried running Smarty out of memcached?

xavier

Re: Anyone tried running Smarty out of memcached?