So I heard rumors that one of the teams at Liip was struggling with the performance of a symfony app, which was casting doubt on symfony in general. Now that team solved their performance issues by using a filter to do some aggressive caching. This fit nicely in the content workflow anyway since updates do not come in all the frequently anyway. So they just cache everything and added a feature to prime the caches for content changes. Problem solved, but doubts remained for other projects. So I wanted to figure out what was going on.
The problem was that I couldn't easily get a good set of test data and that their staging setup wasn't setup with XHProf yet. So I began some static code analysis. Saw a few cases where they were using "include_components" instead of just "include_partial". I noticed that they had a few cross application links, but then again they were using memcache to cache those. I wondered if they should have used pgpool to more efficiently handle the connections to PostgreSQL. Also they didnt seem to be caching Doctrine generated SQL queries yet because they ran into some issues with that enabled during the development.
Now the other day we finally had XHProf on the staging server with a current set of data and all the same setup as production. Jordi and I began disabling all the various caches and performance was indeed super slow, 4-5 second slow! We checked XHProf and found that some Doctrine methods were being called 20k-80k times! The cause seemed to be their custom tree algorithm.
I guess they could have used nestedsets here, since as I mentioned earlier their data doesn't change that often. Looking at the tree algorithm we noticed that getId() and getParentId() where being called multiple times in one of the key methods. So we stored the method return values into a local variable and shaved off 2 seconds. Jordi then continued some more optimization and was able to drop the performance without any caching down to 600ms without any output caching.
I guess by enabling the cache for the header and footer, which change almost never, but require a bit of work, this number will drop even more. The database access now makes up about 120ms per request and this could get tweaked some more. Of course now there are some more areas worth optimizing at this stage and maybe now the things I spotted during the static analysis are starting to matter.
Now we asked the team why they didn't notice that their tree algorithm was this slow. They actually did notice, but it was earlier in the development of the project and obviously with less data in the test env it was noticeable, but not so severe. Now at this stage things were hectic and not everybody was totally familiar with symfony, so it ended up being considered a framework overhead thing. And thanks to aggressive caching there wasn't really an issue for this particular project and the site was able to be launched with great performance.
Now the lesson learned are:
BTW: If you never heard about XHProf, learn about it in this excellent article written by Lorenzo.
At least you dug into it. I recently worked for a client that had one of their sites built by an agency that felt that caching was the solution to everything. They didn't consider the option that the problem might lie elsewhere and they actually needed to solve the problem instead of just turn on caching. Sure, with page caching turned on, their response times were quite nice, but with that they were hiding the real problem at hand: Something really wrong in the code.
Good to see you dug further (though I hadn't expected otherwise), and good to see you sharing the findings with the world.
interesting sum-up and a success story apparently! one question: do you think it is really a good idea to have xhprof running on a production server?
well we have the extension installed, but we do not normally have xhprof enabled (aka we dont call xhprof_enable() by default). however its nice to have the option to at time sample xhprof output from production servers.
I totally agree on having live data since the circumstances are often very different from a development setup - for starters traffic is. ;-)
So I thought about installing xhprof on our app servers as well and I was wondering about the implications it had on a production setup - even when not enabled.
Also, what would be the difference to Xdebug? Xdebug could be installed as well and be "enabled" through the GPC trigger? Are there any advantages xhprof has over xdebug?
I have not really benchmarked it, but afaik Facebook also wanted XHProf to be viable to be used on production machines. But I never benchmarked it myself.
With XDebug I would be more careful. If you are talking about just profiling, you dont want to use XDebug for that. if you also want to be able to do step by step debugging on a production server, then well not sure. Plus I think sometimes XDebug has issues playing with APC on one server. Then again I have both running on my development machine and I never encountered issues.
Benchmark of the debuggers - that should be your next blog post. ;-)
Also had never issues with APC and Xdebug.
Well even Derick says that XHProf is the better profiler. While obviously XHProf does not work as a step by step debugger like XDebug. But yeah knowing the overhead of adding either one of them (or both) to a production setup would be interesting. But I am not really a benchmark-wizard.
You may not want to enable XHProf on all your prod machines, but having it on a few and doing random sampling can be pretty neat. The big advantage of XHProf over Xdebug is the output format it has, which is native php arrays vs that old cachegrind stuff that takes 10-100 times more space.
Given you've got plain arrays, you can very easily work with the data, build graphs of average times etc and see how it evolves over time, which can help you identify issues with newly pushed code or just overall performance trends.
Ok, XHProf is rly nice and powerful but couldn't you detect all those problems using the basic web debug toolbar ?
Numbers of doctrine queries, time spent in specific parts like filters could have been spotted by dev easily during construction.
I tend to disagree to your first conclusion "don't waste your time with static code optimization analysis".
I wouldn't recommend this, since some big mistakes can only be detected watching the code (duplication/MVC respect, huge method, non/bad usage of OO..)
I would recommend usage of CI tools like Sonar, phpUndercontrol who can help targeting the code reviews to sensible parts.
