Poo-tee-weet

Why I am pissed.

Lukas Kahwe Smith — Sat, 13 Mar 2010 14:18:02 +0100

Being the verbose kind, I guess I will put in a summary at the top leaving the original post below for everyone to read all the little details. Essentially the summary is that I (and many other core developers) agree that HEAD/trunk in its current form was stalling future development on PHP. Jani indirectly jump started the discussion of moving this stumbling block out of the way by essentially doing two commits that violate the most fundamental rules of PHP development (and common sense courtesy) we have. I am worried that this will lead to confusion in the user base and other seem to think that the "ends justify the means". I also think that we need clearer processes to make it easier for new developers to have a clear path to follow in order to get things done in PHP.

Full story:
So this all requires a bit of background. I still remember my first contact with a PHP core developer. I do not quite remember the year, but it was quite some time ago. Anyway Jani was great, we filed a bug report on the imap extension. He fixed the bug in no time. IIRC we also ended up sharing a room at the second IPC in Frankfurt I attended. I think only a few years later I realized that Jani is the same guy a lot of people were complaining as being rude. But I guess few of these people realize how much time Jani has spend on verifying bug reports and what a tedious job this is, especially since most bug reports are quite low on the actual relevant details and come with a "fix my code for me, now!" attitude. In other words, I get along fine with Jani and I have great respect for his contributions to PHP.

Next piece of background. I am also one in the long list of people that has made a fool of himself by predicting a release of PHP6 in about 18 to 24 months (quite a few people have given this prediction over the past 3-4 years I guess). I have also been one of the people that has tried to motivated people at various times to finish those last 2% that seem to be missing to complete PHP6 in terms of functionality. As such I have opposed a PHP 5.4 because I felt its time to focus on finish up PHP 6 which we have promised people for so long. After releasing 5.3.0 I was hoping things would happen. But last weekend I came to the realization that we have waited long enough. That even if PHP 6 would be unicode bliss (at least in terms of features, probably not in terms of performance), the fact of the matter is that in all of the many years nobody put in the time to finish it. This imho is an indirect proof that the approach taken apparently does not hold enough merit to the world in general. Furthermore looking at internals since last summer it has been more dead than ever: PHP 6 aka HEAD aka trunk had become such a motivation killer that nothing was happening.

At the same time I knew that suggesting to move trunk to a branch and copying 5_3 to trunk would cause a lot of confusion. Not only among those reading internals, but also that this would quickly make it to the news sites of the world. Call it politics, marketing or just an unwillingness to cause thousands (millions?) of PHP developers distress and uncertainty, but I figured it would be better to propose this with a semi solid plan and giving a number of PHP core developers the heads up that I will bring this to the list, so that they also could prepare themselves. So I started talking to people offlist, either in person, via phone, via IRC/skype or via email with the full intention of going to the list no later than the end of this month, ideally sooner rather than later. I should also note, I do not code C, I also do not consider myself a unicode expert. So if I wanted to present a plan, I obviously needed to talk to people to get my facts straight.

Anyways, this week Jani decided to commit a patch. I guess I didn't mention one more piece of background. Jani had asked to get this patch into PHP 5.3 during the pre alpha stage, but at the time Johannes and I (I was co-RM back then) felt that the patch hadn't been tested enough in HEAD and that while we knew the patch fixes several bugs, we felt the risk of introducing new bugs was too high, especially since Michael who wrote the original patch for HEAD was not always around to fix things. Johannes had tried to get the patch into 5.3 six months earlier, but at the time nobody had time to do the work. So we decided to stick with the known bugs, instead of fixing them and risking new bugs in a very core component of PHP close to going alpha. So basically Jani committed a huge patch into a stable branch, without talking to anyone about this, despite knewing full well that the RM had specifically vetoed this patch in principle. Furthermore the patch from my understanding even breaks the ABI which makes the patch even more a no go.

Now I was quite surprised when I saw this patch and I at first got confused why Johannes would allow this patch in now only to find out that he didn't. Like several others I told Jani that he needs to revert this patch, but I then did the fatal mistake of expressing sympathy for Jani by saying that he probably was acting out because of frustration over PHP 6 been stalled as the patch in question was already applied there. Now instead of reverting the patch Jani decided to take things a step further now and simply created a PHP 5_4 branch, despite the fact that during the last discussion over the creation of 5_4 the consensus was to instead focus on PHP 6. Of course there was little focus on PHP 6, which is why I (and many others) also wanted to drop the current approach. If the next version should be 5.4 or 6.0 is of course not a trivial decision to make. For one we had already decided on a lengthy list of [stuff we wanted to remove from PHP for PHP 6, and even if we were to drop unicode from PHP 6, chances were we would stick to the original plan for all other things. Obviously we can however not start stripping these features in a 5.4.

Anyway, the good news in all of this was that people were starting to get excited about the future of PHP again. But what irritated me was that apparently a fan club for Jani's actions was forming. It seemed that people were saying he was providing a "shock to the system" that revived core. I do not have reason to believe that it would have been less successful to revive internals without basically pissing all over an RM decision and then following this up by pissing over all sorts of unwritten rules that it's not a lone wolf call to create a branch with the name of a possible future PHP version. Worse yet, if this even gets you a fan club, it means that the already insanely maddening work of an RM just got that much harder. Furthermore Jani was happily committing away into the 5_4 branch, while others were still working on 5_3. Why should they suddenly be obligated to merge stuff into 5_4 because of Jani's shooting from the hip decision? Shouldn't the point in time for such a branch be a bit more consciously decided? Calls to get the branch removed were answered that its all good and sooner or later the 5_4 branch will be renamed into trunk anyway, even though there was still no clear decision.

I am sorry, I might be a rule loving German but WTF?!?!?!?! At this point I cannot even try to put into words the level of disappointment. Of course most of us do the bulk of the work on PHP in our spare time, but we do have to have a sense of responsibility to the world out there, with many people depending their professional life's on PHP, something I assume we welcome, and we are toying with the image of PHP here. We are also toying with the position of the RM here, one of the few roles we have in PHP, which is critical to getting at least a tiny bit of structure into the development process. PHP is simply too big to go without any structure at all. So to me the only thing to do at this point was to slam a bit on the breaks. For one I asked that Jani's commit karma be revoked until we sort out the situation. This would have given us the opportunity to communicate to the world at large: "Sorry, someone jumped the gun, do expect some real news to be communicated soon." Instead we have news (heise is probably the most important german speaking tech site) sites and the twitter-verse announcing the creation of 5_4 and the death of native unicode inside the engine. But the cheerleading went on and any criticism of Jani was deemed "bashing". Again WTF?!?!?!

I should note one more thing. A lot of people got pissed off by the fact that I decided to first talk to people offlist. This includes many people who I have not yet contacted and a few I did contact. I realize that it's not ideal to do so. I hope nobody is implying I might be complaining about Jani because I was hoping for a moment of fame singlehandedly reviving PHP development. I also realize that going to the list directly could not have been worse than what Jani has created now. I also realize that with all the unwritten rules and power structures in PHP its hard for new guys to get the things happening they want (the fact that traits is still not committed to trunk is a prime example of this very problematic situation). In that light to many Jani's stepping over all rules and processes might seem empowering to the people that are not part of the "inner circle" of core PHP developers. I think thats very short sighted. Again I am German, so I have a tendency to trust rules a lot, but I feel the more rules and processes the actually write down and follow, the more of a chance new people will have in getting "shit done" without turning the project's principles to "shit" like what Jani did with his commits the past two days. So instead of cheering on Jani to continue pissing all over the very few rules we do have, I feel people should more work towards identifying some more rules we can agree on that will ensure that new guys have a clearer processes to getting things into core.

I hope that with this blog post I have clarified my position a bit more and I hope that I have convinced some of you that thinking first rather than just shooting from the hip is the better approach when it comes to dealing with such large sweeping decisions like reconsidering the branch into which so much work was pored into and on which so many promises for the future of PHP were based. After taking a bit of time to playing some "laser frisbee" and listening to soothing music I am now again a bit hopeful that we can bring some dearly needed structure into the process to determining a realistic plan for the future of PHP that best serves our large user base.

Stop building gold on top of crap

Lukas Kahwe Smith — Wed, 24 Feb 2010 20:00:52 +0100

At the recent Symfony Live conference I used the core team QA to address the audience to push for building new dedicated apps on top of general purpose frameworks like symfony and friends. More importantly those efforts should be released as early as possible in the development process as open source to ensure that others join instead of creating their own and then eventually releasing more and more redundant solutions. See the various symfony CMS solutions are an example of how wrong things can go. We now have several solutions whose architectural differences are either cosmetic or simply bad design decisions probably a result of trying to invent things in the small ecosystem of a company project team. So I was very happy to hear then that phpBB will adopt Symfony 2 for their next version. Hopefully this will become a role model for others.

At the conference we also talked about how to better merge the current CMS efforts for Symfony 2, but I fear that will proof to be tricky. From my experience the easiest approach is to put stuff out there early so that people can pool around a leader (or a hand full of leaders): there might be one or two other competitors that differentiate themselves via fundamentally different approaches (lets say a RDBMS vs. a document store CMS or using flat files or a database for configuration). With the ecosystem we have for CMS solutions for symfony 1.x we have several solutions that were open sourced by companies in the hopes of getting new customers, where collaboration with the community is more a nice side effect than the real focus. There the risk of turf wars is quite high since there would simply be too many potential leaders.

So I hope that there will be a fast mover that will proof to be a good leaders as well. Liip might even attempt to become just that building on top of the Jackalope project to provide a JCR based CMS for the PHP world. I assume phpBB will have a need for various social networking features which many currently popular CMS solutions also offer, so there might also be an opportunity for them to lead some of these aspects, but it might be too large a burden to deal with the Symfony community at large, while they are just joining themselves. Meaning they might also need to convince their developers that this is the way to go, which means they probably want to see more code and less talk.

PHP adopting branching kicking and screaming

Lukas Kahwe Smith — Thu, 11 Feb 2010 18:55:14 +0100

I remember that back when I was co-RM for PHP 5.3 one of the very painful parts was the crying and moaning about the commit freezes we put into place while packaging up a new release. The reason being we were on CVS, if people kept committing while a release was being tested it would effectively prevent any sort of QA. Obviously we could have just tagged at the start of the QA process, but pretty much every time build fixes were necessary, if you then also throw in normal commits it wouldn't work at all. So we usually reserved about 2 days where no commits were allowed. Since there is no truly reliable way to inform people there would frequently be one or two commits still, but it was somewhat manageable. However it obviously made it quite annoying for people to work in this time interval. Either commits had to be delayed or someone had to remember to merge the change from HEAD once the commit freeze was over.

Right after 5.3 came out we made the switch over to Subversion. Yes its not a distributed VCS, but its well known, fairly similar to CVS and someone spend the time to actually make the switch a reality. I still believe that for a large OSS project it makes sense to use SVN as the central repository and if needed simply ensure that there is infrastructure to mirror to some DVCS. Anyways one of the big benefits of the move away from CVS is that we now actually have a bit more fun (merge tracking and friends) in terms of working with branches not only for a minor release (aka 5.2, 5.3 etc) but even for patch releases (aka 5.3.1, 5.3.2 etc).

This means we no longer need to do commit freezes. Of course we could have also done the same with a bit more pain with CVS. Actually even with SVN its still not really trivial to manage all of this and so a lot of core developers have spoken out against using branching even for patch releases. However Pierre has stepped up with a little script that filled in all commits to the main branch and if they have been merged yet or not. The initial version used the wiki and now we have a fancy tool that makes the process very transparent and the risk for things falling through the cracks are now pretty slim. Even if something does slip through its a lot easier to identify which parts exactly didn't get applied. I think this is really a top class effort to make the life of RMs easier. And the work of the RMs is to make the life of all core developers easier .. so everybody wins :)

Update:
Was just pondering my title. I guess the summary of the above is that looks like PHP's kicking and screaming about branching is now more like James Brown: I feel good!

Short HipHop blurp

Lukas Kahwe Smith — Wed, 03 Feb 2010 22:46:53 +0100

Well we all have read the announcement and chatted, twittered or whatever about it. I agree thats its cool this is being open sourced. I do fear however this means that there is no one less company looking after APC (mainly leaving just Yahoo then?). For now I think this is mainly interesting for "researchers" willing to work on improving the compatibility and more importantly the workflow around HipHop. I do not expect that anyone can really get a meaningful return on investment who makes this their business anytime soon (unless they can sell people on the hype), besides FB of course.

Overall I am not so impressed with the numbers. Even a 50% speed gain across the board .. what does that really mean? Lets say Facebook spends one third of the request (and I expect this to be even lower) on network and disk I/O, then this means that they managed a performance increase by a factor of 4 in the pure PHP processing parts. Note that I talked to some of the guys at the presentation and they said that the noted 50% speed up was not a result of switching the webserver (the binary generated by way of the C++ transformation includes a multi threaded webserver).

This is of course a nice speedup but wouldn't we expect at least a 2 fold .. or more increase if we could transform our PHP code into a C extension? This approach seems a lot more feasible for the masses. Like we could just drop in a ext/symfony or ext/ZF and thanks to autoloading magic our setup would continue to run unchanged. While it might not get that 4 fold speed up .. I would be willing to wager that it would get a 2-3 speed increase in the pure PHP processing.

So again I welcome any company open sourcing stuff that solved real world problems, especially if they are also willing to support the community in understanding their solution. But to me this solution will only matter for the ultra large companies where every single percent of saved CPU cycles matter. For the rest of us I think we need to rely on APC for a bit longer. But maybe with this transformator out there, other people will start working on the much more feasible approach of writing a PHP Code to C extension transformator.

Wanna help out on wiki.php.net?

Lukas Kahwe Smith — Sat, 16 Jan 2010 20:42:15 +0100

Recently I have devoting my attention towards other projects like djtechtools.com and un-informed.org/un-i.org away from php.net. That is not to say I have stopped following whats going on or that I do not care anymore. Its just that I feel that I can make more of a difference in these other projects. Also it might just be sort of a phase thing. I am sure one day I will become focused on php.net stuff again. Until then I hope others will pick up the slack and make sure that PDO gets love, at people diligently follow up on todo items mentioned on internals and that someone else picks up wiki.php.net and takes care of some regular maintenance stuff. Like there is a new version of dokuwiki out that fixes some issues, adds some features and adds PHP 5.3 support. If you are interested drop me a line or submit a comment. Who ever sticks around for a while and gives the wiki some love will obviously get commit access with that ever so shiny @php.net email alias.