Poo-tee-weet

Query parameter handling in Symfony2

Lukas Kahwe Smith — Sat, 12 May 2012 16:02:25 +0200

So this topic has been going back and forth in my head a lot over the last months: how do we best handle query parameters in Symfony2? Obviously you can already access query parameters today already but it could be easier. Essentially what I want is a way for developers to easily configure what query parameters they expect and what values they expect. This is useful for several things like easier reading and validating of query parameters, self documenting API both for API docs for humans but also for machines. Now thanks to Alexander we have a solution that works. But there is the big question if this is really the right approach. For now ignore the fact that it only works with annotations for now, because that is fixable. But it does point to the question if this shouldn't be integrated into the routing configuration itself by adding query support for our implementation of uri-templates.

While this at first seems like the right place for this sort of configuration, it raises the questions of its own since path parameters (i.e. /{foo}) should be handled differently than query parameters (i.e. /{?bar}). For one I don't think that a mismatch on a route requirement of a query parameter cause the route to not match. However then it can quickly become confusing for the end user or it would require adding more and more syntax to handle all the different cases.

Feedback much appreciated!

Update: I have just merged the change to FOSRestBundle.

State of PHPCR

Lukas Kahwe Smith — Thu, 10 May 2012 12:45:28 +0200

It feels like every minute a PHP developer somewhere on this planet starts implementing something aching to a CMS from scratch. Some do it because their project is "so big" it that it "obviously needs" a custom solution. Some do it because their project is "so small" it "obviously needs" just a few days of hacking .. to build a custom solution. Let me briefly focus on the later group. Working in a company with less than 10 people building websites for customers a project needs a bit of a CMS to manage those 10 "semi static" pages seems to be the poster child example of this group. The devs whip up a DB table, slap an ORM in front, maybe even use some generator for the admin UI. Done. Later the clients also wants versioning and luckily many ORMs provide some solution for that. Easy. Permissions? Most frameworks provide some ACL system. Child pages? ORM has some tree algorithm supported. Fulltext search? Integrate ElasticSearch. Custom page types? Uhm well by now you have enough sunken costs that you will make that happen somehow too. Some morning you wake up and you have created the next Drupal or Typo3. If you did, then it would be hard to claim that you did it wrong because both are very successful projects. What PHPCR aims to be is to provide you with a short cut for this path. Or in other words there should be a PHPCR implementation that is so easy to use, with so many helpful higher level components in your favorite framework, that it becomes the natural choice for your next CMS needing project.

Every time anyone talks about PHPCR, they will mention Jackrabbit sooner or later as it is the basis for the currently most mature PHPCR implementation. Here, I just did it too. Jackrabbit requires Java. More importantly it is not trivial and most of you have never heard about it, let alone used it. I think for many high scale use cases its great that PHPCR has been integrated with Jackrabbit, but it needs to be relegated to a side note you mention when someone starts talking about scaling to millions of documents and many GBs of data. Once there is a PHPCR implementation that works with pure PHP, using any RDBMS (including SQLite which is bundled with PHP) it will become easier to just use that instead of whipping up your own tables. No more convincing the sys-admin guy about adding a new daemon to the setup. Instead being able to tell the client what other features they could get in the next code sprint.

So today, we are not there yet. We have an implementation of PHPCR that works on top of Doctrine DBAL to in theory support any RDBMS. Well in reality it already does though the search API only works with MySQL and PostgreSQL. It also doesn't support versioning and ACLs. Oh and if you drop more than 50 documents into it, search performance will start to hurt quickly. Bummer. But there is good news too. The core infrastructure is laid out thanks to Benjamin. Progress on it, while not rocket fast, is continuous thanks to Liip intern Adrien .. as a matter of fact if you go to our demo page it would run all but the admin interface on top of it just fine (yes this includes the cool inline editing!).

I believe that a decently experienced database developer would need a man month to fix up the searching to perform decently unto lets say 1k documents as well as implement simple versioning on top of SQLite. Another 2 man months should enable implementing permissions and support for even more documents, ACLs and a few other goodies. Is this something that a 10 developer company can commit to when offering that simple CMS solution to one of their customers? Obviously not. This takes a few developers who are willing to invest into the future. Until they come along unfortunately PHPCR will continue to not be a viable option for many small CMS projects.

Update: In this post I failed to mention that there is also already a pretty solid implementation of PHPCR written in C by the Midgard team. However just as a Java solution can't be our pitch, I believe that a custom extension cannot either.

Doctrine looking for feedback from its old contributors

Lukas Kahwe Smith — Wed, 09 May 2012 14:49:27 +0200

So Benjamin just went and wrote a license migration tool, which he will hopefully soonish have time to release as OSS itself. So far things are going well. The tool basically reads out the list of authors and then can list up their commits. Here is for example my list of commits. As some users didn't use the same config for all their commits, we might have authors in our system that in the real world map to the same people. Via an admin tool we can manually update the emails. The system can generate a hash for each author and send out notification emails where the author can then approve or disapprove of the change .. or simply reply that we got the wrong email. Works great as can be seen in the projects overview except that some users have not configured an email address properly when they setup git. However the bigger issue are old contributors back in the svn days. While many of these commits technically do not need to be approved since the code is actually no longer found inside Doctrine, it would be still great to get their approval. So if you find yourself or someone you know in the list of authors who have not yet approved, please let us know!

Not all licenses are created equal

Lukas Kahwe Smith — Fri, 27 Apr 2012 11:49:08 +0200

IANAL. As you are all are hopefully aware there are huge differences the exact "freedoms" allowed by the various open source licenses. I find that many younger developers have a natural affinity to the GPL, because they seem to feel its important to prevent someone from just taking their code, building upon it and not releasing their changes under an open source license when they distribute. Maybe with enough experience you start to realize that it happens close to never that a proprietary fork of an open source project ends up outpacing the original project. So why bother regulating this? It just makes legitimate business uses harder and in the grand scheme of things, I don't worry about this. People who prefer to go proprietary are likely not in the state of mind yet where I would want to work with them anyway.

So these days I prefer BSD, Apache and MIT licenses. My first open source project PEAR MDB was licensed as BSD. The Symfony2 eco-system is mostly licensed under the MIT. With basically Johannes Schmitt being the odd one out using the Apache license. We moved PHPCR to the Apache license in the early stages of the development push thinking this makes sense given that JCR is also licensed under the Apache license. We also moved Jackalope to Aapche thinking we might one day contribute it to the Apache foundation. There is one trouble here in that I recently learned with Apache 2.0, the FSF considers it incompatible with (L)GPLv2. Note that (L)GPLv3 is considered compatible.

Popular projects like Drupal being GPLv2+ and Doctrine LGPLv2 there is a problem. As Drupal is considering adopting PHPCR/Jackalope they would have to move to GPLv3+, which is easy for them to do since they made sure to include the "or later" when they went with GPLv2. Now for Doctrine the situation isn't so easy since its LGPLv2 without the "or later" option. Meaning we couldn't just drop the LGPLv2. That being said to me the LGPL is very fuzzy in combination with an interpreted language like PHP, but it would be good to avoid such a grey area. As a result all the current core developers have agreed to try and switch Doctrine to the MIT license. Due to its longer history moving the Doctrine ORM and more importantly the DBAL over is going to be quite hard, however the rest of the project is sufficiently young that its feasible to attempt this. Note that the DBAL actually started its life in Doctrine as part of the 1.x branch which in turn was a fork of my MDB project which was BSD licensed. Still since then a lot of code went in and out.

Now the big question is how on earth can we work through such a license change? In short its going to be really really hard. At least in the days of git and github we will not be faced with quite as many patches that were committed by someone else than the creator as back in the day many changes came in via patch files submitted to issue tickets or mailing lists. But there is still the risk of code lifts from other projects that might also be using a non MIT compatible license. So it seems like we really need a tool to help manage determining all the contributors and if they have approved the license change. Then also helping in reviewing individual commits if they are worthy of a copyright (typo fixes and obvious bug fixes do not). Maybe even do some analysis to show is which code portions remain without consent for the license change so that we can choose to rewrite them. Unfortunately I am not aware of any such tool. The only high profile project I remember having done such a change is VideoLan, but at least in their blog posts it seemed like they simply got the permission of their few dozen core developers. Anyway I will follow up this blog post with another one outlining some ideas I have for how such a tool could work.

Now to finish up this blog post I want to touch upon the topic of a CLA. When I first mentioned this issue on twitter, people were quick to point out that a CLA would make such a license change easier. Avid readers of my blog should know that I have spoken out against CLAs quite a lot in the past. The two main reasons is that it adds an annoying legal hurdle for contributors and that all CLAs I have seen include patent clauses which I reject. But it is true that a CLA would enable the entity to which the CLA would be signed to could easily do a license change (which could also be an irritating fact for some). Now there is a different kind of agreement called a Fiduciary Licensing Agreement which is used by the KDE. Essentially it gives the KDE guys the right to change the license, but also to defend the rights of the original creators in court. The difference to the CLA is that a FLA is also useful even if you do not force everyone to sign it. As a result the practice is to simply get people to sign it eventually after several contributions and not like a CLA before even the first patch can be accepted. As such an FLA does not suffer from the above mentioned legal hurdle and there is also no patent clause. Now with an FLA there can still be a number of patches from people that have not signed the FLA and before any license change these will need to be addressed by either: getting the contributor to sign the FLA, getting the contributor to at least allow the license change, determining that the patch was not copyright worthy or by throwing out or rewriting the given code. The benefit is that the work as to be done for much less code. So maybe step one before bothering with the license change is establishing an FLA.

Update: Instead of writing a blog post on a concept for a license change tool, Benjamin just wrote one with some feedback from me.

My take on the MicroPHP manifesto buzz

Lukas Kahwe Smith — Wed, 11 Jan 2012 10:54:38 +0100

Ed's recent blog post labeled the The MicroPHP Manifesto got a lot of attention. Personally I feel like Ed's way of expressing himself feels mostly like the rants of an angry person and not really a way of expressing a clear message. That is why I appreciate a much saner formulated follow up by Kris Jordan. In general I totally agree with Ed on the point that we need more decoupled components in the PHP world. The timing seems a bit odd since exactly that seems to be an emerging trend with all the various libraries cropping up since PHP 5.3. Where he looses me is with his seeming obsession with LOC minimization. It sounds to me like he assumes that any 3rd party code he pulls in, he automatically also has to maintain. To me that seems a bit odd. Obviously any 3rd party lib might contain bugs, which one might need to debug. But when choosing a 3rd party lib I focus on seeing if I trust the development team. I apply metrics such as track record, existing tests and documentation (oh and don't forget license). And of course I also browse the code and the response time and attitude in the bug tracker. If I hit an issue I do expect to be able to fix it myself if necessary, but in most cases I will just do my best to file a bug report with steps for how to reproduce the issue. Obviously being somewhat known in the PHP community for doing my share, I might be getting slightly better response times than most devs. But I still do not see how LOC in a 3rd party lib need to be a criteria for how hard it will be to maintain said dependency.

That part aside I would also like to comment a bit on micro vs. full stack frameworks. I don't have a perfect definition of either framework handy but I guess the main difference is that a micro framework aims to keep the number of out of the box supported use cases to an absolute minimum, while a full stack framework aims to cover a very broad sets of use cases. Traditionally micro frameworks also seem to have a much higher code to configuration ratio compared to full stack frameworks. Aka implement a certain behavior with a micro framework one usually needs to write code, while in a full stack framework quite a few things can be achieved by configuration alone. Furthermore full stack frameworks also define much more conventions (f.e. directory structures) and high level extension points (f.e. authentication and permission).

As a result to me the use case of micro frameworks is mostly in cases where I have very little business logic on the server. For example think of an application that basically just provides an API to fetch json data structures that map more or less 1:1 to some externally mapped database using some javascript MVC framework like Backbone.js for the actual business logic. However as soon as I have business logic on the server side I tend to prefer a full stack framework. The main reason is that I work at a web agency usually with team sizes of at least 5 developers but often more. Also the team compositions often change too. Furthermore frequently we integrate additional developers from the client. Here it is absolutely vital we do not waste time figuring out details such as directory structures. Furthermore many projects tend to have a lot of very similar requirements but rarely exactly the same. With the approaches that Ed and Kris seem to be describing, especially using (anonymous) functions I will find myself copy pasting code around to adapt it to specific needs of each project. This will require additional time and will obviously lead to a maintenance nightmare as this way I will end up maintaining more rather than less code.

If I gather things properly for Ed the situation is very different. He works on a set of products for his employer with a more or less fixed team. As such they can make sure that their solutions match exactly the current range of use cases. Furthermore the team will develop a set of implicit conventions and styles that will ensure consistency. They will likely have the chance of incrementally going over older, but still profitable, products to update them to their latest conventions. This way it is way more feasible for them to keep the number of required extension points to a bare minimum, because they have much more control over their current and future use cases. If they decide to use an anonymous function as an extension point, we are maybe talking about 2-3 different uses. Where as if I in a web agency design an extension point I expect 10-20 uses at least, by people in different teams. But as a web agency that believes in OSS we also release as much code as our clients let us, so the number of different use cases can grow to 100 or even more. Btw the situation is quite similar for companies that have large development teams spanning multiple offices or companies developing products that they expect will be customized by their clients. So imho frameworks such as Lithium and Slim mostly make sense for development teams of 5 or less developers working on in house SaaS applications that tend to have a large chunk of their business logic implemented in the client.

So the key take a way point is that when choosing to go micro or full stack its very important to consider in what kind of company on what kind of products you are working on. Unfortunately I have not seen any of the proponents of the MicroPHP movement discuss this aspect.