ramblings on PHP, SQL, the web, politics, ultimate frisbee and what else is on in my life

Where have all the core devs gone?

Symfony2 is without a doubt a very lively open source project. No small part thanks to Github, there is a flurry of tickets and more importantly pull requests coming in daily with dozens merged every week. These changes cover everything from small typos, to bigger refactorings, new test cases, documentation, performance optimizations. What even more positively stunning is that while there are a bunch of "usual suspects" the bulk of all of this comes from people that just submit the occasional change. Which means we have managed to make the community very inviting for new contributors. Something many other open source projects struggle with. So in a way one could say all is well. However I think its also important to define a direction for a project as a whole. I guess at a high level we are fine there too. We rarely have fundamental issues with determining what should go into Symfony2 and what should not. But what I feel is severely lacking is a prioritization and focus what should be happening within the next few months.

Back when we were preparing the very first stable release, there were several developers that knew the bulk of the Symfony2 code well and that were actively participating in discussing most of the proposed changes. They were also the ones who actually proposed the bulk of the changes. Due to their general overview of the code base even bigger changes tended to more quickly turn into mergeable change sets. Of course the fact that we could more liberally break backwards compatibility was also helpful. However it seems like many of these people have now shifted their focus to other areas. I, for example, am devoting most of my open source time to the Symfony2 CMF initiative and what time I have left I use to work on FOSRestBundle and the various Liip Bundles. Jordi is busy with Composer, while Johannes is busy with scrutinizer-ci.com and the serializer, Konstantin with Behat. People like Benjamin, Bulat, Jon and Kris also seem less visible. Even Stof seems to have cut down a bit, though he remains one of the more visible developers from back then. Additionally we had weekly (or was it bi-weekly) IRC meetings and a much more active developer mailing list. Heck the mailing list is seeing so little traffic and good questions and proposals often remain unanswered for days that we even considered killing the mailing list entirely.

The end result is to me that right now I have no clue how things make it or not make it into the next release. Fabien is tasked with going through all these pull requests of varying quality. This obviously takes a lot of time. There are people that are helping with the review, but if the PR is even slightly more involved it simply takes someone that understands the entire code base to make a call on if its ready or not. At the same time there is no way to know about Fabien's prioritizing. For example there is an open PR for adding default CLI events. It was almost ready for 2.1, it was definitely ready for 2.2 but its still not merged. Will it be merged before 2.3? Similarly we have gotten worse and worse and pooling resources to finish more complex pull requests. The resource watcher pull requests has been there for ages too. The issue have long been identified yet nobody is picking up the ball.

In a way we have moved from an onion structure of Fabien at the core and a group of 10 developers as a layer around Fabien and a hundred additional contributors around those 10, to a community with Fabien still at the core but with 500+ contributors directly around him. Again in many ways this is great because we have managed to prevent the creation of old boys clubs that put themselves above other contributors. But at the same time we have failed in networking all of the contributors into smaller units that can coordinate working and thereby pushing specific topics. And the end result is that such more complicated things fall into the lap of Fabien who remains a miracle in productivity, but even he has limits.

Now I have proposed some steps to solve this. Namely I believe there should be a release manager, similar to PHP core, that maintains a list of mid to large topics we want to focus on for the next release, that reminds people of the release process etc. This person should try to actively network developers to ensure that those topics get the attention they need to actually get done. Furthermore I think we should go back to at least bi-weekly IRC meetings to realign ourselves on focus topics. I am of course open to other suggestions. It could also be that I am just realizing that as I do not read all pull requests and tickets anymore that I have simply dropped out of that first layer around Fabien and that other people have now taken this role and that these people do indeed know what the focus areas are for the next release..

Update: I wrote this post last week and send it to Fabien in advance. He does not agree with all of my points but I will let him answer himself. Well he did already answer by reviewing the CLI events PR and figuring out a much better solution, writing it and getting it merged. Fabien-style!


Re: Where have all the core devs gone?

Semi related an interesting blog post:

Re: Where have all the core devs gone?

Personally I'm very happy to see you working FOSRestBundle and having a nice job at Liip, to see Jordi working with Composer, to see Johannes working on scrutinizer-ci.com and the serializer, to see Konstantin working on Behat and phpspec2.

I'd rather see them working on great tools for PHP community than working on Symfony core, because Symfony is in stabilization process anyway and they also need to benefit of their knowledge of Symfony for making great tools we are all using.

I see lot of ppl contributing every day like Tobias Schultze, Victor Berchet, Jean-François Simon, Florin Patan, Martin Hason and lot more. They know pretty much Symfony and there contributions are really great.

I see too the contribution of some recently arrived people who are giving back to the community as they are real user of Symfony and are giving a very good and appreciated feedback to Symfony.

You have noticed that I never used the term "Core" dev or team as i don't really believe in it. Every contributor is the core team or nobody is. As you said you can be really active one week then you have other stuff to do and you are not anymore, if the community is waiting for some individuals things won't change fast as they need.

I said in my comment just almost what you said in your post except that I see the glass half full and you half-empty.

Re: Where have all the core devs gone?

Hi Lukas

IMHO, having a release manager is a good idea. This does not solve all the issues you mention, and that every OS communities might face at some point, but it can help.
Regarding PRs, I'm not sure what to say. For eZ Publish, we have the Diff Squad for the review. It seems that we more or less have this at the moment for Symfony as you pointed out. However, having only one person merging can be an issue indeed.

@Pascal : I agree and disagree with you. Of course every contributors are important ! However, having a kernel in the team is also good and avoids dilution. It's good but we also need to avoid the ivory tower syndrom like it was the case in the past for PHP core. What I'm saying is that the kernel team should stay open :-).

Re: Where have all the core devs gone?

I have never seen a better solution to the scaling problem than lieutenants. How those lieutenants get appointed, how formal they are, how much real authority they have vs virtual authority, what their scope is (version vs. subsystem vs. whatever), etc. can vary widely. But the only way to scale a person is delegation, and that delegation has to come with authority as well as expertise.

I've yet to see an exception to that.

Of course, the other factor is that Symfony core itself is pretty darned stable, so it doesn't need dozens of people working on it anymore. That's not a bad problem to have.

Where will new core devs come from?

As a newb, both in the Symfony2 and general Open Source communities, I would love to see some kind of contributor training session at Symfony Live conferences.

Perhaps I'm just weak sauce, but I'd love to be a contributing member of the community - I'm just not sure how to go about it and know from experience that ignorant enthusiasm can cause a lot more harm than good in volunteer projects.

Maybe if contribution education/orientation could be incorporated as a regular part of conferences, a path to ramping up involvement could be developed to turn users into casual contributors and casual contributors into core contributors over time.

Re: Where have all the core devs gone?

Most (all?) Symfony Live conferences have a hackday now. This is the perfect place to get the low down on how to contribute and submit your first PR.

Re: Where have all the core devs gone?

I was discusing this topic with my colleague the other day and we've came to an agreement that it's hard to get on Symfony when it's constantly changing. What is needed right know is finished LTS release which will be used by many vendors out there for some period of time.

With release 2.2 and 2.3 release in short period of time it's hard to upgrade bigger systems as they tend to depend on many bundles which aren't available on 2.2 or 2.3 yet.