One of the key things that attracts developers to the OSS world is the no nonsense attitude people are allowed to display in OSS development. Where in the closed source world, developers are rarely allowed to openly explain the pro's and con's of their product, its expected in the OSS world. People that BS will get slapped hard quickly. It doesn't pay off and at best the benefit would be short term until someone takes another look. There are no restrictions on publishing benchmarks etc. This is awesome.
Now given that everything is out in the open and that people expect perfect honesty when discussing your project or even more importantly when comparing your product with other products, it can be very non trivial. The reason being that its damn near impossible to be objective, simply because everybody weights things sightly different based on their experience. Also one of course knows the own product best.
That being said, I think Greg Smith did a very good job doing exactly that in a comparison of PostgreSQL and MySQL that was recently published on techdocs. I especially liked the approach how the document was build up. He first created a wiki page on the internal developer wiki (note if you are browsing this wiki, you may find other marketing type pages still under the development) along with jump starting a discussion on the PostgreSQL developers list. Many people participated by bringing up topics, discussing others and helping to refine the language. Once the document got to a stable state, he moved the content to the official PostgreSQL techdocs.
He also made sure that the bias and core assumption of the document are explained in clear. Furthermore he expressly has asked for criticism and noted that he is willing to link to any solid commentary. Another key thing is that he noted which versions of each product he was comparing. This should help ensure that even as both products mature, the information will remain valid. It should also make things easier when it comes time to update the document.
So what are your thoughts on the general challenge on comparing two or more OSS products by developers of one of the products being compared? Also what are your thoughts on this specific comparison?
